Senior Data Engineer – Databricks

Job Description

· We are looking for an experienced Senior Data Engineer – Databricks to design, develop, and maintain scalable data pipelines on the Databricks platform.

· The role requires strong expertise in PySpark, Databricks, and modern data engineering practices, along with experience modernizing legacy data pipelines and building high-performance data solutions in cloud environments.

Key Responsibilities

· Design, build, and maintain scalable data pipelines using Databricks and PySpark.

· Develop end-to-end data workflows including data ingestion, transformation, and consumption.

· Optimize Spark jobs, cluster configurations, and pipeline performance.

· Manage and orchestrate workflows using Databricks Jobs and notebooks.

· Refactor legacy ETL pipelines into modern PySpark-based ELT frameworks.

· Implement data quality checks, monitoring, and error handling mechanisms.

· Design and maintain Delta Lake tables with proper optimization strategies.

· Collaborate with data architects, analysts, and infrastructure teams to deliver reliable data solutions.

· Troubleshoot and resolve production data pipeline issues.

Required Skills & Qualification

· Strong Data Engineering fundamentals including ETL/ELT pipeline design.

· Hands-on experience with PySpark (DataFrames API, Spark SQL, performance tuning).

· Experience with Databricks platform including workspace, clusters, notebooks, and job orchestration.

· Knowledge of Delta Lake features such as ACID transactions and schema evolution.

· Strong Python programming for data processing and automation.

· Experience with data modelling (Dimensional modeling, Data Vault, or Lakehouse architecture).

· Strong SQL skills for data transformation and analysis.

· Experience with cloud platforms (Azure, AWS, or GCP).

· Experience with Git version control and CI/CD practices.

· Minimum 8 years of experience in data engineering or related roles.

· 2–3 years of hands-on experience with Databricks.

· Experience building and maintaining production-grade data pipelines at scale.

· Proven experience modernizing legacy data pipelines.

· Experience working in Agile development environments.

· Certifications (Mandatory) Databricks Certified Data Engineer Associate OR Databricks Certified Data Engineer Professional