Senior Data Engineer

Senior Data Engineer

Work location- Paya Lebar

1. Lead the design and operation of scalable batch and streaming data pipelines on AWS and Databricks using Python, PySpark, and SQL.
2. Build Bronze, Silver, and Gold layers in Delta Lake and develop reusable data models and curated datasets for analytics, reporting, and ML.
3. Integrate databases, files, APIs, and event streams using incremental loads and change data capture, with support for late-arriving data, backfills, and replay.
4. Enforce data contracts, schema validation, and reconciliation, and implement end-to-end lineage, encryption, and least-privilege access.
5. Orchestrate workflows with Databricks and AWS services, and automate testing and deployment across environments using Git, CI/CD, and Terraform.
6. Monitor pipeline health and data freshness, lead incident troubleshooting, and optimise Spark and SQL workloads for performance and cloud cost.