Data Engineer (Control M, Hadoop, Spark, Hive, Big Data)

Responsibilities:

• Design, implement, and maintain data pipelines for the migration of data from on-premises systems to Hadoop-based platforms.
• Develop and optimize data processing jobs using Spark, Hive, and Python.
• Manage job orchestration and scheduling using Control-M, ensuring timely and accurate data delivery.
• Collaborate with cross-functional teams to understand data requirements and deliver efficient solutions.
• Perform code quality checks and peer reviews to ensure best practices, maintainability, and adherence to coding standards.
• Ensure end-to-end operational stability of data pipelines by proactively identifying and resolving bottlenecks, failures, and data quality issues.
• Ensure data quality through rigorous cleaning and validation processes.
• Documented data flow processes, transformation logic, and framework usage to support onboarding and troubleshooting.

Requirements:

• Proficiency in Python and SQL.
• Strong experience with Hadoop ecosystem tools (Hive, Spark).
• Worked extensively with transformation components, mapping development, and workflow orchestration in Informatica/DataStage.
• Experience with job scheduling and monitoring using Control-M.
• Familiar with pipeline-as-code concepts and using Jenkins files for automation of build and deployment processes.
• Solid understanding of database systems including Teradata, Oracle, and SQL Server.
• Ability to analyze and troubleshoot large-scale data processing systems.
• Experience in the banking or financial services industry.
• Knowledge of data warehousing concepts and star/snowflake schema design.