Associate III - Data Engineering

Job Role: Senior Data Engineer
Job Location: Any UST Location
Notice Period: Immediate Joiners Only

Job Summary
We are seeking a highly skilled Senior Data Engineer with 6+ years of experience in designing, developing, and optimizing large-scale data platforms and pipelines. The ideal candidate will have strong expertise in Databricks, Apache Spark, ETL/ELT development, cloud data engineering, and data modelling. This role involves building scalable data solutions, implementing modern Lakehouse architectures, and collaborating with cross-functional teams to support data-driven business initiatives.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks and distributed data processing frameworks.
• Build and optimize high-performance big data solutions using Apache Spark with Python, Scala, or SQL.
• Develop and manage Data Lake and Lakehouse architectures on cloud platforms such as AWS, Azure, or GCP.
• Implement batch and real-time data ingestion pipelines from diverse structured, semi-structured, and unstructured data sources.
• Optimize Databricks workloads through performance tuning, cluster configuration, workload management, and cost optimization.
• Create and maintain reusable Databricks notebooks, workflows, jobs, and automation frameworks.
• Leverage Delta Lake features including ACID transactions, schema evolution, partitioning, and data versioning.
• Design efficient data models to support analytics, reporting, and machine learning workloads.
• Collaborate with Data Scientists, Data Analysts, Product Teams, and Business Stakeholders to deliver scalable data solutions.
• Implement data quality checks, monitoring frameworks, and automated validation processes.
• Ensure compliance with enterprise data governance, security, privacy, and regulatory standards.
• Troubleshoot production issues, perform root cause analysis, and drive continuous platform improvements.
• Contribute to CI/CD implementation and DevOps best practices for data engineering solutions.

Required Skills & Qualifications

• 6+ years of experience in Data Engineering and Big Data technologies.
• Strong hands-on experience with Databricks and Apache Spark.
• Proficiency in Python, Scala, and SQL for data processing and transformation.
• Experience designing and implementing ETL/ELT pipelines in enterprise environments.
• Strong knowledge of Data Modeling, Data Warehousing, and Lakehouse architectures.
• Experience with Delta Lake, data partitioning, schema management, and performance optimization.
• Hands-on experience with cloud platforms:
• • Amazon Web Services (AWS)
  • Microsoft Azure
  • Google Cloud Platform (GCP)
• Knowledge of distributed computing concepts and large-scale data processing.
• Experience implementing CI/CD pipelines and deployment automation.
• Strong troubleshooting, debugging, and performance tuning skills.
• Excellent communication and stakeholder management abilities.

Preferred Qualifications

• Experience with streaming technologies such as Spark Structured Streaming, Kafka, or Event Hubs.
• Knowledge of Data Governance, Data Quality, and Data Security frameworks.
• Experience with Infrastructure as Code (Terraform, CloudFormation, ARM Templates).
• Databricks, AWS, Azure, or GCP certifications are preferred.
• Exposure to MLOps and advanced analytics platforms is a plus.

Data Engineering, Databricks, Data Processing, Data Modeling