Databricks Data Engineer (Azure + Databricks AI)

Databricks Data Engineer (Azure + Databricks AI)

Location: Coimbatore, Tamil Nadu, India

Employment Type: Full Time

Experience: 5–8+ years in data engineering, with 3+ years hands-on in Databricks

Notice Period: Immediate joiners to 30 days preferred

Shift: Partial US overlap, e.g., 12:30 PM – 9:30 PM IST

About the Role

We are looking for a Databricks Data Engineer to design, build, and optimize scalable data and AI pipelines on Microsoft Azure. You will own lakehouse architecture using Delta Lake and Unity Catalog, and help bring GenAI and machine learning use cases into production using Databricks Mosaic AI capabilities. You will work closely with data scientists, analysts, and business stakeholders to turn raw data into trusted, AI-ready assets.

Key Responsibilities

• Design and build batch and streaming pipelines in Azure Databricks using PySpark, Spark SQL, and Delta Lake, following medallion (Bronze/Silver/Gold) architecture.
• Develop declarative pipelines and orchestrate workloads using Lakeflow (Delta Live Tables) and Databricks Workflows/Jobs.
• Implement data governance, lineage, and access controls using Unity Catalog.
• Ingest data from multiple sources using Azure Data Factory, Event Hubs, ADLS Gen2, Auto Loader, and REST APIs.
• Build and support AI/GenAI solutions on Databricks, including RAG pipelines with Vector Search, model deployment through Model Serving, and LLM integration via Databricks Foundation Model APIs or Azure OpenAI.
• Manage the ML lifecycle with MLflow, covering experiment tracking, model registry, evaluation, and monitoring.
• Prepare feature sets and curated datasets for ML models and AI agents, using Databricks Feature Store and Agent Framework where applicable.
• Tune Spark jobs and clusters for performance and cost, including partitioning, liquid clustering/Z-ordering, Photon, and serverless compute.
• Implement CI/CD for data and AI assets using Databricks Asset Bundles, Azure DevOps or GitHub Actions, and infrastructure as code (Terraform/Bicep).
• Ensure data quality through pipeline expectations, validation frameworks, and Lakehouse Monitoring.
• Collaborate with stakeholders to gather requirements, estimate effort, and document solutions.

Required Skills:

• Bachelor's degree in Computer Science, IT, or a related field (B.E./B.Tech/MCA or equivalent).
• 5+ years of data engineering experience, including 3+ years with Databricks on Azure.
• Strong proficiency in Python (PySpark) and SQL.
• Deep knowledge of Delta Lake, Spark internals, and performance tuning.
• Hands-on experience with Azure services: ADLS Gen2, Azure Data Factory, Event Hubs, Key Vault, Entra ID (Azure AD), and Azure Monitor.
• Practical experience with Unity Catalog and lakehouse governance.
• Experience with Databricks AI features such as Mosaic AI Model Serving, Vector Search, AI Functions, or MLflow for GenAI.
• Understanding of LLM concepts including embeddings, chunking, RAG, prompt engineering, and evaluation.
• Experience with Git-based development and CI/CD pipelines.
• Solid grasp of data modeling (dimensional, Data Vault, or lakehouse patterns).

Preferred Qualifications

• Databricks certifications: Data Engineer Associate/Professional, Generative AI Engineer Associate, or Machine Learning Associate.
• Microsoft certifications: Fabric Data Engineer Associate (DP-700), Azure Data Engineer Associate (DP-203), or Azure AI Engineer Associate (AI-102).
• Experience building AI agents or chatbots with Databricks Agent Framework, LangChain, or LlamaIndex.
• Familiarity with Databricks SQL, AI/BI Dashboards, and Genie spaces.
• Experience with real-time streaming (Structured Streaming, Kafka).
• Exposure to Microsoft Fabric, Power BI, or Azure Synapse.
• Experience working with US/UK clients or in regulated domains such as BFSI, healthcare, or insurance.

Soft Skills

• Strong problem-solving and communication skills, with the ability to explain technical trade-offs to non-technical audiences.
• Comfortable working in Agile/Scrum teams and owning deliverables end to end.
• Ability to mentor junior engineers and contribute to best practices.