Data Engineering

STS-DATA ENGINEERING-1

Posted a few seconds ago

Erode

Job Summary:
We are looking for motivated Data Engineers to join our team and help design, build, andmaintain scalable data pipelines and warehouse solutions. The ideal candidates will havehands-on experience with modern data engineering tools and a strong interest in real-time and batch data processing.

Responsibilities

• Design and implement data models to support analytical and reporting needs, ensuringefficient schema design for warehouse and transactional systems
• Build, maintain, and optimize ETL/ELT pipelines using Python, Apache Kafka, Apache Flink for batch and real-time data processing and PySpark
• Set up and manage data warehouse infrastructure (Doris), including table design,partitioning strategies, and performance tuning
• Develop and schedule data workflows using Apache Airflow, ensuring reliability andmonitoring of pipeline jobs
• Integrate data from multiple sources including PostgreSQL, MS SQL, and MinIO into the data warehouse
• Containerize data pipeline components using Docker for consistent deployment across environments
• Ensure data quality, consistency, and integrity across pipelines through validation and testing
• Collaborate with analysts, data scientists, and other engineering teams to understand data requirements and deliver solutions
• Troubleshoot and resolve issues related to data pipelines, storage, and processing performance
• Document data flows, schemas, and pipeline architecture for team reference

Required Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field
• 1 - 3 years of hands-on experience in data engineering
• Proficiency in Python and SQL for data processing and transformation
• Experience with Apache Kafka for stream processing and messaging
• Experience with Apache Flink or similar stream/batch processing frameworks
• Hands-on experience with PySpark for large-scale data processing
• Familiarity with Apache Airflow for workflow orchestration and scheduling
• Working knowledge of relational databases (PostgreSQL, MS SQL)
• Exposure to OLAP/data warehouse systems (Doris or similar columnar databases)
• Experience with Docker for containerization of data pipelines
• Familiarity with object storage solutions (MinIO or similar S3-compatible storage)
• Understanding of data warehousing concepts, data modeling, and ETL/ELT pipeline

design

Preferred Skills

• Experience building real-time data pipelines using Kafka + Flink
• Knowledge of distributed computing concepts (partitioning, parallelism, fault tolerance)
• Familiarity with CI/CD practices for deploying data pipelines
• Experience with version control (Git)
• Understanding of data quality, validation, and monitoring practices
• Basic knowledge of cloud platforms (AWS, Azure, or GCP) is a plus

Soft Skills

• Strong problem-solving and debugging skills
• Good communication and collaboration abilities
• Eagerness to learn and adapt to new tools in a fast-paced environment