Senior Associate-Date Engineer
Acuity Analytics
We are seeking a highly skilled Data Engineer with a strong software engineering mindset and deep expertise in Python-based data engineering. This is not a traditional SQL-centric data engineering role. We are specifically looking for engineers with strong programming, debugging, and distributed data processing experience, who can build scalable and maintainable data platforms using modern cloud and lakehouse technologies.
The ideal candidate will have experience developing data pipelines using Python and Spark, working with modern data lake platforms such as Databricks and Snowflake, and deploying solutions on AWS. Experience with Dagster for data orchestration is highly desirable.
Key Responsibilities
Data Platform Development
- Design, develop, and maintain scalable data pipelines using Python and distributed processing frameworks.
- Build reusable, production-grade data engineering components and frameworks.
- Develop and optimize high-volume batch and streaming data processing solutions.
- Implement robust testing, monitoring, and observability for data pipelines.
- Troubleshoot complex data processing and performance issues.
- Build and maintain modern lakehouse architectures using Databricks, Snowflake, and AWS data services.
- Design scalable data models and storage strategies for structured and semi-structured data.
- Develop ingestion, transformation, validation, and enrichment pipelines.
- Support data quality, lineage, and governance initiatives.
- Develop and manage workflow orchestration solutions using Dagster or similar orchestration frameworks.
- Automate data operations and deployment processes.
- Monitor pipeline performance and optimize reliability and efficiency.
- Build cloud-native data solutions on AWS.
- Integrate data services across storage, processing, orchestration, and analytics platforms.
- Implement security, scalability, and resiliency best practices.
- Write clean, maintainable, and well-tested code.
- Participate in peer code reviews and technical design discussions.
- Collaborate with product managers, architects, and engineering teams.
- Follow software engineering best practices, including CI/CD and version control.
Experience
- 5+ years of professional experience in Data Engineering.
- Hands-on experience building production-grade data platforms and pipelines.
- Strong software engineering and coding background.
Primary Skills (Must Have)
- Python (Expert Level)
- PySpark / Apache Spark
- Data Pipeline Development
- Object-Oriented Programming
- Debugging and Performance Optimization
- REST APIs
- Git
- Experience with one or more:
- Databricks
- Snowflake
- Delta Lake
- Apache Iceberg
- Modern Lakehouse Architectures
Experience with multiple services such as:
- S3
- Lambda
- Glue
- EMR
- ECS/EKS
- Step Functions
- RDS
- CloudWatch
- ETL/ELT Development
- Data Modeling
- Data Quality Frameworks
- Data Validation and Monitoring
- CI/CD for Data Pipelines