Senior Data Engineer
IDFC FIRST Bank
Date: 1 day ago
City: Thane, Maharashtra
Contract type: Full time
Job Requirements
Job Requirements
Role/ Job Title: Senior Data Engineer
Business: New Age
Function/ Department: Data & Analytics
Place of Work: Mumbai/Bangalore
Roles & Responsibilities
'Minimum 6 years of Data Engineering experience and 3 years in large scale Data Lake ecosystem
Proven expertise in SQL, Spark Python, Scala, Hadoop ecosystem,
Have worked on multiple TBs/PBs of data volume from ingestion to consumption
Work with business stakeholders to identify and document high impact business problems and potential solutions
First-hand experience with the complete software development life cycle including requirement analysis, design, development, deployment, and support
Advanced understanding of Data Lake/Lakehouse architecture and experience/exposure to Hadoop (cloudera,hortonworks) and AWS
Work on end-to-end data lifecycle from Data Ingestion, Data Transformation and Data Consumption layer. Versed with API and its usability
A suitable candidate will also be proficient Spark, Spark Streaming, AWS, and EMR
A suitable candidate will also demonstrate machine learning experience and experience with big data infrastructure inclusive of MapReduce, Hive, HDFS, YARN, HBase, Oozie, etc.
The candidate will additionally demonstrate substantial experience and a deep knowledge of data mining techniques, relational, and non-relational databases.
Advanced skills in technical debugging of the architecture in case of issues
Creating Technical Design Documentation (HLD/LLD) of the projects/pipelines
Secondary Responsibilities
'Ability to work independently and handle your own development effort.
Excellent oral and written communication skills Learn and use internally available analytic technologies
Identify key performance indicators and establish strategies on how to deliver on these key points for analysis solutions
Use educational background in data engineering and perform data mining analysis
Work with BI analysts/engineers to create prototypes, implementing traditional classifiers and determiners, predictive and regressive analysis points
Engage in the delivery and presentation of solutions
Managerial & Leadership Responsibilities
'Lead moderately complex initiatives within Technology and contribute to large scale data processing framework initiatives related to enterprise strategy deliverables
Build and maintain optimized and highly available data pipelines that facilitate deeper analysis and reporting
Review and analyze moderately complex business, operational or technical challenges that require an in-depth evaluation of variable factors
Oversee the data integration work, including integrating a data model with datalake, maintaining a data warehouse and analytics environment, and writing scripts for data integration and analysis
Resolve moderately complex issues and lead teams to meet data engineering deliverables while leveraging solid understanding of data information policies, procedures and compliance requirements
Collaborate and consult with colleagues and managers to resolve data engineering issues and achieve strategic goals
Key Success Metrics
'Ensure timely deliverables. Spot Data fixes. Lead technical aspects of the projects. Error free deliverables.
Job Requirements
Role/ Job Title: Senior Data Engineer
Business: New Age
Function/ Department: Data & Analytics
Place of Work: Mumbai/Bangalore
Roles & Responsibilities
'Minimum 6 years of Data Engineering experience and 3 years in large scale Data Lake ecosystem
Proven expertise in SQL, Spark Python, Scala, Hadoop ecosystem,
Have worked on multiple TBs/PBs of data volume from ingestion to consumption
Work with business stakeholders to identify and document high impact business problems and potential solutions
First-hand experience with the complete software development life cycle including requirement analysis, design, development, deployment, and support
Advanced understanding of Data Lake/Lakehouse architecture and experience/exposure to Hadoop (cloudera,hortonworks) and AWS
Work on end-to-end data lifecycle from Data Ingestion, Data Transformation and Data Consumption layer. Versed with API and its usability
A suitable candidate will also be proficient Spark, Spark Streaming, AWS, and EMR
A suitable candidate will also demonstrate machine learning experience and experience with big data infrastructure inclusive of MapReduce, Hive, HDFS, YARN, HBase, Oozie, etc.
The candidate will additionally demonstrate substantial experience and a deep knowledge of data mining techniques, relational, and non-relational databases.
Advanced skills in technical debugging of the architecture in case of issues
Creating Technical Design Documentation (HLD/LLD) of the projects/pipelines
Secondary Responsibilities
'Ability to work independently and handle your own development effort.
Excellent oral and written communication skills Learn and use internally available analytic technologies
Identify key performance indicators and establish strategies on how to deliver on these key points for analysis solutions
Use educational background in data engineering and perform data mining analysis
Work with BI analysts/engineers to create prototypes, implementing traditional classifiers and determiners, predictive and regressive analysis points
Engage in the delivery and presentation of solutions
Managerial & Leadership Responsibilities
'Lead moderately complex initiatives within Technology and contribute to large scale data processing framework initiatives related to enterprise strategy deliverables
Build and maintain optimized and highly available data pipelines that facilitate deeper analysis and reporting
Review and analyze moderately complex business, operational or technical challenges that require an in-depth evaluation of variable factors
Oversee the data integration work, including integrating a data model with datalake, maintaining a data warehouse and analytics environment, and writing scripts for data integration and analysis
Resolve moderately complex issues and lead teams to meet data engineering deliverables while leveraging solid understanding of data information policies, procedures and compliance requirements
Collaborate and consult with colleagues and managers to resolve data engineering issues and achieve strategic goals
Key Success Metrics
'Ensure timely deliverables. Spot Data fixes. Lead technical aspects of the projects. Error free deliverables.
See more jobs in Thane, Maharashtra