StreamSets Developer - Data Integration

Date: 1 week ago

City: Indore, Madhya Pradesh

Contract type: Full time

Key Responsibilities

Development and Implementation :

Design, develop, and implement data pipelines using StreamSets Data Collector to ingest, transform, and deliver data from various sources to target systems.
Write and maintain efficient, reusable, and reliable StreamSets pipelines, adhering to coding standards and best practices.
Develop custom processors and stages within StreamSets to address unique data integration challenges.
Implement data validation and quality checks within StreamSets pipelines to ensure data accuracy and consistency.
Optimize pipeline performance and resource utilization for high-volume data processing.
Automate deployment and monitoring of StreamSets pipelines using CI/CD tools.

Quality Assurance And Testing

Develop and implement comprehensive test plans and test cases to validate StreamSets pipeline functionality and data integrity.
Conduct thorough testing, debugging, and troubleshooting of StreamSets pipelines to identify and resolve issues.
Standardize and enforce quality assurance procedures for StreamSets development.
Perform performance testing and tuning to ensure optimal pipeline performance.

Problem Solving And Support

Research and analyze complex software-related issues and provide effective solutions.
Respond to and resolve production issues related to StreamSets pipelines in a timely manner.
Provide technical support and guidance to other team members on StreamSets development.
Monitor and analyze pipeline logs and metrics to identify potential issues and proactively address them.

Strategic Alignment And Collaboration

Understand and align with department, segment, and organizational strategy and operating objectives.
Collaborate with data engineers, data analysts, and other stakeholders to understand data requirements and deliver effective solutions.
Document StreamSets pipeline designs, configurations, and procedures.
Participate in code reviews and knowledge sharing sessions.
Contribute to the development of data integration best practices and standards.
Makes decisions regarding own work methods, occasionally in ambiguous situations, and requires minimal direction and receives guidance where needed.
Follows established Qualifications :
Bachelor's Degree in Computer Science, Information Technology, or a related field.
3-5 years of hands-on experience in systems analysis or application programming development, with a focus on data integration.
Proven experience in developing and deploying StreamSets Data Collector pipelines.
Strong understanding of data integration concepts and best practices.
Proficiency in SQL and experience with relational databases.
Experience with various data formats (e.g., JSON, XML, CSV, Avro, Parquet).
Familiarity with cloud platforms (e.g., AWS, Azure, GCP) and cloud-based data services.
Experience with version control systems (e.g., Git).
Strong analytical and problem-solving skills.
Excellent communication and collaboration skills.
Ability to work independently.

(ref:hirist.tech)

Post a CV