Data Engineer (Web Scraping/Crawling)
OSI Digital
Experience:3-5 Years
Location: Hyderabad
We are looking for a skilled Web Crawling Engineer with strong expertise in Python and hands-on experience building scalable crawlers and scrapers. The ideal candidate should have deep knowledge of crawling frameworks, anti-blocking techniques, and proxy management, along with experience solving challenges like captchas and rate-limiting.
Key Responsibilities
- Design, develop, and maintain scalable and efficient web crawlers for data extraction.
- Work with Scrapy, Requests, and other Python libraries to implement crawling solutions.
- Handle anti-scraping measures, including IP blocks, rate limits, and captchas.
- Implement and manage proxy rotation and session management strategies.
- Ensure high-quality and structured data extraction, cleaning, and storage.
- Monitor crawler performance, troubleshoot issues, and optimize for reliability and efficiency.
- Collaborate with the data engineering team to integrate crawled data into pipelines.
- Stay updated with the latest tools, techniques, and best practices in web crawling.
- Strong programming skills in Python.
- Proven experience with Scrapy, Requests, BeautifulSoup, Selenium (if required).
- Hands-on experience solving blocking issues, handling
- Good understanding of proxy usage, rotation, and fingerprinting avoidance techniques.
- Knowledge of HTTP, cookies, headers, sessions, and request/response cycles.
- Experience with databases (SQL/NoSQL) for storing crawled data.
- Strong debugging, problem-solving, and analytical skills.
- Experience with cloud environments (AWS, GCP, Azure).
- Knowledge of data pipelines and ETL processes.
- Bachelor’s degree in Computer Science, Information Technology, or equivalent practical experience.
- 3+ years of experience in web crawling or related fields