Senior Data Collection Engineer

Job Summary

A company is looking for a Senior Data Collection Engineer.

Key Responsibilities

• Design and build robust web crawlers for high-scale data extraction using Scrapy
• Enhance and maintain infrastructure for automated testing, deployment, and monitoring of spiders
• Ensure data integrity and accuracy through robust validation mechanisms and collaboration with internal teams

Required Qualifications

• Experience with Git workflows, code reviews, and CI/CD pipelines
• Familiarity with cloud infrastructure, preferably AWS
• Knowledge of web environment standards and networking protocols
• Proficient in developing scalable web crawlers and data pipelines using Python and Scrapy
• Prior experience mentoring or leading junior developers is a plus