Senior research crawling engineer

About the Role

Build large-scale web crawling and data acquisition infrastructure for AI research and model development. You will work with research teams to create reliable, high-quality datasets from a changing and often adversarial web.

What You'll Do

• Build and operate crawlers across diverse domains, including social media, travel, and multilingual websites.
• Design high-throughput, fault-tolerant systems that collect millions to billions of URLs per day.
• Handle anti-bot systems, rate limits, and dynamic, JavaScript-heavy sites.
• Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.
• Work with research teams to align data collection and dataset quality with modeling needs.
• Monitor crawl coverage, performance, and data quality, and optimize infrastructure for cost, latency, and reliability.

What We're Looking For

• At least 2 years of experience building web crawlers or large-scale data pipelines, with 4 or more years preferred; candidates with PhDs may qualify with 2 or more years.
• Experience operating distributed data pipelines at terabyte scale and building high-throughput, fault-tolerant data acquisition systems.
• Strong proficiency in Go, Rust, Python, Java, or C++, with deep knowledge of HTTP, networking, and browser behavior.
• Hands-on experience addressing anti-bot systems, rate limits, and JavaScript-heavy websites; experience with proxies, IP rotation, and request orchestration is valuable.
• Familiarity with headless browsers such as Playwright or Puppeteer, and cloud or bare-metal infrastructure.
• Experience with data quality evaluation, NLP pipelines, or machine learning dataset curation is a plus.

Compensation & Benefits

Base salary is $160,000 to $250,000 USD annually. Visa sponsorship is not available.

Location

Remote. No primary location was specified.