Overview

In this role you will design and operate scalable distributed scraping systems to extract high-quality data from thousands of domains. You will work within the Scraping and R&D; team to push innovative approaches and share knowledge, collaborating with Customer Success, Operations, and Data. You’ll tackle anti-bot challenges and build monitoring to prevent blockers, contributing to fast, scalable visibility for retailers and brands. This is a hands-on, problem‑solving role for someone who loves turning difficult data problems into reliable solutions.

Compensaciones / Ventajas

- Flexible working environment
- Volunteer time off
- LinkedIn Learning
- Employee-Assistance-Program (EAP)

Responsabilidades

- Design and build scalable distributed scraping architectures for thousands of domains
- Reverse engineer APIs and mobile apps using tools like Frida, mitmproxy, Charles Proxy, burp
- Detect and bypass anti-bot measures to create human-like crawlers
- Lead technical innovation in scraping, prototype approaches, share knowledge with the team
- Set up anomaly detection and monitoring to catch blockers or failures early





Requisitos principales

- 5+ years of Python experience and involvement in large-scale scraping projects
- Deep understanding of anti-bot technologies and bypass techniques, fingerprinting, WAFs, JavaScript challenges
- Experience with Scrapy, Requests, httpx, Selenium, Playwright or similar headless drivers
- Proficiency with proxies (residential, datacenter, rotating), session management, cookie injection, TLS tweaking
- Strong debugging skills across browser, HTTP traffic, and device-level interactions
- Independent problem solver with a track record of solving new problems
- Familiarity with Parquet, Avro or real-time stream processing
- Knowledge of databases like MongoDB, InfluxDB, and Redis and how to optimize them

- Independent thinking
- Strong problem-solving abilities
- Self-motivation and proactivity
- Python for large-scale scraping
- Anti-bot bypass techniques, fingerprinting, WAFs, JavaScript challenges
- Scrapy, Requests, httpx, Selenium, Playwright or headless drivers

📌 Senior Software Engineer (Web Scraping, Data acquisition, Data collection) - barcelona
🏢 NielsenIQ
📍 Barcelona