Radwan Faris

Senior Python Web Scraping and Anti bot Automation Engineer

Freelancer.com • Full time

Dec 2025 - Jun 2026 • 6 mos

Joined as the sole engineer at Drivebuy responsible for the company's core data acquisition infrastructure — a nationwide lead generation system targeting automotive marketplaces at scale. Key Contributions: Refactored and optimized legacy scraping scripts, migrating undocumented, resource-intensive architecture into a reliable, scalable JSON API flow capable of smoothly processing up to 10 million daily requests and thousands of concurrent connection System Modernization: Redesigned the data collection pipeline with a production-grade stack—AWS SQS, Apache Kafka, ClickHouse, and AI-powered data enrichment—significantly improving throughput, reliability, and data quality. Proprietary Anti-Bot Solution: Engineered an in-house hCaptcha solving package after evaluating and finding all third-party APIs insufficient, eliminating dependency on volatile open-source alternatives. Cost & Infrastructure Optimization: Conducted a deep system audit to replace expensive third-party APIs with proprietary, in-house solutions—reducing operational overhead by over 75% across compute, proxies, and observability tools. Monitoring & Observability: Built custom New Relic dashboards providing real-time visibility into data pipelines, enabling proactive alerting and faster incident resolution. Business Impact & Cross-Functional Synergy: Collaborated closely with the lead-gen team and leadership to align technical deliverables with core business needs, directly contributing to a 30% increase in lead volume and higher lead quality.