ajay

Senior Software developer

Wealth Excel • Full time

Aug 2019 - Mar 2024 • 4 yrs 7 mos

- Designed and maintained scalable web scraping infrastructure to extract structured data from complex, dynamic websites using Python, Scrapy, Selenium, and Playwright. - Built robust anti-detection systems including proxy rotation, user-agent spoofing, CAPTCHA solving integration, and request throttling to ensure high success rates. - Developed data pipelines to clean, validate, and ingest scraped data into databases (PostgreSQL/MongoDB) and data warehouses for downstream analytics. - Reverse-engineered JavaScript-rendered sites and mobile APIs to identify reliable data endpoints and minimize scraping overhead. - Mentored junior developers on scraping best practices, legal compliance (robots.txt, terms of service), and ethical data collection standards. ## Key Achievements - Reduced scraping failure rates from ~35% to under 5% by architecting a distributed crawling system with automatic retry logic and dynamic fingerprint rotation. - Scaled data collection from 100K to 2M+ pages per day by migrating from single-instance scripts to a Kubernetes-based microservices architecture. - Cut infrastructure costs by 40% through intelligent caching, selective headless browsing, and switching to lightweight HTTP-based scraping where possible. - Delivered a real-time competitor pricing monitoring system that processed 500K+ SKUs daily, directly supporting revenue optimization decisions. - Implemented a monitoring and alerting dashboard (Grafana/Prometheus) that reduced mean time to detection (MTTD) for scraper failures from hours to under 10 minutes.