Tech
When curl, TLS spoofing and headless all fail: how I got past Akamai with a real browser
I scrape data for a living, and most of that job is the same fight over and over: a site has data I need, and it really doesn't want me to have it. This is the story of one wall that beat every trick in the usual playbook — and what finally worked.
The target
I needed a set of listings from a large classifieds site (mobile.de — car listings). Nothing exotic: title, price, a few fields per listing. The kind of thing that should take twenty minutes.
It took a lot more than twenty minutes....
Read the full discussion on Dev.to
This article was aggregated from Dev.to. Click to join the conversation.
View on Dev.to