4get needs your help!
Google has been hard at work lately patching out ways for me to scrape search results without a chrome webdriver. Over the last 2 years I have maintained this project, the scraper has broke like, 6 times.
- First, I was scraping the search page for the LG-p500H. They changed the page completely, the scraper broke.
- Then, I was scraping the normal web version of Google, they added a JavaScript challenge.
- Then, I was scraping the page shown to Lynx users, they also broke that.
- Right now, I'm scraping some shitty pagination endpoint that (somehow) doesn't require JavaScript.
Once they patch that endpoint out, the only option remaining will be to scrape the search endpoint for older iPods.
4get is unable to render any sort of JavaScript. To do that, I would need some sort of webdriver, which would consume a lot of resources considering I have around 1200 users doing searches everyday.
This is where I need your help: You can create API keys to do searches, provided you have a Google account here: https://developers.google.com/custom-search/v1/overview
From there, you can click on Get a Key and E-Mail me the API key you get at will <at> lolcat.ca
Each key gets me 100 searches/day, provided they don't change anything!
I already have like, 22 keys littered around my hard drive, if everyone gives me API keys we can do some epic trolling!!!!!!
Now, if you don't see any activity on the API keys, that's normal. It's because those API keys are my last resort, I will attempt everything I can before resorting to using them. Maybe I'll find another hidden endpoint, who knows...
Thank you all so much for the support over the years!!
- will
Update
I got 50!!! keys. Thank you all for your support. I won't respond "thanks" to every email I get, but trust me I'm taking note of everything! :)
I'll consider this operation a success when I have 100 keys, which should cover every search request we would ever need at the current scale!
Update #2
Thank you all again for the keys. I got all 50 of them!! :) I don't reply most of the time cause there's so many of you emailing me so if its just about giving me an API key I won't reply, but trust me I am truly grateful to have you guys help me out. Anyways, I have to announce some bad news, they recently patched out the iPod UA from their noJS search page. The only endpoint remaining is the one SearxNG and 4get uses, the pagination endpoint. After they patch that one out, it will be truly over. The only way to get results will be through Google CSE or the API. I will keep you guys posted xoxo
Update #3
Yeah... They got rid of every single noJS search endpoint. The pagination endpoint now requires JS. I am now scraping the API, so each key counts. Im at 51 keys right now, let's keep going!