This content originally appeared on DEV Community and was authored by Darshan
Introduction
In this article, I will teach you to scrape Google News Results using Node JS.
Requirements
Before we begin this tutorial, we need to install some NPM packages, which we will use further in this tutorial.
We will use NPM libraries Axios to extract our target HTML data and Cheerio JS for the parsing the extracted HTML data.
Target
We will scrape the Google News Results of query Football.
Procedure
We have installed and set up all the things to prepare our scraper. Now we will use Axios to make a get request to our target URL, which is: https://www.google.com/search?q=football&gl=us
, and Cheerio JS to parse the HTML.
If you inspect the Google News page you will find that all the main articles are contained inside this div.BGxR7d container. By searching in this container we got the tag for title as .MBeuO, for the snippet as .GI74Re, for the thumbnail as div.NUnG9d img, for the date as div.ZE0LJdand for the link as .WlydOe.
This makes our code as:
You can copy the code from the following link: https://github.com/Darshan972/GoogleScrapingBlogs/blob/main/GoogleMapsNewsScraper.js
Results:
By using Google News API
If you want to scrape Google News Results easily, without making a scraper, as scraping can take a lot of time sometimes, you can try this API.
Serpdog also provides 100 free searches per month, and if you want to scale your requests quota, you can buy paid plans also.
Example request code:
Where APIKEY - the key you will receive on signing up on Serpdog.
Results:
Conclusion
In this tutorial we learned to scrape Google News Results. If you have any questions please feel to ask me in the comments. Follow me on Twitter. Thanks for reading!
Additional Resources
This content originally appeared on DEV Community and was authored by Darshan
Darshan | Sciencx (2022-07-18T17:37:56+00:00) Scrape Google News Results. Retrieved from https://www.scien.cx/2022/07/18/scrape-google-news-results/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.