This content originally appeared on DEV Community and was authored by joedev090
Hey coders!!
A famous term in web development, Web Scraping
Web scraping is the process of collecting unstructured and structured data in an automated manner.
Main use cases include price monitoring, price intelligence, news monitoring, lead generation, market research and more.
Let's check how to create a small script to get data from a popular website to see the temperature.
Previous to continue you have to install the dotnet cli.
If you want to check in more details, please check the next reference:
https://learn.microsoft.com/en-us/dotnet/core/install/macos
We are going to build the script in VS code with c#.
- Open a folder in VS code and open a terminal
- The next step, run this command to create a console app in c#
dotnet new console -o webscrapping
- Then, we proced to build a small app, so the next step we will paste this code in program.cs
using HtmlAgilityPack;
using System;
using System.Net.Http;
namespace WebScrapping
{
class Program
{
static void Main(string[] args)
{
// send get request to weather.com
String url = "https://weather.com/weather/today/l/ad585d4294c07f7d8e78c8e7d6d3945a4e7e67f135f13f6c013df05e0d6f728e";
var httpClient = new HttpClient();
var html = httpClient.GetStringAsync(url).Result;
var htmlDocument = new HtmlDocument();
htmlDocument.LoadHtml(html);
// get the temperature degree
var temperatureElement = htmlDocument.DocumentNode.SelectSingleNode("//span[@class='CurrentConditions--tempValue--MHmYY']");
var temperatureValue = temperatureElement.InnerText;
Console.WriteLine(temperatureValue);
}
}
}
- Finally in the terminal run the command
dotnet build
You can see the next result in your console:
60°
And it will be the same we can see in the website
Note:
To verify the class name, please check the dev tools and in the html section you can verify the html tags and so on.
This content originally appeared on DEV Community and was authored by joedev090
joedev090 | Sciencx (2024-10-03T22:34:38+00:00) How to code a web scraper in c# dotnet 8. Retrieved from https://www.scien.cx/2024/10/03/how-to-code-a-web-scraper-in-c-dotnet-8/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.