This content originally appeared on DEV Community and was authored by Anuj Singh
Overview of My Submission
Archives headlines/content from news websites (selected news website which are for now New18 and IndiaToday it can be change) every 3hr. which allows users to historically see the news and how it’s reported on different sites.
By using a package node-schedule to run background job every 3hr. and puppeteer for scraping the content from a website and this date will be saved in Appwrite database.
Render this data on a client-side application (website).
Submission Category:
Web2 Wizards
Link to Code
Frontend/Client-Side Application
New Archiver
Backend/Server-Side Application
Additional Resources / Info
Do check it out in 1.5x for a quick walkthrough on my application and how it works
A quick walk through of my Application and basically how it works
PS- I didn't focus on security so that's why my id is still shows in main app. And if I have to I can just set it the .env
Backend
- Run node index.js
- Cron schedule can be set to any time (for now let say every min * * * * *)
- After that it Scrape data mainly img and headline
- Now coming up it check the collection list
- Basically for 1 things
- Does a collection exit with current date it if not then create a collection with given attributes
- If collection exits then create the document in that collection with the scraped data
1.
2.
That's it for the Backend
FontendRenders the data that was in collection
And how it render the data of the selected date?
Well it's easy To start of I created the collection with Date id
For more info you can connect with me on Twitter
OoO
The backend can be hosted on digitalocean so it can always keep running with appwrite droplet. If you are wondering.
This content originally appeared on DEV Community and was authored by Anuj Singh
Anuj Singh | Sciencx (2022-04-22T14:02:21+00:00) News Archiver. Retrieved from https://www.scien.cx/2022/04/22/news-archiver/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.