This content originally appeared on Level Up Coding - Medium and was authored by Rohan Krishna Ullas
Use web scraping to give your subscribers a puzzle to solve everyday
In this article we’ll build and deploy a simple python script to send out a mail to all the subscribers, containing a new puzzle every day! In the end, we’ll deploy the project to Heroku so that the script runs at a scheduled time every day.
Here is the GitHub link for the project.
I have divided the article into 4 parts for better understanding:
- Setting Up
- Web Scraping and Sending Mails
- Scheduling the run time
- Deploying to Heroku
All right then, let’s start building!
1. Setting Up
First, let’s set up a virtual environment for our project.
C:\Github\Daily-Puzzle2>virtualenv venv1
C:\Github\Daily-Puzzle2>call venv1/Scripts/Activate
(venv1) C:\Github\Daily-Puzzle2>
Next, create a file requirements.txt which contains the names of the packages we need for our project.
After this, we’ll install them into our virtual environment.
(venv1) C:\Github\Daily-Puzzle2>pip install -r requirements.txt
2. Web Scraping and Sending Mails
In this step, we’ll perform web scraping to identify the link of the puzzle we need to send to the recipients. We use the requests library to get a response and then scrape the response to get the correct links.
GeeksforGeeks has a collection of interesting puzzles all numbered from 1 to 85. So, we’ll identify the link beginning with :
"https://www.geeksforgeeks.org/puzzle-"+day_number
day_number is a variable to keep track of the number of the puzzle we need to send. It is stored in a .txt file and initially set as ‘1’. It is incremented after sending the link of that particular puzzle to the subscribers.
We’ll separate the functionality of ‘getting the correct link’ and ‘sending mails’ into two methods getLink() and sendMail() respectively.
Note : If you are using a Gmail account to send your emails, you’ll need to set ‘Allow less secure apps’ to ON. Be aware that this makes it easier for others to gain access to your account, so it will be better to use a throwaway account for development.
3. Scheduling the run time
While deploying to Heroku, we can either run the script 24x7 and use time.sleep() to call sendMail() periodically, or we can use a scheduler to run the worker process at regular intervals. Here, I have used APScheduler , a lightweight, in-process task scheduler. It provides a clean, easy-to-use scheduling API, has no dependencies and is not tied to any specific job queuing system.
We’ll need to create a clock.py file to define the schedule.
Next, we need to configure our Procfile. Procfile is a file that is essential while deploying to Heroku which specifies the commands that are executed by the app on startup. You can use a Procfile to declare a variety of process types, including:
- Your app’s web server
- Multiple types of worker processes
- A singleton process, such as a clock
- Tasks to run before a new release is deployed
Here’s how our Procfile looks like.
4. Deploying to Heroku
First we’ll need to create a new app from the Heroku dashboard.
We can deploy to Heroku directly from our local machine using Heroku CLI or connect the app to our GitHub repository. Here, I have used Heroku CLI for deployment. For installing the CLI, head over to the downloads page and install it.
Steps to deploy using Heroku CLI :
- Go to the directory containing the files. If not initialized as a git repo, then do these steps :
C:\Github\Daily-Puzzle2>git init
C:\Github\Daily-Puzzle2>git add .
C:\Github\Daily-Puzzle2>git commit -m “commit message”
2. Login to your Heroku Account
C:\Github\Daily-Puzzle2>heroku login -i
3. Connect to the App
C:\Github\Daily-Puzzle2>heroku git:remote -a app_name
4. Push to Heroku
C:\Github\Daily-Puzzle2>git push heroku master
5. Scale up the clock process
C:\Github\Daily-Puzzle2>heroku ps:scale clock=1
That’s it!! Now we should be getting our new puzzle every day at 8 am!
In case you want to test it out first, change scheduling in clock.py like this :
Resources :
Automate Sending Daily Mails Using Python was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Level Up Coding - Medium and was authored by Rohan Krishna Ullas
Rohan Krishna Ullas | Sciencx (2021-05-12T19:36:08+00:00) Automate Sending Daily Mails Using Python. Retrieved from https://www.scien.cx/2021/05/12/automate-sending-daily-mails-using-python/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.