This content originally appeared on Level Up Coding - Medium and was authored by Pratik Choudhari
Easy to implement, without much overhead
As a developer, you might have experience dealing with long running tasks, and you wish this time-consuming and blocking task could be worked in the background while the main app serves other requests. I know, there is asynchronous programming, but it won’t work when we have CPU bound operations.
In python, we can set up task queues that work in background. I am talking about Redis queues, and in this article, we will see how to create a task from a function/method and execute it in the background.
Prerequisites
Redis ≥ 3.0.0 and Python ≥ 3.5 must be installed in the system.
Install Redis: https://redis.io/download
Install Python: https://www.python.org/downloads/
Introduction to Python RQ
Python RQ is a library that allows a user to create background jobs out of functions. These jobs can be scheduled and managed via RQ APIs.
IMAGE
Components of RQ
- Job
A job is a task to execute. It has its own id, status(running, failed, completed), submission time, target function, and the keyword arguments for the target function.
2. Queue
A queue, as the name suggests, is a system used to line up jobs and dispatch them in FIFO fashion to the worker.
3. Worker
Probably the most important part of RQ, a worker takes a job from a queue, specified while starting the worker, and executes the target function with the keyword arguments passed. A worker can work multiple queues, and multiple workers can work the same queue, increasing the throughput of the system.
4. Registry
A registry acts as a categorizer for jobs. For example, all failed jobs are moved into Failed Job Registry and completed jobs in Finished Job Registry.
Working
Installation
Install using pip:
pip install rq
or the latest version(might contain bugs)
pip install git+https://github.com/nvie/rq.git@master#egg=rq
Using Python RQ
First, start by importing the required classes and initialize the redis queue object.
Next, to add our long running task to a queue we will use the enqueue() method, it takes the target function as a required argument.
As soon as a task is queued, it will wait in the queue until it is picked up by a worker and after completion job status, end time and result will be updated.
Start a worker via terminal by specifying the queue to serve.
rq worker long_task_queue
After a few moments, we can check the submitted task results using the job id obtained after enqueuing the job.
Conclusion
Thank you for reading this tutorial. I will be coming up with more tutorials on Python RQ and automation, stay tuned!
Level Up Coding
Thanks for being a part of our community! Before you go:
- 👏 Clap for the story and follow the author 👉
- 📰 View more content in the Level Up Coding publication
- 🔔 Follow us: Twitter | LinkedIn | Newsletter
🚀👉 Top jobs for software engineers
How to handle long-running tasks in Python using Redis queues was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Level Up Coding - Medium and was authored by Pratik Choudhari
Pratik Choudhari | Sciencx (2022-07-24T23:50:35+00:00) How to handle long-running tasks in Python using Redis queues. Retrieved from https://www.scien.cx/2022/07/24/how-to-handle-long-running-tasks-in-python-using-redis-queues/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.