This content originally appeared on DEV Community and was authored by Sibelius Seraphini
Running code concurrently
To make execution faster, we usually run code concurrently.
One way to run code concurrently in JavaScript is to call many promises at the same time without waiting for their results, then you use Promise.all to await all the promises to finish. Check the example below:
const promiseA = asyncFnA();
const promiseB = asyncFnB();
const results = await Promise.all([promiseA, promiseB]);
The code above will execute asyncFnA and asyncFnB concurrently, and Promise.all will await both execution to resolve.
Running many promises at the same time
Let's take a look at this code
const users = await User.find(); // return all users in the database
const results = await Promise.all(users.map(async (user) => processUser(user));
This code will run as many promises as users in your database. Node and JavaScript does not handle so well many promises running concurrently, Go handles this well.
This code will probably consume a lot of CPU and memory and will run out of memory.
To solve this, we need to process all these promises in batch
Processing promises in Batch
export async function processPromisesBatch(
items: Array<any>,
limit: number,
fn: (item: any) => Promise<any>,
): Promise<any> {
let results = [];
for (let start = 0; start < items.length; start += limit) {
const end = start + limit > items.length ? items.length : start + limit;
const slicedResults = await Promise.all(items.slice(start, end).map(fn));
results = [
...results,
...slicedResults,
]
}
return results;
}
Usage
const results = await processPromisesBatch(users, 100, processUser)
processPromisesBatch will slice your items in chunks of size N, and will execute N promises at the same time.
This ensures it won't consume a lot of CPU and memory, and exhaust the event loop.
In Conclusion
Understanding the limitation of your programming language and runtime, can help you design a solution to workaround them.
Share solutions that you design based on limitations of your programming language, runtime or framework.
Woovi
Woovi is a Startup that enables shoppers to pay as they like. To make this possible, Woovi provides instant payment solutions for merchants to accept orders.
If you want to work with us, we are hiring!
This content originally appeared on DEV Community and was authored by Sibelius Seraphini
Sibelius Seraphini | Sciencx (2023-03-17T12:58:31+00:00) Processing promises in Batch. Retrieved from https://www.scien.cx/2023/03/17/processing-promises-in-batch/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.