Common Mistake — Processing Big Data in a Transaction

This content originally appeared on Level Up Coding - Medium and was authored by Anas Anjaria

Database

Common Mistake — Processing Big Data in a Transaction

There is a concept of a transaction in a database system. It means:

Executing all operations or nothing at all.

It ensures data integrity. But do you know how to handle big data in a transaction?

I came across a piece of code that executes big data in a single transaction.

In my opinion, it is a bad practice.

Reasons — Bad practice to handle big data in a single transaction

The problem with processing all the chunks in a single transaction is nondeterministic execution time. It means that overall execution time depends on big data, which will increase as the data grows.

Normally, we set query timeout in our production environment so that our system does not wait forever for executing database operations.

In this specific use case, our system will keep throwing timeout exceptions since it can’t complete a big transaction in a given time frame.

Another problem is that it might increase the load on our database system.

The recommended practice to handle big data in a transaction

The recommended way to process big data is to process each (equally sized-) chunk in a transaction. This way,

• We have a deterministic execution time.
• It would not overload our database system.

Thanks for reading.

If you have any questions, please feel free to ask.

If you enjoyed this post, you might also like the following series.

• Databases
• Site reliability engineering

Want to connect?
Facebook | LinkedIn | Twitter

Subscribe to get my work directly into your inbox.
https://medium.com/subscribe/@anasanjaria

Common Mistake — Processing Big Data in a Transaction was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

This content originally appeared on Level Up Coding - Medium and was authored by Anas Anjaria

Print Share Comment Cite Upload Translate Updates

APA

Anas Anjaria | Sciencx (2022-10-12T17:28:24+00:00) Common Mistake — Processing Big Data in a Transaction. Retrieved from https://www.scien.cx/2022/10/12/common-mistake-processing-big-data-in-a-transaction/

MLA

" » Common Mistake — Processing Big Data in a Transaction." Anas Anjaria | Sciencx - Wednesday October 12, 2022, https://www.scien.cx/2022/10/12/common-mistake-processing-big-data-in-a-transaction/

HARVARD

Anas Anjaria | Sciencx Wednesday October 12, 2022 » Common Mistake — Processing Big Data in a Transaction., viewed ,<https://www.scien.cx/2022/10/12/common-mistake-processing-big-data-in-a-transaction/>

VANCOUVER

Anas Anjaria | Sciencx - » Common Mistake — Processing Big Data in a Transaction. [Internet]. [Accessed ]. Available from: https://www.scien.cx/2022/10/12/common-mistake-processing-big-data-in-a-transaction/

CHICAGO

" » Common Mistake — Processing Big Data in a Transaction." Anas Anjaria | Sciencx - Accessed . https://www.scien.cx/2022/10/12/common-mistake-processing-big-data-in-a-transaction/

IEEE

" » Common Mistake — Processing Big Data in a Transaction." Anas Anjaria | Sciencx [Online]. Available: https://www.scien.cx/2022/10/12/common-mistake-processing-big-data-in-a-transaction/. [Accessed: ]

rf:citation

» Common Mistake — Processing Big Data in a Transaction | Anas Anjaria | Sciencx | https://www.scien.cx/2022/10/12/common-mistake-processing-big-data-in-a-transaction/ |

Please log in to upload a file.

There are no updates yet.
Click the Upload button above to add an update.

You must be logged in to translate posts. Please log in or register.