This content originally appeared on Bits and Pieces - Medium and was authored by Matteo Pampana
How to deal with data replication in a microservices architecture with PostgreSQL and event sourcing
If you are working with a microservices architecture you may have encountered the need of duplicating a large table between different services.
In theory, this result can be obtained using event sourcing. You publish all the transitions of your entity in a topic, following the order of modifications, and then you can reconstruct the state of that entity in a different service.
But let’s say we are dealing with an entity that has a huge state. You can imagine the accounts on a platform like Amazon. How many accounts are we dealing with? How many states do we need to read and how many I/O operations do we need to perform on our DB? Moreover, if you are running your platform on AWS you may have limited credits on your RDS instance based on your instance type and your storage size.
You can adopt a mixed solution in this case:
- You take a photograph of the state at a certain point in time;
- You start reconstructing the state from that point in time.
Taking a photograph of the state of a certain entity means replicating the data, at time T, from the database of the service that owns that entity. You actually dump the data and import it into the new service database.
Today, I have discovered a way to perform this operation quickly and without the need of downloading an additional file. In fact, most solutions you can find on the web may suggest you dump the data in a .csv file, and then import this file into the new database.
The following command, instead, will dump the data and redirect the output directly into your new database:
pg_dump -a -t <table_name> <full_source_db_uri> | psql -v ON_ERROR_STOP=1 <full_destination_db_uri>
This is fast — you are not creating any intermediate file — and optimal, since you are performing only the strictly necessary writing operations on the database.
Unfortunately, you cannot stop here. The state of the entity is in continuous evolution and you have just taken a photograph at a certain point in time. You are not consistent with the rest of the system.
Now you want to evolve the state of the entity to keep pace with the rest of the microservices. You will never reach perfect consistency (see CAP Theorem) but eventually, you can be consistent. And you reach this goal by exploiting the above-mentioned event sourcing.
But at this point, you have a great advantage: you can skip all the events produced before the time in which you have taken the photograph of the state. Actually, you may be conservative and start consuming events with a minimum overlapping period to be sure you are not missing any updates of the entity’s state. Anyway, you have potentially skipped millions of events.
Note: As an aside, if you’re using event sourcing to replicate data between microservices, an open-source toolchain like Bit can be especially helpful in managing and sharing the code components needed for this architecture. By providing a platform for teams to collaborate on building and sharing reusable code components, Bit can help streamline the development process and ensure consistency across different services.
To conclude, this technique can boost the kick-off time of your microservice since you are skipping tons of events and potentially avoiding millions of I/O operations on your database.
Build Apps with reusable components, just like Lego
Bit’s open-source tool help 250,000+ devs to build apps with components.
Turn any UI, feature, or page into a reusable component — and share it across your applications. It’s easier to collaborate and build faster.
Split apps into components to make app development easier, and enjoy the best experience for the workflows you want:
→ Micro-Frontends
→ Design System
→ Code-Sharing and reuse
→ Monorepo
Learn more
- How We Build Micro Frontends
- How we Build a Component Design System
- How to reuse React components across your projects
- 5 Ways to Build a React Monorepo
- How to Create a Composable React App with Bit
Boost Your Microservices Data Initialization was originally published in Bits and Pieces on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Bits and Pieces - Medium and was authored by Matteo Pampana
Matteo Pampana | Sciencx (2023-02-17T13:21:30+00:00) Boost Your Microservices Data Initialization. Retrieved from https://www.scien.cx/2023/02/17/boost-your-microservices-data-initialization/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.