This content originally appeared on DEV Community and was authored by adriens
🪝 Teaser
Did you ever find yourself in a situation where :
- Team 🇦 pushes data in a given database (let's say MySQL,...) with its very own custom software
-
Team 🇧 needs to get these data changes (
INSERT,UPDATE,DELETE) as events so they can put them in let's say... an another database (MariaDB, PostgreSQL,...) - Base software cannot be changed : you have to "deal with it"
Eg, team B's motivation maybe to achieve datascience, RealTime Analytics, store in a datalake,...
👉 This blog post is dedicated to this case... and surprisingly : open source solutions do exist to achieve this magic!
🤔 About the "why"
Debezium Project's "why" is pretty straightforward :
"Turn your databases into change event streams"
... even for "legacy" like systems:
// Detect dark theme var iframe = document.getElementById('tweet-1808144012127171020-834'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1808144012127171020&theme=dark" }
👂 How it does NOT work (why it's awesome)
The key thing here to remind is that Debezium does NOT act as a proxy in front of the database, and that's the most elegant part.
The key point is that Debezium is literally listening to database changes, whatever you call them :
-
MySql
binlog - MariaDB
-
PostgreSQL
WALs for -
Oracle
archivelog - ...
, then send these events in a common standard format into Kafka messages... waiting to be used later by one or many consumers.
🪄 How it works
The magic resides in the following workflow :
-
Capture data changes at the database level (
WALfor postgres, archivelogs, whatever you call them...) - Send/Stream events to Kafka
- Consume Kafka events so they they can be pushed to any third party data service 3'. JDBC : for example "consume events from multiple source topics, and then rite those events to a relational database by using a JDBC driver."
🍿 Demo from scratch
Below the live demo I was able to do, from scratch, but by following default instructions for a MySQL instance :
🔭 Going further
This content originally appeared on DEV Community and was authored by adriens
adriens | Sciencx (2024-07-22T20:58:14+00:00) 🪄 Debezium: the magic behind data capture & async replication (for free). Retrieved from https://www.scien.cx/2024/07/22/%f0%9f%aa%84-debezium-the-magic-behind-data-capture-async-replication-for-free/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.