1. Inverted Index

---

1️⃣ What is CDC (Change Data Capture)?

CDC is a mechanism to capture and stream database changes (INSERT, UPDATE, DELETE) in real time, typically by reading transaction logs, to keep systems synchronized efficiently.

👉 Instead of repeatedly scanning the whole database, CDC only captures what changed, making systems more efficient.

---

2️⃣ Why do we use CDC?

📌 Key Use Cases

• Real-time data pipelines
  • Send DB changes to Kafka, analytics systems, etc.
• Microservices sync
  • Keep multiple services/databases in sync
• Data warehousing
  • Replicate production DB → data warehouse
• Audit logs
  • Track who changed what and when

---

3️⃣ How CDC Works (Conceptually)

When a change happens in DB:

User updates row → DB logs change → CDC captures change → sends to consumers

🔍 Common ways to implement CDC

1. Log-based CDC (⭐ Best)

• Reads database transaction logs
• Example tools:
  • Debezium
  • Apache Kafka (with connectors)
• 📌 Pros:
  • No performance impact on DB
  • Captures all changes reliably

---

2. Trigger-based CDC

• DB triggers fire on INSERT/UPDATE/DELETE
• Store changes in another table
• 📌 Pros:
  • Easy to implement
• ⚠️ Cons:
  • Slows DB
  • Hard to scale

---

3. Timestamp / Polling-based CDC

• Query rows where updated_at > last_checked_time
• 📌 Pros:
  • Simple
• ⚠️ Cons:
  • Misses deletes
  • Not truly real-time

---

4️⃣ Simple Example

Table:

users
id | name | updated_at

Change:

UPDATE users SET name = "Omi" WHERE id = 1;

CDC Output Event:

{
  "operation": "UPDATE",
  "before": { "id": 1, "name": "Om" },
  "after": { "id": 1, "name": "Omi" },
  "timestamp": "2026-04-14T04:00:00Z"
}

---

5️⃣ Where CDC fits in System Design

Architecture flow:

                ┌──────────────────────┐
                │      Application     │
                │ (User / Service API)│
                └──────────┬──────────┘
                           │
                           ▼
                ┌──────────────────────┐
                │      Database        │
                │ (MySQL / Postgres)  │
                └──────────┬──────────┘
                           │
        (Writes: INSERT / UPDATE / DELETE)
                           │
                           ▼
                ┌──────────────────────┐
                │  Transaction Logs    │
                │ (Binlog / WAL)       │
                └──────────┬──────────┘
                           │
                 (Log-based CDC reads)
                           │
                           ▼
                ┌──────────────────────┐
                │   CDC Tool           │
                │  (Debezium)          │
                └──────────┬──────────┘
                           │
                (Convert to events)
                           │
                           ▼
                ┌──────────────────────┐
                │   Message Broker     │
                │     (Kafka)          │
                └──────────┬──────────┘
                           │
        ┌───────────────┬───────────────┬───────────────┐
        ▼               ▼               ▼               ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│   Analytics  │ │   Cache      │ │ Search Index │ │ Notifications│
│  (Warehouse) │ │  (Redis)     │ │(Elastic)     │ │   Service    │
└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘

---

6️⃣ Key Benefits

• Real-time data sync
• Efficient (no full table scans)
• Event-driven architecture
• Enables streaming systems

---