Om's Brain

6. Message Queue

5 min read

1️⃣ Why Message Queues Exist

Problem Without Queue

Imagine Instagram photo upload flow:

Client → Server → Resize → Filters → Moderation → Response

Everything happens synchronously.

Problems

  • ⚠️ High Latency
    • User waits for all processing:
    • Resize image
    • Apply filters
    • Run moderation checks
    • Response may take: 5-10 secondsBad UX ❌
  • ⚠️ Fragile System
    • If one service crashes: Moderation service fails
    • Entire upload fails.
    • Already completed work gets wasted.
  • ⚠️ Bursty Traffic
    • Example:
    Normal traffic: 50 req/sec
Spike traffic: 50,000 req/sec
    • Servers cannot handle sudden spikes.
    • Requests fail or timeout.

2️⃣ Solution: Message Queue

Architecture

Client -> Upload Server -> ⭐ [Message Queue] ⭐ -> Worker Pool -> Image Processing

Flow

  • Upload server stores image.
  • Server pushes message to queue: Photo 456 needs processing
  • Server immediately responds to client: Upload successful
  • Workers consume messages asynchronously.

4️⃣ Core Benefit: Decoupling

Producer and consumer do NOT know about each other.

Benefits

✅ Independent Scaling

More uploads?
Scale producers.
 
Heavy processing?
Scale consumers.

✅ Better Reliability

  • Consumer crashes?
  • Queue still stores message.

✅ Async Processing

Slow tasks move to background.

Exmaple:

  • Waiter places order and leaves.
  • Cook processes later.
  • Exactly how queues work.

6️⃣ Acknowledgements (ACK)

Problem

What if worker crashes while processing?

Without ACK:

Message gets lost forever ❌

Solution

Consumer must explicitly acknowledge:

ACK = "Processing completed"

Only then queue deletes message.

Flow

Consumer receives message
       ↓
Processes message
       ↓
Sends ACK
       ↓
Queue deletes message

7️⃣ Visibility Timeout

Problem

While Worker A processes message: Can Worker B also process it? That causes duplicate processing.

Solution (SQS)

Message becomes: Invisible for 30 seconds Other consumers cannot see it. If worker crashes: Visibility timeout expires Message becomes visible again.

8️⃣ Delivery Guarantees

Very important interview topic ⚠️

1. At Least Once Delivery ✅ (>=1) (Most Common)

  • Every message gets delivered: At least once
  • But duplicates may happen.
  • Requirement:
    • Consumers must be → Idempotent (Running same operation twice gives same result
      • Good Ex. Set profile photo = photo_5 ⇒ Running twice ⇒ Still photo_5
      • Bad Ex. Send 10 rs to OM ⇒ running twice total 20rs
  • Interview Recommendation ⭐
Use at-least-once delivery
with idempotent consumers

2. At Most Once Delivery - 1 or 0

  • fire & forget - Message may be lost. But never duplicated.
    • Best case - Atmost one guy processed it
    • At worst - nobody processed it
  • Where small data loss is acceptable like below
  • Useful for:
    • Analytics
    • Metrics
    • Logging

3. Exactly Once Delivery - 1

  • Meaning - Message processed exactly one time.
  • Reality: Very difficult in distributed systems. (Complex and expensive.)
  • Interview Advice
    • Avoid claiming: Exactly once delivery
    • Unless you can deeply explain implementation.

9️⃣ When Should You Use Message Queues?

Ask a Question - the user need the result of this operation right now or can he wait a little bit

🔟 When NOT to Use Queue

Synchronous Workloads ❌

If requirement is: < 500ms response time - Queue may violate latency requirements. Queues are mainly for: Async background processing

1️⃣1️⃣ Queue Scaling

Problem

Single queue has throughput limit.

Solution: Partitioning

Split queue into multiple partitions.

Partition 1
Partition 2
Partition 3

Each processed independently.

Benefit

Parallel consumption. Higher throughput.

1️⃣2️⃣ Consumer Groups

Definition

Pool of consumers sharing partitions.

Example

6 partitions
3 consumers

Each consumer handles:

2 partitions

Important Rule ⚠️

Consumers <= Partitions

Extra consumers stay idle.

1️⃣3️⃣ Partition Key

Extremely important topic.

Purpose

Determines:

Which partition receives message

Goals

GoalWhy Important
OrderingRelated messages stay together
DistributionLoad spreads evenly

Banking Example

Operations:

Deposit $100
Withdraw $50

Must happen in order.

Correct Partition Key

account_id

Both operations go to same partition.

Ordering preserved ✅

1️⃣4️⃣ Hot Partition Problem

Bad Partition Key Example

Partition by city

Result:

New York overloaded
Small city idle

This is:

Hot partition

Better Key

ride_id

More evenly distributed.

1️⃣5️⃣ Back Pressure

Problem

Producers faster than consumers.

Example:

Incoming = 300 msg/sec
Processing = 200 msg/sec

Queue grows forever.

Important Concept

Queue does NOT solve capacity problem.

It only delays it.

Solutions

✅ Autoscaling

Add more consumers.

✅ More Partitions

Increase parallelism.

✅ Back Pressure

Slow producers down.

Example:

429 Too Many Requests

✅ Monitoring

Track:

  • Queue depth
  • Consumer lag
  • Processing time

1️⃣6️⃣ Poison Messages ☠️

Problem

Some messages always fail.

Example:

Corrupted image

Retries forever.

Consumes resources endlessly.

Solution: Dead Letter Queue (DLQ)

After max retries:

Move message → DLQ

Benefits

  • Main queue continues
  • Failed messages isolated
  • Easier debugging

Interview Tip ⭐

Mention DLQ proactively.

Strong senior-level signal.

1️⃣7️⃣ Durability & Fault Tolerance

What if Queue Crashes?

Modern queues:

  • Persist messages to disk
  • Replicate across brokers

Kafka Feature

Messages retained for configurable duration.

Example:

1 day
1 week
Forever

1️⃣8️⃣ Message Replay

Huge Kafka advantage.

Consumers can:

Re-read old messages

Useful for:

  • Bug fixes
  • Reprocessing
  • Recovery

1️⃣9️⃣ Popular Queue Technologies

Apache Kafka

Best for:

  • High throughput
  • Distributed systems
  • Streaming
  • Replay support

Features

FeatureSupported
Partitioning✅
Consumer groups✅
Replay✅
Durability✅

Amazon SQS

AWS managed queue service.

Types

Queue TypeCharacteristics
Standard QueueHigh throughput
FIFO QueueStrict ordering

RabbitMQ

Traditional message broker.

Good for:

  • Complex routing
  • Enterprise workflows

2️⃣0️⃣ Kafka vs SQS vs RabbitMQ

FeatureKafkaSQSRabbitMQ
Managed❌✅❌
Replay Support✅❌Limited
OrderingPer partitionFIFO onlyQueue level
ThroughputVery highHighMedium
ComplexityHighLowMedium
Best Use CaseStreamingSimple async jobsRouting workflows

2️⃣1️⃣ Common Interview Deep Dives

Interviewers LOVE these ⚠️

Be ready for:

Scaling

  • Partitioning
  • Consumer groups

Ordering

  • Partition keys
  • FIFO guarantees

Reliability

  • ACKs
  • Retries
  • DLQ

Capacity

  • Back pressure
  • Autoscaling

Fault Tolerance

  • Replication
  • Persistence

2️⃣2️⃣ Interview Cheat Sheet 🧠

Best Default Answers

QuestionRecommended Answer
Delivery guarantee?At least once
Duplicate handling?Idempotent consumers
Failed messages?DLQ
Scaling?Partitioning + consumer groups
Traffic spikes?Queue buffering
Ordering?Partition key
Queue durability?Replication + disk persistence

2️⃣3️⃣ Important Keywords

Producer
Consumer
Partition
Consumer Group
ACK
Visibility Timeout
Idempotency
DLQ
Back Pressure
Replay
Hot Partition
At-least-once Delivery

2️⃣4️⃣ Final Summary

Message Queues Help With

✅ Async processing ✅ Traffic spikes ✅ Reliability ✅ Decoupling ✅ Scalability

Core Tradeoff

Higher reliability
vs
Higher complexity

Golden Interview Line ⭐

I would use at-least-once delivery
with idempotent consumers,
partitioning for scalability,
and DLQ for failed messages.

Mindmap

Graph View