Om's Brain

6. Message Queue

6 min read

1️⃣ Why Message Queues Exist

Problem Without Queue

Imagine Instagram photo upload flow:

Client → Server → Resize → Filters → Moderation → Response

Everything happens synchronously.

Problems

⚠️ High Latency

User waits for all processing:

  • Resize image
  • Apply filters
  • Run moderation checks

Response may take:

5-10 seconds

Bad UX ❌

⚠️ Fragile System

If one service crashes:

Moderation service fails

Entire upload fails.

Already completed work gets wasted.

⚠️ Bursty Traffic

Example:

Normal traffic: 50 req/sec
Spike traffic: 50,000 req/sec

Servers cannot handle sudden spikes.

Requests fail or timeout.

2️⃣ Solution: Message Queue

Architecture

Client
   ↓
Upload Server
   ↓
Message Queue
   ↓
Worker Pool
   ↓
Image Processing

Flow

Step 1

Upload server stores image.

Step 2

Server pushes message to queue:

"Photo 456 needs processing"

Step 3

Server immediately responds to client:

Upload successful

Step 4

Workers consume messages asynchronously.

3️⃣ What is a Message Queue?

Definition

A message queue is:

A buffer between producers and consumers

Components

ComponentResponsibility
ProducerCreates work
QueueStores work
ConsumerProcesses work

Example

RoleExample
ProducerUpload service
QueueKafka/SQS
ConsumerImage processing worker

4️⃣ Core Benefit: Decoupling

Producer and consumer do NOT know about each other.

Benefits

✅ Independent Scaling

More uploads?
Scale producers.
 
Heavy processing?
Scale consumers.

✅ Better Reliability

Consumer crashes?

Queue still stores message.

✅ Async Processing

Slow tasks move to background.

5️⃣ Kitchen Analogy 🍽️

Real World Mapping

KitchenQueue System
WaiterProducer
Ticket railQueue
CookConsumer

Waiter places order and leaves.

Cook processes later.

Exactly how queues work.

6️⃣ Acknowledgements (ACK)

Problem

What if worker crashes while processing?

Without ACK:

Message gets lost forever ❌

Solution

Consumer must explicitly acknowledge:

ACK = "Processing completed"

Only then queue deletes message.

Flow

Consumer receives message
       ↓
Processes message
       ↓
Sends ACK
       ↓
Queue deletes message

7️⃣ Visibility Timeout

Problem

While Worker A processes message:

Can Worker B also process it?

That causes duplicate processing.

Solution (SQS)

Message becomes:

Invisible for 30 seconds

Other consumers cannot see it.

If worker crashes:

Visibility timeout expires

Message becomes visible again.

8️⃣ Delivery Guarantees

Very important interview topic ⚠️

1. At Least Once Delivery ✅ (Most Common)

Meaning

Every message gets delivered:

At least once

But duplicates may happen.

Requirement

Consumers must be:

Idempotent

Idempotent Meaning

Running same operation twice gives same result.

Good Example

Set profile photo = photo_5

Running twice:

Still photo_5

Safe ✅

Bad Example

Increment post count by 1

Running twice:

+2 increment ❌

Better Design

Instead of:

Increment count by 1

Use:

Set count = 54

Now operation becomes idempotent.

Interview Recommendation ⭐

Usually say:

Use at-least-once delivery
with idempotent consumers

Safest practical answer.

2. At Most Once Delivery

Meaning

Message may be lost.

But never duplicated.

Usage

Useful for:

  • Analytics
  • Metrics
  • Logging

Where small data loss is acceptable.

3. Exactly Once Delivery

Meaning

Message processed exactly one time.

Reality

Very difficult in distributed systems.

Complex and expensive.

Interview Advice

Avoid claiming:

Exactly once delivery

Unless you can deeply explain implementation.

9️⃣ When Should You Use Message Queues?

1. Async Work

User does NOT need immediate result.

Examples:

  • Sending emails
  • Video processing
  • Report generation
  • Notifications

2. Bursty Traffic

Queue absorbs spikes.

Acts as:

Traffic buffer

3. Decoupling

Different scaling requirements.

Example:

ServiceRequirement
Upload APILightweight
ML ProcessingGPU heavy

4. Reliability

Queue stores work if downstream service fails.

🔟 When NOT to Use Queue

Synchronous Workloads ❌

If requirement is:

< 500ms response time

Queue may violate latency requirements.

Queues are mainly for:

Async background processing

1️⃣1️⃣ Queue Scaling

Problem

Single queue has throughput limit.

Solution: Partitioning

Split queue into multiple partitions.

Partition 1
Partition 2
Partition 3

Each processed independently.

Benefit

Parallel consumption.

Higher throughput.

1️⃣2️⃣ Consumer Groups

Definition

Pool of consumers sharing partitions.

Example

6 partitions
3 consumers

Each consumer handles:

2 partitions

Important Rule ⚠️

Consumers <= Partitions

Extra consumers stay idle.

1️⃣3️⃣ Partition Key

Extremely important topic.

Purpose

Determines:

Which partition receives message

Goals

GoalWhy Important
OrderingRelated messages stay together
DistributionLoad spreads evenly

Banking Example

Operations:

Deposit $100
Withdraw $50

Must happen in order.

Correct Partition Key

account_id

Both operations go to same partition.

Ordering preserved ✅

1️⃣4️⃣ Hot Partition Problem

Bad Partition Key Example

Partition by city

Result:

New York overloaded
Small city idle

This is:

Hot partition

Better Key

ride_id

More evenly distributed.

1️⃣5️⃣ Back Pressure

Problem

Producers faster than consumers.

Example:

Incoming = 300 msg/sec
Processing = 200 msg/sec

Queue grows forever.

Important Concept

Queue does NOT solve capacity problem.

It only delays it.

Solutions

✅ Autoscaling

Add more consumers.

✅ More Partitions

Increase parallelism.

✅ Back Pressure

Slow producers down.

Example:

429 Too Many Requests

✅ Monitoring

Track:

  • Queue depth
  • Consumer lag
  • Processing time

1️⃣6️⃣ Poison Messages ☠️

Problem

Some messages always fail.

Example:

Corrupted image

Retries forever.

Consumes resources endlessly.

Solution: Dead Letter Queue (DLQ)

After max retries:

Move message → DLQ

Benefits

  • Main queue continues
  • Failed messages isolated
  • Easier debugging

Interview Tip ⭐

Mention DLQ proactively.

Strong senior-level signal.

1️⃣7️⃣ Durability & Fault Tolerance

What if Queue Crashes?

Modern queues:

  • Persist messages to disk
  • Replicate across brokers

Kafka Feature

Messages retained for configurable duration.

Example:

1 day
1 week
Forever

1️⃣8️⃣ Message Replay

Huge Kafka advantage.

Consumers can:

Re-read old messages

Useful for:

  • Bug fixes
  • Reprocessing
  • Recovery

1️⃣9️⃣ Popular Queue Technologies

Apache Kafka

Best for:

  • High throughput
  • Distributed systems
  • Streaming
  • Replay support

Features

FeatureSupported
Partitioning✅
Consumer groups✅
Replay✅
Durability✅

Amazon SQS

AWS managed queue service.

Types

Queue TypeCharacteristics
Standard QueueHigh throughput
FIFO QueueStrict ordering

RabbitMQ

Traditional message broker.

Good for:

  • Complex routing
  • Enterprise workflows

2️⃣0️⃣ Kafka vs SQS vs RabbitMQ

FeatureKafkaSQSRabbitMQ
Managed❌✅❌
Replay Support✅❌Limited
OrderingPer partitionFIFO onlyQueue level
ThroughputVery highHighMedium
ComplexityHighLowMedium
Best Use CaseStreamingSimple async jobsRouting workflows

2️⃣1️⃣ Common Interview Deep Dives

Interviewers LOVE these ⚠️

Be ready for:

Scaling

  • Partitioning
  • Consumer groups

Ordering

  • Partition keys
  • FIFO guarantees

Reliability

  • ACKs
  • Retries
  • DLQ

Capacity

  • Back pressure
  • Autoscaling

Fault Tolerance

  • Replication
  • Persistence

2️⃣2️⃣ Interview Cheat Sheet 🧠

Best Default Answers

QuestionRecommended Answer
Delivery guarantee?At least once
Duplicate handling?Idempotent consumers
Failed messages?DLQ
Scaling?Partitioning + consumer groups
Traffic spikes?Queue buffering
Ordering?Partition key
Queue durability?Replication + disk persistence

2️⃣3️⃣ Important Keywords

Producer
Consumer
Partition
Consumer Group
ACK
Visibility Timeout
Idempotency
DLQ
Back Pressure
Replay
Hot Partition
At-least-once Delivery

2️⃣4️⃣ Final Summary

Message Queues Help With

✅ Async processing ✅ Traffic spikes ✅ Reliability ✅ Decoupling ✅ Scalability

Core Tradeoff

Higher reliability
vs
Higher complexity

Golden Interview Line ⭐

I would use at-least-once delivery
with idempotent consumers,
partitioning for scalability,
and DLQ for failed messages.

Mindmap

Graph View