Message Queues & Pub/Sub

Overview

Message queues and Pub/Sub systems decouple services by allowing producers to send messages without knowing who will consume them, and consumers to receive messages without knowing who sent them. This enables asynchronous processing, load levelling, and resilience, if a consumer is down, messages accumulate and are processed when it recovers.

Origin

IBM MQ (1993) was among the first enterprise message brokers. RabbitMQ (2007, AMQP), Apache Kafka (LinkedIn, 2011), and Google Pub/Sub established modern patterns. Kafka's log-based approach (messages retained indefinitely, consumers track their position) fundamentally changed the pattern.

Examples

# Producer: enqueue work without waiting for it
class OrdersController < ApplicationController
  def create
    order = Order.create!(order_params)
    # Fire and forget, returns immediately
    ProcessOrderJob.perform_later(order.id)
    render json: order, status: :created
  end
end

# Consumer: runs in a separate process pool
class ProcessOrderJob < ApplicationJob
  queue_as :orders
  retry_on ExternalServiceError, wait: :exponentially_longer, attempts: 5

  def perform(order_id)
    order = Order.find(order_id)
    PaymentService.charge(order)
    InventoryService.deduct(order.items)
    OrderMailer.confirmation(order).deliver_now
    order.update!(status: :confirmed)
  end
end

import { Kafka } from 'kafkajs'

const kafka    = new Kafka({ brokers: ['kafka:9092'] })
const producer = kafka.producer()
const consumer = kafka.consumer({ groupId: 'order-service' })

// Producer (Order Service)
await producer.send({
  topic: 'order.placed',
  messages: [{ key: order.id, value: JSON.stringify(order) }],
})

// Consumer (Fulfillment Service), independent, different team, different deployment
await consumer.subscribe({ topic: 'order.placed', fromBeginning: false })
await consumer.run({
  eachMessage: async ({ message }) => {
    const order = JSON.parse(message.value.toString())
    await fulfillmentService.schedule(order)
  },
})

Use Cases

• 01Decoupling order placement from downstream processing (payment, fulfilment, notifications)
• 02Load levelling: absorb traffic spikes without overloading downstream services
• 03Fan-out: one event consumed by multiple independent services (billing AND shipping AND analytics)
• 04Durable retry: failed jobs are retried automatically; work is not lost if a service crashes

When Not to Use

• //Operations requiring immediate results returned to the caller, asynchrony is incompatible with synchronous request-response
• //When the operational overhead of a broker exceeds the complexity it solves (use in-process queues for simple cases)
• //When exactly-once delivery is required, most brokers guarantee at-least-once; idempotent consumers are necessary

Technical Notes

• At-least-once delivery means messages may be delivered more than once under failure. Design consumers to be idempotent: processing the same message twice must have the same effect as once
• Dead-letter queues capture messages that fail all retries, monitor them; they indicate systemic bugs
• Kafka topics are partitioned logs. Message ordering is guaranteed within a partition. Use the same partition key (e.g., customer ID) for events that must be processed in order
• Back-pressure: if consumers fall behind producers, the queue grows. Design consumers to scale horizontally and monitor queue depth as a key operational metric