Event-Driven & Asynchronous Systems

Synchronous communication couples services in time. Asynchronous communication couples them only in contract. The producer publishes an event; the consumer reads it whenever it is ready, retries if it fails, and runs at a different rate than the producer. That decoupling is what lets event-driven systems scale to billions of events per day — and what creates the operational failure modes you have to learn to recognise.

Queue vs Pub/Sub vs Event Streaming — Pick One Deliberately

• Message queue (RabbitMQ, SQS): each message is delivered to one consumer. Used for work distribution: a queue of jobs, workers pull and process. Messages disappear after ack.
• Pub/sub (Redis Pub/Sub, NATS, Google Pub/Sub): each message is delivered to all subscribers. Used for fan-out notifications. Often ephemeral — missed messages are missed.
• Event streaming (Kafka, Kinesis, Pulsar): messages are appended to a durable log; consumers read at their own pace, can replay history, can have many independent groups. The dominant pattern for high-throughput data pipelines.

Kafka in Production

Kafka's mental model: a topic is a named, durable, append-only log split into partitions. Each partition is replicated across brokers (typically 3x). A producer writes records, optionally with a key; the key's hash determines the partition. A consumer group reads the topic; Kafka assigns each partition to one consumer in the group, so partition count caps consumer parallelism.

Three properties to internalise:

1. Ordering is per-partition. Records with the same key land in the same partition and are read in order. Across partitions, ordering is undefined. Choose your key to align with the units you need ordered (e.g. user_id for per-user event ordering).
2. Consumer groups are independent. Two consumer groups reading the same topic do not affect each other — each tracks its own offset. This is the foundation of event-driven architectures: the orders topic feeds a billing pipeline AND a search-indexer AND an audit log, all reading the same stream independently.
3. Replication is for durability, not for read scale. Reads always go to the partition leader. Replicas exist so you can survive broker loss; they do not load-balance reads.

Delivery Guarantees — What Exactly-Once Actually Means

Three levels:

• At-most-once — fire and forget; on failure the message is lost. Acceptable for telemetry where loss is fine.
• At-least-once — the message will be delivered; possibly more than once. The default in most systems. Requires consumer-side idempotency (deduplication via idempotency keys).
• Exactly-once — the message is processed exactly once, end-to-end. Kafka's exactly-once semantics work between Kafka topics; once a message leaves Kafka and writes to an external system, you are back to “at-least-once + idempotency”.

Practical guidance: design for at-least-once with consumer-side idempotency as the default. Reach for exactly-once only when you genuinely cannot make consumers idempotent, and accept the operational cost.

Backpressure

Backpressure is the signal flowing upstream from a saturated consumer to a producer: “slow down, I cannot keep up”. Without it, producers happily fill queues until memory or disk runs out. Mechanisms:

• Bounded queues with blocking enqueue; the producer blocks when the queue is full.
• Reactive streams (Project Reactor, RxJava) with explicit demand signals.
• HTTP/2 flow control built into the protocol.
• Consumer-lag-driven producer throttling: if Kafka consumer lag exceeds a threshold, the producer service intentionally slows.

The opposite anti-pattern: an unbounded in-memory queue that quietly grows until the JVM OOMs. Always bound your queues.

Common Production Failures

• Consumer lag spike: the canonical Kafka alert. A consumer group falls behind the head of the log. Causes: consumer slowed down, partition imbalance, downstream dependency degraded. The metric to watch: kafka_consumer_lag_max per group.
• Rebalance storm: every time a consumer joins or leaves the group, all partition assignments are recomputed and consumers pause. Frequent rebalances kill throughput. Causes: aggressive session timeouts, slow processing exceeding heartbeat, scaling churn. Mitigation: tune session.timeout.ms, heartbeat.interval.ms, and use cooperative rebalance.
• Hot partition: a partition key with skewed traffic (e.g. one big tenant) overloads one broker while others sit idle. Mitigation: salt the key, increase partition count, or use a different partitioning scheme.
• Stuck consumer: a consumer hangs on one bad message and stops making progress. Mitigation: per-message timeouts, dead-letter queue, observability on per-message processing time.

Backpressure Propagation

Self-Check Quiz

1. You have a Kafka topic with 4 partitions and a consumer group of 6 consumers. What happens? (Answer: 4 consumers each own one partition; 2 are idle. Partition count caps consumer parallelism in a group.)
2. Your team wants exactly-once delivery for a billing pipeline. What should you actually build? (Answer: at-least-once + consumer-side idempotency via idempotency keys. Kafka exactly-once works topic-to-topic but not topic-to-database.)
3. Consumer lag is climbing for one group, while another group reading the same topic is fine. What do you check? (Answer: the lagging group's downstream dependency or processing logic. Same topic + different lags = consumer-side issue, not Kafka.)
4. Why is partitioning by random key dangerous for an event stream where order matters per user? (Answer: messages for the same user end up on different partitions; per-user ordering is lost. Key by user_id.)