Queue, Topic, and Stream

Somebody says we will put it on a queue, everyone nods, and the meeting moves on. It is one of the few phrases in this industry that reliably produces agreement, and it does so because each person hears the thing they already had in mind. One is picturing work handed out to whichever worker is free. One is picturing an event announced to whoever cares. One is picturing a durable record of what happened, readable again next year.

Those are three data structures with three different guarantees, and only the first is a queue. The other two are a topic and a log, and the confusion survives because a broker will sell you all three through an interface that looks the same from the sending side. You publish a message. The difference is entirely in what the receiving side may do, and nobody in the room represents the receiving side.

A queue distributes work. Many consumers compete for the same messages, each message goes to exactly one of them, and when that consumer acknowledges it the broker deletes it. The value on offer is load balancing: add a worker and throughput rises, lose one and its unacknowledged messages go to somebody else. The question a queue answers is who does this job, and once the job is done the question is closed.

A topic copies. Every subscriber gets its own copy of every message, so adding one takes nothing away from the others. The value on offer is decoupling: the publisher does not know who is listening, which is what lets a consumer be added without touching the producer. Classic pub/sub is also ephemeral. A subscriber that was not connected when the message went out never sees it, unless a durable subscription was arranged in advance.

A stream keeps the record. Messages are appended to an ordered log and reading does not remove them. A consumer holds an offset, moves it forward as it goes, and can move it back. Consumers do not interfere with each other because each has its own position, and a new one can attach at the beginning, at a timestamp, or at the end. The log is the system of record for what happened, rather than a conveyor for what needs doing.

The whole difference is who owns the position. On a queue the broker owns it, and the acknowledgment is how a consumer tells the broker to advance and forget. On a stream the consumer owns it, and the broker is an ordered file with no opinion about who has read what. Every other property follows. Replay, fan-out, ordering, the shape of failure: all downstream of one decision about where the pointer lives.

Six months after a system ships, somebody wants the same events again. They want to build a report, or a search index, or a second service that reacts to orders the way the first one reacts to payments. On a stream, this is an afternoon: point a new consumer at offset zero and let it run. On a queue it is not expensive, it is impossible, because the messages were deleted at acknowledgment time and the only surviving trace is whatever the original consumer happened to write down.

So here is the sentence the broker decision actually turns on, and it is not about a broker. Are these messages instructions or are they a record? An instruction is finished when it has been carried out, and keeping it afterward buys you nothing. A record is the thing you go back to when a question arrives that nobody thought of when it was written, and a system that discards its records can only ever answer questions somebody anticipated.

Global order is not on offer. Every system that scales ordering scopes it to a subset. Kafka orders within a partition, SQS FIFO within a message group, Azure Service Bus within a session. These are the same idea wearing three names: a key that selects a lane, and a promise that holds inside the lane and says nothing across lanes. There is no product that will order everything, because ordering everything means one writer and one reader.

The design act is choosing the key. Per customer, per account, per device, per order. What a business almost always needs is that events about one thing arrive in order relative to each other, and it almost never needs that an event about one customer is ordered against an event about a different customer. Getting this right converts an unscalable requirement into a scalable one, and it is done in a single line of producer configuration.

Start with the failure that generates it. A producer sends a message, the broker writes it down, and the acknowledgment is lost coming back: the network dropped it, or the broker died between committing and replying. The producer now knows nothing. Resend and there may be two copies; do not and there may be none. No protocol removes that choice, because the problem is not a missing feature. It is that two parties cannot agree about an event across a channel that can fail, and every messaging system has to pick a side of it.

Picking loss is at-most-once: fire, do not wait, accept that failures drop messages. Picking duplication is at-least-once: retry until acknowledged, accept that some arrive twice. Essentially everything serious runs at-least-once, because losing a payment notification is worse than processing one twice. Systems that claim to be above the choice have not escaped it. They have put deduplication on top of it, inside a boundary they define.

The boundaries are worth reading carefully, because each is a window and each has an edge. Kafka's idempotent producer stamps a producer identifier and a sequence number on each record so the broker can discard a retransmission of something it already has; Kafka 3.0, in September, turned that on by default along with acks=all, which means the strongest producer guarantee is now what you get for doing nothing. Amazon's FIFO queues deduplicate on an identifier for 5 minutes; send the same thing at minute six and it is a new message. Azure Service Bus keeps a window of message identifiers that defaults to 10 minutes and can be stretched to 7 days, which costs throughput in proportion, because every send is checked against everything remembered.

Kafka's transactions go further and are the strongest thing on offer: a producer can write to several partitions and commit its own consumer offsets in one atomic operation, so a loop that reads from a topic, computes, and writes to another topic either happens completely or not at all. That is a real exactly-once guarantee and I do not want to undersell it. Read the sentence again for the boundary, though. Both ends are Kafka. The atomicity comes from the offset commit and the output write being the same commit, in the same system.

Which is where the promise breaks, and it breaks on the only part anybody cares about. Charge a card, send an email, write to a database the broker does not manage, call a partner's API: none of those participate in the transaction, none of them can be rolled back by an offset, and a consumer that crashes after the side effect and before the commit will do it again. So the honest statement is narrow. Exactly-once is available between two points inside one system. End to end, across the effects your business is actually made of, it is a property of your consumer, and a team promising it in a design review is promising something they will have to build themselves.

Give the work an identity the producer chose. Not the broker's message identifier, which changes when an upstream component retries and hands you a fresh message carrying the same meaning. A key derived from the business event survives that: the order identifier plus the step, the invoice number plus the action. If two messages mean the same thing they must carry the same key, and deciding what that means is the design work.

Make the effect conditional on the key. A unique constraint, an upsert, a compare-and-set, an insert that fails on a duplicate. The deduplication then lives in the database that holds the effect, which is the only place that can commit both the check and the change together. A separate deduplication cache is a second system that can disagree with the first, and it will, at exactly the moment the first system is under load.

The consumer that falls permanently behind fails differently in each model, and it is worth knowing which failure you signed up for. On a queue you get one number, the depth, and it grows. It will not tell you which consumer is slow or how far behind any reader is, only that the pile is larger and that the broker will eventually push back on producers. Loud, and coarse.

On a stream you get the opposite: lag is per consumer and measured exactly, in offsets or in seconds, so you know precisely who is behind. What you also get is a deadline, because the far end of the log is being deleted on a schedule that has nothing to do with your consumer's progress. When the reader's position falls off that end, it does not stop and it does not alarm. It jumps forward to the oldest surviving record and carries on, and the messages in between are gone without anything in the pipeline having failed.

I should be careful not to turn this into an argument for streams, because it is not one, and the more common mistake now is the opposite of the one I have described. Most work is just work: distribute it, do it, forget it, and a queue is the right shape for that. A log is a commitment to a lifetime, an offset somebody has to own, and a failure mode where nothing breaks and data disappears anyway. Reaching for one because it is more capable is how a team ends up operating a distributed log to run a job queue.

What I would take into the meeting is one question, and it does not mention a product. Six months from now, when somebody asks for these same events again, what is the answer? If the answer is that we would write a script against the source database and reconstruct approximately what happened, then the messages were never a record of anything. They were a way of asking somebody to do something, which is a perfectly good thing for them to be, and asking is not the same as remembering.