The Webhook Endpoint Nobody Reviewed

The ticket says to accept callbacks from the payment provider, and it takes about an hour. Add a route, parse the JSON, switch on the event type, update a row, return 200. It works the first time you test it, which is unusual enough to feel like a good omen. What you have built is a publicly reachable endpoint with no authentication in front of it that writes to your database whenever a stranger posts to it.

Nobody describes it that way, because the requests are coming from a company you have a contract with, and the contract does a great deal of quiet work in everyone's head. It does none on the wire. HTTP carries no memory of the agreement. Anything that can resolve your hostname can send a request of exactly the same shape, and the handler that trusts the body has no way at all to tell the two apart.

The defense people reach for first is that nobody knows the URL, and that is not a defense. The path is in your source repository, your deployment config, your vendor dashboard, your access logs and the ticket where somebody pasted it. Even if it were secret, a secret you cannot rotate without opening a support case with a vendor is not a credential, it is a rumor with good uptime.

So here is the sentence I would want in front of anyone approving the integration, and it is not a sentence about payments. This is a public, unauthenticated, externally triggered write path into your system, held to a latency budget you did not set and a retry policy you cannot change. Every clause there is a design decision. Nearly all of them get made by omission, in the hour it took to add the route.

There is no signature check at all. The handler trusts the shape of the body: it looks like an event, it has the right fields, the amount is plausible. None of that is authentication. Anyone who learns the URL can post an event saying an invoice was paid, and your system will believe it, because believing the body is the only thing the handler knows how to do.

There is a signature with nothing about time in it. The second version adds verification, and it still accepts a request captured a year ago. A signature proves who wrote the bytes. It says nothing about when, so unless freshness is part of what was signed, a valid signature is a token that can be replayed for as long as the secret lives, which is to say indefinitely.

There is no idempotency key. Delivery from every vendor I have integrated with is at-least-once by design, because the alternative is losing events when your endpoint is briefly unreachable, and no vendor is going to choose that. So duplicates are not a fault condition. They are the contract, and without a key to recognize them by, the second delivery is a second side effect.

Handlers apply deltas instead of setting state. Increment a counter, append a row, send an email, charge a card. Run twice, and the state is wrong in a way that no error appears for. A retry-safe handler is one whose second run changes nothing, and the way to get there is to write the resulting state rather than the difference, which is a change to how the handler is written and not a wrapper around it.

The work is done before the response. The handler does everything inline and returns 200 at the end. On an ordinary day this is fine. On the day your database is slow the response misses the vendor's timeout, the vendor records a failure and retries, the retry arrives while the first one is still running, and the load you could not handle has just been doubled by the mechanism designed to help you.

Stripe sends a Stripe-Signature header carrying a timestamp as t= and one or more signatures as v1=. The signed material is the timestamp, a literal dot, and the raw request body, hashed with HMAC and SHA-256 under a secret specific to that endpoint. Putting the timestamp inside the signed material is the whole trick, because it cannot be edited without invalidating the signature, and the official libraries reject anything more than five minutes old by default.

Shopify signs the raw body with the app's client secret, base64 encoded, in X-Shopify-Hmac-Sha256. Twilio does something else again: HMAC with SHA-1, keyed on the account auth token, over the full request URL with its POST parameters sorted alphabetically and concatenated onto the end, and for a JSON body a bodySHA256 parameter added to the query string instead. Three vendors, three schemes, and no chance of writing the verification once.

Two things fall out of the Twilio design that are worth stating plainly. The signed material includes the URL, so any proxy or load balancer that rewrites the path, adds a trailing slash or changes the port breaks verification in a way that looks like an attack and is not. And the key is the account auth token, which is also the credential that authorizes your outbound API calls, so one secret both verifies what arrives and spends money on your behalf.

The trap all three share is the body. Your framework parses JSON before your code runs, and if you verify a re-serialized copy the key order and the whitespace have already changed, so the signature fails. The fix people reach for is to sign the re-serialized version, which verifies that your own parser is self-consistent and nothing else. Verify the bytes that arrived, before anything touches them, and make sure the middleware that trims form fields or checks a cross-site token is not in the way.

The retry is a new request, not a replayed one. Stripe retries a failed delivery with exponential backoff for up to three days, and generates a fresh signature and timestamp for every attempt. That is why a five-minute tolerance never rejects a legitimate retry. It also means the bytes differ between attempts, so the thing you deduplicate on has to be the event identifier in the payload and cannot be a hash of the request.

Nothing arrives in order. Stripe documents that events are not delivered in the order they were generated, and Shopify documents no ordering within a topic or across topics for the same resource. A handler that assumes the created event precedes the updated one is wrong on a schedule you cannot observe. Order by the resource's own timestamp, or treat the event as a signal and fetch the current state.

Your latency budget belongs to the vendor. Shopify allows one second to establish the connection and five seconds for the whole request. That is the budget for your framework, your authentication middleware, your logging, your database and whatever else the request path picked up since anyone last looked at it. It is not a target to design toward. It is a limit somebody else set, and it applies on your worst afternoon rather than your median one.

Repeated failure gets you switched off. Continuous failures end with the vendor disabling the endpoint or removing the subscription and emailing whoever is listed as the technical contact, which is frequently a person who has left. The vendor is protecting its own delivery infrastructure and is right to. The consequence is that your integration stops, without anyone in your organization deciding it should, at whatever hour the counter runs out.

The payload shape moves on the vendor's calendar. Stripe names API versions for their release date and lets each endpoint pin one, defaulting to the account's version otherwise. Version 2022-08-01 landed three weeks ago with breaking changes to Checkout Sessions, Invoicing and Issuing. If your endpoint is pinned you keep the old shape until you choose to move. If it is not, the JSON reaching your handler changes the moment somebody clicks upgrade in a dashboard.

Read the raw body. Verify the signature and, where the scheme provides one, the timestamp, before parsing anything. Write the event to durable storage keyed on the vendor's event identifier, with a uniqueness constraint on that column. Return 200. Do the actual work afterward, in a worker, reading from that storage. Four steps, and the ordering is the design: everything that can be slow or wrong now happens after the acknowledgment rather than in front of it.

The uniqueness constraint is doing more than it looks like. It is your idempotency key, and the vendor gave it to you for free in every payload. An insert that ignores conflicts turns a duplicate delivery into a no-op at the storage layer, before any business logic is reachable, which is a much stronger guarantee than a check-then-act in application code that two concurrent retries will step straight through.

This costs something and I would rather price it than skip past it. You now have a queue, a worker, a dead letter path and a second thing that can be down, and for a genuinely small endpoint that writes one row on one event type, doing the work inline is defensible. The test I would apply is not volume. It is whether the handler can exceed the vendor's timeout on its worst day, and whether running it twice does anything a customer would notice. If either answer is yes, the queue is not optional at any volume.

The piece almost nobody builds is the way back. Retries stop, three days is generous and finite, and an endpoint that was broken across a long weekend has permanently lost whatever arrived during it. The vendors that retry also expose the events over their API, so a reconciliation job that lists events since a checkpoint and feeds them through the same handler is an afternoon of work. It is worth having before the weekend rather than during the Monday.

I should resist making this sound like architecture, because it is not. A signature check, a unique index on an identifier the vendor already sends, and a queue between the endpoint and the work. A week, at the outside, and most of it is the queue you probably already run. The reason it is so consistently absent is not difficulty. It is that the webhook endpoint is the last line item on an integration ticket, and it is considered finished the moment the vendor's test event turns green, at which point it looks like an ordinary route handler and gets reviewed like one.

What stays with me is how little the endpoint resembles the rest of the system it sits in. Every other write path has a login in front of it and a person behind it, and somebody has thought about who that person is allowed to be. This one has a URL, a stranger's retry timer, a five-second budget set in another company's documentation, and whatever your framework did to the request body before your code was allowed to look at it.