Every business application you build in 2026 talks to something else. Payments come from Stripe, deals come from HubSpot, deploys come from GitHub, and increasingly your AI agents want to be told when something happened rather than asking every thirty seconds. The connective tissue for almost all of it is the humble webhook: an HTTP POST fired at a URL when an event occurs.
Webhooks are trivially easy to start with and surprisingly hard to get right. Most teams sit in what we have started calling the valley: past the point where a single endpoint and a prayer is adequate, but nowhere near the scale that justifies a dedicated platform team building delivery infrastructure. That valley is where integrations quietly rot, where a missed payment event becomes a support ticket three weeks later, and where nobody can tell you whether the event was ever sent at all.
TL;DR
- Webhooks fail silently by design: HTTP gives you no delivery guarantee, and most teams have no visibility into what was sent, retried, or dropped.
- Receivers should acknowledge fast and process asynchronously. Return 202, enqueue, and do the real work in a worker. Anything else turns a slow database write into a lost event.
- Assume duplicates and out-of-order delivery. Idempotency keys and event timestamps are not optional extras, they are the contract.
- If you send webhooks, you owe subscribers signed payloads, documented retry behaviour, versioning, and a way to replay history.
- Build versus buy is a real decision now. Svix, Hookdeck and Convoy exist because delivery infrastructure is genuinely hard, but a queue plus a worker plus a dead letter table covers most SME requirements.
- AI agents are becoming heavy webhook consumers, which makes payload design and signature verification a security concern rather than a convenience.
Why webhooks break
The failure modes are boringly consistent across every codebase we audit.
The receiver does the work inline. A webhook arrives, the handler validates it, writes to three tables, calls a third-party API, sends an email, and then returns 200. Total time: nine seconds. The sender’s timeout was five. The sender records a failure and retries, so now you have processed the same event twice, and your customer has two confirmation emails. Under load, the handler times out consistently and the sender eventually gives up entirely.
Nobody verifies signatures. A webhook endpoint is a public, unauthenticated URL that mutates your database. Stripe, GitHub and most serious providers sign payloads with an HMAC and include a timestamp. A surprising number of production endpoints check neither, which means anyone who guesses the URL can mark invoices as paid. Verifying the signature is roughly six lines of code, and it needs to happen before parsing, using the raw request body rather than the re-serialised object.
Payloads are trusted as truth. Webhook payloads are snapshots of a moment that has already passed. By the time you process an “order updated” event, the order may have been updated twice more. For anything that matters, treat the webhook as a signal and refetch current state from the source API. It costs an extra request and removes an entire category of race condition.
There is no record. This is the one that turns a ten minute fix into a two day investigation. When a client asks why a booking never synced, the honest answer at most agencies is that we cannot tell whether the event arrived, was rejected, or was never sent. Persist every inbound webhook with its headers, raw body, and processing outcome before you do anything else with it. Storage is cheap. Reconstructing history is not.
The receiver checklist
If you are consuming webhooks, the architecture that survives contact with production looks like this:
- Verify the signature against the raw body, with a timestamp tolerance window of around five minutes to blunt replay attacks.
- Persist the raw event immediately, keyed by the provider’s event ID.
- Return 202 Accepted in well under a second. You are acknowledging receipt, not completion.
- Enqueue the work and process it in a background worker where you control concurrency, retries and backoff independently of the sender.
- Deduplicate on the event ID. If you have seen it, skip it. Providers explicitly warn that at-least-once delivery means duplicates.
- Handle out-of-order arrival by comparing event timestamps against your stored state, and discard events older than what you already hold.
- Fail loudly. Dead letter anything that exhausts retries, and alert on the dead letter queue depth rather than on individual errors.
The step most teams skip is the second one, and it is the cheapest insurance in the entire list.
The sender checklist
Sending webhooks is the harder half, because you are now operating delivery infrastructure on behalf of customers whose endpoints you do not control. One subscriber with a flaky server should never affect anyone else’s delivery, which means per-subscriber queues rather than a shared one.
At minimum, a credible webhook product provides HMAC signing with a documented verification recipe, a rotating secret with an overlap window so customers can rotate without downtime, exponential backoff with jitter across a defined retry schedule, automatic disabling of endpoints that fail persistently along with a notification, a versioned payload schema so you can evolve fields without breaking subscribers, and a dashboard where customers can inspect and replay past deliveries.
Also worth stating plainly in your documentation: whether you guarantee ordering. Most systems do not, and pretending otherwise sets subscribers up to build fragile assumptions. Send an event ID, a timestamp, and an event type, keep the payload small, and let consumers refetch the detail they need.
Build or buy
Five years ago, rolling your own was the default. It is now a genuine decision. Svix, Hookdeck and Convoy all sell the delivery layer as a product, and if webhooks are a headline feature of your platform rather than an implementation detail, buying the fan-out, retry and replay infrastructure is usually cheaper than staffing it.
For most SMEs, though, the middle path is right: a transactional outbox table, a queue, a worker with backoff, and a dead letter table. That is a few days of work, it runs on infrastructure you already have, and it removes ninety percent of the pain. Reach for a vendor when you need customer-facing delivery dashboards, per-subscriber rate limiting, or compliance-grade audit trails.
What changes with AI agents
The interesting shift in 2026 is that a growing share of webhook consumers are autonomous. Agents subscribe to events, and event payloads become part of an agent’s context window. That has two consequences worth taking seriously.
First, an unverified webhook endpoint feeding an agent is an indirect prompt injection vector. Text fields in a payload can carry instructions, and if that text reaches a model with tool access, an attacker who can post to your endpoint can influence what your agent does. Signature verification stops being hygiene and becomes a control.
Second, agents are far less tolerant of ambiguity than the humans who wrote the original integration. Vague event types, inconsistent field naming and undocumented nullability all degrade agent reliability in ways that are hard to debug. Well-designed webhook payloads with explicit event types and stable schemas are quietly becoming an AI readiness requirement.
The pragmatic takeaway
Webhooks are infrastructure pretending to be a feature. The teams that treat them as such, with persistence, idempotency, signatures and observability built in from day one, spend almost no time on integration incidents. The teams that treat a webhook as “just an endpoint” spend a disproportionate share of their support budget on events that may or may not have happened.
If your integrations are unreliable and nobody on your team can explain exactly why, that is not bad luck. It is an architecture problem with a well understood solution.
REPTILEHAUS builds and repairs integration layers for businesses across Ireland and Europe, covering event architecture, API design, DevOps and AI agent infrastructure. If your webhooks are costing you more than they should, get in touch.
📷 Photo by Zachary Moneypenny on Unsplash
