FeaturesCopierAnalyticsWebhooksPricingPartnershipBlog
Sign inGet started

Webhook retries, ordering, and idempotency: what to assume

8 min readMetaKit

Your database says a position closed twice. Or it has a position.closed row for a ticket that, according to the same database, never opened. Nothing is wrong with the sender. Your consumer just assumed guarantees that no webhook system gives you.

What a consumer must assume comes down to three things: at-least-once delivery, no ordering, and a sender that will eventually give up. We'll use MetaKit's own webhook retries and delivery behaviour as the worked example, because the numbers are public and the llms.txt spells them out. The rules apply to Stripe, GitHub, and anything else that POSTs JSON at you.

Webhook retries mean at-least-once, and nothing stronger

MetaKit delivers each event to each of your webhook endpoints with up to five attempts. The delays between attempts are 0.5s, 1s, 2s and 4s. Each attempt has a 5-second request timeout. A non-2xx response or a timeout counts as a failure and triggers the next attempt.

Do the arithmetic on the worst case: five attempts that each hang for the full five seconds, plus 7.5 seconds of backoff, is about 32 seconds from first attempt to final give-up. The best case is a single 50ms POST.

"At least once" hides in the word timeout. Your handler receives the request, writes to the database, and returns 200 at 5.2 seconds. The sender stopped listening at 5.0. It records a failure and sends the same event again half a second later. You now have the event twice, and from the sender's side, it did exactly what it promised.

There is no delivery mode that fixes this. Exactly-once across a network boundary isn't a thing a sender can provide; it's a property your consumer builds by being idempotent.

Retries reuse the same timestamp and signature

Here's the detail that matters for dedupe. MetaKit stamps the t in X-MetaKit-Signature: t=...,v1=... once per event, not per attempt, and the body is serialised once. Every retry is byte-identical: same body, same t, same v1.

Two consequences.

First, there is no delivery id. Nothing in the headers or the payload says "this is attempt 3 of the event you saw at attempt 1". If you want a dedupe key, make one: the v1 HMAC is a fine candidate, since it's a function of the secret, the timestamp and the exact bytes. Store t:v1 in a table or a Redis set with a TTL of a day, and drop anything you've seen.

Second, that key isn't enough on its own. A dedupe table protects you against the same delivery arriving twice. It doesn't protect you against two genuinely distinct events that mean the same thing to your code, and it doesn't protect you if the table is wiped. The real fix is making the handler itself idempotent. For trade events the natural key is data.ticket plus the event name: "ticket 99302156 is closed" is a fact you can write any number of times with ON CONFLICT DO NOTHING or an upsert, and the second write changes nothing.

A small honesty note: two different events with identical bodies in the same second would collide on t:v1. That can only happen for something like two identical position.modified events, and if your handler is idempotent the collision is harmless. Which is the point.

The timestamp reuse also affects your freshness window. A delivery that lands on the fifth attempt is about 30 seconds older than "now". Reject deliveries older than five minutes, not five seconds, or you'll reject your own retries.

Ordering is not guaranteed, so don't replay events

Each event gets its own retry schedule, independent of every other event. Put two together and ordering falls apart.

A scalp opens and closes in one second. The position.opened delivery hits your handler while it's warming a database connection and times out at 5s. The position.closed delivery, sent a second after the open, hits a warm handler and returns 200 in 40ms. Your consumer processes the close, then half a second later processes the open. If your code does "on opened, insert row; on closed, update row", the update finds nothing, and then the insert creates a position that looks open forever.

The tempting fix is to buffer and reorder by timestamp. Don't. You'd need to know how long to wait, and the answer is "up to 32 seconds, unless the fifth attempt also failed, in which case forever".

The fix that works: treat events as hints that state changed, not as a log to replay. When a trade event arrives for account 2, fetch GET /v1/accounts/2/positions and reconcile your table against it. The API is the source of truth; the webhook is the doorbell. A handler built this way is naturally idempotent (fetching twice is harmless), naturally order-independent (state is state), and naturally recovers from a lost delivery the next time any event arrives.

You give up a little: a reconcile costs an API call, and for a closed position you'll want the deal from /v1/accounts/2/deals rather than the (now absent) position, because in MT5 the close is a deal, not a position (the orders, deals and positions post is the primer). You gain a consumer that's correct.

Respond 200 first, work later

Five seconds sounds generous until you list what's in it: TLS handshake, your reverse proxy, framework startup on a cold serverless function, a database connection from a cold pool, the write itself, and the response. A Lambda cold start alone can eat a meaningful chunk of that.

The rule is old and still right: verify the signature, persist the raw delivery somewhere durable, return 200, and do the real work afterwards.

"Somewhere durable" is load-bearing. On a long-running server, a queue or an inbox table is fine. On serverless, a background thread is not, because the platform freezes or kills the process once the response is sent. Write the raw body to a table, a queue, or object storage inside the request, and process it from a worker.

Verify before you persist, so a forged payload never touches your queue. And verify on the raw bytes: the signature covers the body as sent, and re-serialising parsed JSON produces different bytes.

After the fifth failure, the event is gone

This is the part to be clear-eyed about. After five failed attempts MetaKit stops. The webhook's lastDeliveryStatus becomes failed and lastDeliveryAt records when. That's visible in the dashboard and from GET /v1/webhooks. The event itself is not queued anywhere you can reach.

What MetaKit does not do, as of llms.txt today:

  • No replay endpoint. You can't ask for "everything since Tuesday".
  • No delivery log. You get last status and last time, per endpoint, not a history.
  • No per-event subscriptions. Every endpoint receives every event; you filter by X-MetaKit-Event and account_id.
  • No dead-letter queue you can drain.

So your consumer needs a second path to correctness that doesn't depend on webhooks at all. Two habits cover it. Run a periodic reconcile (every few minutes) that fetches positions for each account and diffs against your state, which catches whatever the webhook path dropped. And watch lastDeliveryStatus: if it flips to failed, something on your side was down for 30 seconds, and you probably want to know why.

If your handler is already "fetch state and reconcile", the periodic job is the same function on a timer. That's the payoff of building it that way.

A reference consumer

Node, Express, Postgres. Verify, dedupe, persist, ack. A worker does the rest.

import crypto from 'node:crypto';
import express from 'express';
import { Pool } from 'pg';
 
const pool = new Pool({ connectionString: process.env.DATABASE_URL });
const SECRET = process.env.METAKIT_WEBHOOK_SECRET!;
const app = express();
 
function verify(raw: Buffer, header: string | undefined, toleranceSec = 300) {
  if (!header) return null;
  const parts = Object.fromEntries(header.split(',').map((kv) => kv.trim().split('=')));
  const { t, v1 } = parts;
  if (!t || !v1) return null;
  if (Math.abs(Date.now() / 1000 - Number(t)) > toleranceSec) return null;
  const expected = crypto.createHmac('sha256', SECRET).update(`${t}.${raw}`).digest('hex');
  const a = Buffer.from(expected);
  const b = Buffer.from(v1);
  if (a.length !== b.length || !crypto.timingSafeEqual(a, b)) return null;
  return `${t}:${v1}`;                       // the dedupe key
}
 
app.post('/hooks/metakit', express.raw({ type: 'application/json' }), async (req, res) => {
  const key = verify(req.body, req.get('X-MetaKit-Signature'));
  if (!key) return res.sendStatus(400);
 
  // Durable inbox. Duplicate deliveries are a no-op at the database.
  await pool.query(
    `INSERT INTO webhook_inbox (delivery_key, event, body)
     VALUES ($1, $2, $3) ON CONFLICT (delivery_key) DO NOTHING`,
    [key, req.get('X-MetaKit-Event'), req.body.toString('utf8')],
  );
  res.sendStatus(200);                       // ack; nothing slow above this line
});
 
app.listen(3000);
CREATE TABLE webhook_inbox (
  delivery_key text PRIMARY KEY,
  event        text NOT NULL,
  body         text NOT NULL,
  received_at  timestamptz NOT NULL DEFAULT now(),
  processed_at timestamptz
);

The worker polls webhook_inbox WHERE processed_at IS NULL, parses the body, and for any trade.* event fetches the account's positions from the API and reconciles. It does not trust data in the payload as the final state; it trusts the API. When it's done it sets processed_at. If the worker crashes mid-way, the row is still unprocessed and gets picked up again, which is fine, because reconciling twice is harmless.

Ordering never comes up. Duplicates never come up. A dropped delivery is caught by the next event or the next scheduled reconcile. Every property the sender couldn't give you, the consumer built for itself.

For a small consumer built on exactly this pattern, the Grafana dashboard post stores trade events with a unique constraint and lets Postgres do the deduplication.

The signature function above is the minimum. The verification post covers rotation (accept both secrets for a short window after POST /v1/webhooks/{id}/rotate) and the raw-body pitfalls in each framework. Build the inbox first.

← All posts