Workflows

Reliable Webhook Delivery

Send your app's events to customer endpoints in order, retry failed deliveries, and turn off endpoints that keep failing.

See the full example on GitHub. View on GitHub

When your customers subscribe to webhooks, their endpoints will be slow, down for a deploy, or broken for days. This guide gives each customer endpoint its own workflow that delivers events in order, retries with backoff, keeps a log of every attempt, and stops sending to an endpoint that keeps failing until the customer turns it back on.

What we’ll build:

An endpoint workflow per customer endpoint that holds an ordered outbox of events to deliver.
Signed payloads, so customers can check that each webhook came from you.
Retries with backoff and a record of every attempt, with its status code and timing.
A rule that disables an endpoint after repeated failures and keeps its events until it is turned back on.
Actions your customer portal calls to show the delivery log and re-enable an endpoint.

Install

npm add @rivet-dev/workflows rivetkit hono @hono/node-server

Sign payloads

Every request carries a signature, so a customer can tell your webhooks apart from anyone else posting to their URL:

import { createHmac, randomBytes, timingSafeEqual } from "node:crypto";

// Signatures follow the Standard Webhooks spec (standardwebhooks.com), so
// customers can check them with any Standard Webhooks library. A secret is
// `whsec_` followed by the base64 key bytes.
export function createSecret(): string {
	return `whsec_${randomBytes(24).toString("base64")}`;
}

function sign(
	secret: string,
	id: string,
	timestamp: number,
	body: string,
): string {
	const key = Buffer.from(secret.slice("whsec_".length), "base64");
	return createHmac("sha256", key)
		.update(`${id}.${timestamp}.${body}`)
		.digest("base64");
}

export function signatureHeaders(
	secret: string,
	id: string,
	body: string,
): Record<string, string> {
	const timestamp = Math.floor(Date.now() / 1000);
	return {
		"webhook-id": id,
		"webhook-timestamp": String(timestamp),
		"webhook-signature": `v1,${sign(secret, id, timestamp, body)}`,
	};
}

// Reject requests signed longer ago than this, so a captured request
// cannot be replayed later.
const TOLERANCE_SECONDS = 5 * 60;

// What your customers run on their side to check that a webhook came from you.
export function verifyWebhook(
	secret: string,
	headers: Headers,
	body: string,
): boolean {
	const id = headers.get("webhook-id");
	const timestamp = Number(headers.get("webhook-timestamp"));
	const header = headers.get("webhook-signature");
	if (!id || !header || !Number.isFinite(timestamp)) return false;
	if (Math.abs(Date.now() / 1000 - timestamp) > TOLERANCE_SECONDS) return false;

	const expected = Buffer.from(sign(secret, id, timestamp, body), "base64");
	// The header holds several space-separated signatures while a secret is
	// being rotated.
	return header.split(" ").some((entry) => {
		const [version, signature] = entry.split(",", 2);
		if (version !== "v1" || !signature) return false;
		const given = Buffer.from(signature, "base64");
		return given.length === expected.length && timingSafeEqual(given, expected);
	});
}

The headers follow the Standard Webhooks spec. webhook-signature is an HMAC-SHA256 of <id>.<timestamp>.<body>, keyed by the endpoint’s secret, so customers can check it with any Standard Webhooks library.
webhook-id is the event id. It stays the same on every retry, so customers can use it to skip events they have already handled.
Each endpoint gets its own secret from createSecret. A leaked secret affects only that endpoint.

Create the endpoint workflow

Each customer endpoint is its own endpoint workflow, keyed by an endpoint id. Its queue is the outbox:

import { Loop, queue, workflow } from "@rivet-dev/workflows";
import { signatureHeaders } from "./sign";

export type WebhookEvent = {
	id: string;
	type: string;
	data: unknown;
	createdAt: number;
};

export type Attempt = {
	eventId: string;
	eventType: string;
	attempt: number;
	status: number | null;
	error: string | null;
	durationMs: number;
	at: number;
};

// A failed delivery is retried 5 more times, 2s, 4s, 8s, 16s, and 32s apart,
// before the endpoint is disabled. Raise these numbers for production.
const MAX_RETRIES = 5;
const RETRY_BACKOFF_BASE_MS = 1000;
const RETRY_BACKOFF_MAX_MS = 60 * 60 * 1000;
const REQUEST_TIMEOUT_MS = 10_000;
// How many attempts the delivery log keeps.
const LOG_SIZE = 100;

// One workflow per customer endpoint, keyed by the endpoint id.
export const endpoint = workflow({
	state: {
		url: "",
		secret: "",
		disabled: false,
		// The id of the event being delivered, if any.
		pending: null as string | null,
		delivered: 0,
		attempts: [] as Attempt[],
	},
	// The backend passes the URL and signing secret when it creates the endpoint.
	onCreate: (c, input: { url: string; secret: string }) => {
		c.state.url = input.url;
		c.state.secret = input.secret;
	},
	queues: {
		events: queue<WebhookEvent>(),
		enable: queue<Record<string, never>>(),
	},
	run: async (ctx) => {
		await ctx.loop("delivery-loop", async (loopCtx) => {
			// The queue is the outbox. Events come off it one at a time, in order.
			const { body: event } = await loopCtx.queue.next("wait-event", {
				names: ["events"],
			});
			await loopCtx.step("mark-pending", async (step) => {
				step.state.pending = event.id;
			});

			// Keep at this event until it is delivered. Later events wait behind it.
			await loopCtx.loop("deliver-event", async (eventCtx) => {
				const delivery = await eventCtx.tryStep({
					name: "deliver",
					maxRetries: MAX_RETRIES,
					retryBackoffBase: RETRY_BACKOFF_BASE_MS,
					retryBackoffMax: RETRY_BACKOFF_MAX_MS,
					timeout: REQUEST_TIMEOUT_MS + 5_000,
					retryOnTimeout: true,
					run: async (step) => {
						const body = JSON.stringify(event);
						const startedAt = Date.now();
						let status: number | null = null;
						let error: string | null = null;
						try {
							const res = await fetch(step.state.url, {
								method: "POST",
								headers: {
									"content-type": "application/json",
									...signatureHeaders(step.state.secret, event.id, body),
								},
								body,
								signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS),
							});
							status = res.status;
						} catch (err) {
							error = err instanceof Error ? err.message : String(err);
						}

						// Record the attempt before throwing. A failed step keeps its
						// state changes, so every attempt is logged.
						const log = step.state.attempts;
						log.push({
							eventId: event.id,
							eventType: event.type,
							attempt: log.filter((a) => a.eventId === event.id).length + 1,
							status,
							error,
							durationMs: Date.now() - startedAt,
							at: startedAt,
						});
						if (log.length > LOG_SIZE) log.splice(0, log.length - LOG_SIZE);

						// Any 2xx answer counts as delivered. Anything else is retried.
						if (status === null || status < 200 || status >= 300) {
							throw new Error(
								`delivery failed: ${error ?? `status ${status}`}`,
							);
						}
					},
				});

				if (delivery.ok) {
					await eventCtx.step("record-delivery", async (step) => {
						step.state.pending = null;
						step.state.delivered += 1;
					});
					return Loop.break(undefined);
				}

				// Out of retries. Stop sending until the endpoint is turned back on.
				await eventCtx.step("disable", async (step) => {
					step.state.disabled = true;
					step.log.warn({ msg: "endpoint disabled", eventId: event.id });
				});
				await eventCtx.queue.nextBatch("wait-enable", {
					names: ["enable"],
					count: 10,
				});
				await eventCtx.step("enable", async (step) => {
					step.state.disabled = false;
				});
				// The loop runs again and retries the same event.
			});
		});
	},
	actions: {
		listAttempts: (c) => ({
			url: c.state.url,
			disabled: c.state.disabled,
			delivered: c.state.delivered,
			pending: c.state.pending,
			attempts: c.state.attempts.toReversed(),
		}),
		enable: async (c) => {
			if (c.state.disabled) await c.queue.send("enable", {});
		},
	},
});

Events come off the queue one at a time, in the order your backend sent them. The next event stays in the queue until the current one is delivered, so a customer never sees event 2 before event 1.
The inner deliver-event loop keeps working on the same event until it is delivered, including across a disable and re-enable.
One slow or broken endpoint backs up only its own outbox. Every other endpoint has its own workflow and keeps receiving events.
The URL and secret are set in onCreate from the input your backend passes when it creates the endpoint.

Deliver with retries

The delivery itself is one step that posts the signed event and records the attempt:

import { Loop, queue, workflow } from "@rivet-dev/workflows";
import { signatureHeaders } from "./sign";

export type WebhookEvent = {
	id: string;
	type: string;
	data: unknown;
	createdAt: number;
};

export type Attempt = {
	eventId: string;
	eventType: string;
	attempt: number;
	status: number | null;
	error: string | null;
	durationMs: number;
	at: number;
};

// A failed delivery is retried 5 more times, 2s, 4s, 8s, 16s, and 32s apart,
// before the endpoint is disabled. Raise these numbers for production.
const MAX_RETRIES = 5;
const RETRY_BACKOFF_BASE_MS = 1000;
const RETRY_BACKOFF_MAX_MS = 60 * 60 * 1000;
const REQUEST_TIMEOUT_MS = 10_000;
// How many attempts the delivery log keeps.
const LOG_SIZE = 100;

// One workflow per customer endpoint, keyed by the endpoint id.
export const endpoint = workflow({
	state: {
		url: "",
		secret: "",
		disabled: false,
		// The id of the event being delivered, if any.
		pending: null as string | null,
		delivered: 0,
		attempts: [] as Attempt[],
	},
	// The backend passes the URL and signing secret when it creates the endpoint.
	onCreate: (c, input: { url: string; secret: string }) => {
		c.state.url = input.url;
		c.state.secret = input.secret;
	},
	queues: {
		events: queue<WebhookEvent>(),
		enable: queue<Record<string, never>>(),
	},
	run: async (ctx) => {
		await ctx.loop("delivery-loop", async (loopCtx) => {
			// The queue is the outbox. Events come off it one at a time, in order.
			const { body: event } = await loopCtx.queue.next("wait-event", {
				names: ["events"],
			});
			await loopCtx.step("mark-pending", async (step) => {
				step.state.pending = event.id;
			});

			// Keep at this event until it is delivered. Later events wait behind it.
			await loopCtx.loop("deliver-event", async (eventCtx) => {
				const delivery = await eventCtx.tryStep({
					name: "deliver",
					maxRetries: MAX_RETRIES,
					retryBackoffBase: RETRY_BACKOFF_BASE_MS,
					retryBackoffMax: RETRY_BACKOFF_MAX_MS,
					timeout: REQUEST_TIMEOUT_MS + 5_000,
					retryOnTimeout: true,
					run: async (step) => {
						const body = JSON.stringify(event);
						const startedAt = Date.now();
						let status: number | null = null;
						let error: string | null = null;
						try {
							const res = await fetch(step.state.url, {
								method: "POST",
								headers: {
									"content-type": "application/json",
									...signatureHeaders(step.state.secret, event.id, body),
								},
								body,
								signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS),
							});
							status = res.status;
						} catch (err) {
							error = err instanceof Error ? err.message : String(err);
						}

						// Record the attempt before throwing. A failed step keeps its
						// state changes, so every attempt is logged.
						const log = step.state.attempts;
						log.push({
							eventId: event.id,
							eventType: event.type,
							attempt: log.filter((a) => a.eventId === event.id).length + 1,
							status,
							error,
							durationMs: Date.now() - startedAt,
							at: startedAt,
						});
						if (log.length > LOG_SIZE) log.splice(0, log.length - LOG_SIZE);

						// Any 2xx answer counts as delivered. Anything else is retried.
						if (status === null || status < 200 || status >= 300) {
							throw new Error(
								`delivery failed: ${error ?? `status ${status}`}`,
							);
						}
					},
				});

				if (delivery.ok) {
					await eventCtx.step("record-delivery", async (step) => {
						step.state.pending = null;
						step.state.delivered += 1;
					});
					return Loop.break(undefined);
				}

				// Out of retries. Stop sending until the endpoint is turned back on.
				await eventCtx.step("disable", async (step) => {
					step.state.disabled = true;
					step.log.warn({ msg: "endpoint disabled", eventId: event.id });
				});
				await eventCtx.queue.nextBatch("wait-enable", {
					names: ["enable"],
					count: 10,
				});
				await eventCtx.step("enable", async (step) => {
					step.state.disabled = false;
				});
				// The loop runs again and retries the same event.
			});
		});
	},
	actions: {
		listAttempts: (c) => ({
			url: c.state.url,
			disabled: c.state.disabled,
			delivered: c.state.delivered,
			pending: c.state.pending,
			attempts: c.state.attempts.toReversed(),
		}),
		enable: async (c) => {
			if (c.state.disabled) await c.queue.send("enable", {});
		},
	},
});

Any 2xx answer counts as delivered. Any other status, a network error, or no answer within 10 seconds throws, and tryStep retries after 2, 4, 8, 16, and 32 seconds.
The waits between retries are durable. The Rivet Actor can sleep or restart during a long backoff and picks up at the next attempt.
The attempt is pushed to attempts before the step throws. A failed step keeps the changes it made to state, so the log shows every try, not just the last one.
The log keeps the newest 100 attempts. Store attempts in your own database if you need a longer history.

These retries span about a minute so the example is quick to try. Real endpoints can be down for hours, so raise MAX_RETRIES and RETRY_BACKOFF_MAX_MS for production. For example, 20 retries capped at 1 hour keep trying for about 10 hours.

Delivery is at least once. If a customer’s server handles a request but its answer is lost or arrives after the timeout, the event is sent again. Tell customers to skip any webhook-id they have already handled.

Disable failing endpoints

When the retries run out, the workflow disables the endpoint instead of dropping the event:

import { Loop, queue, workflow } from "@rivet-dev/workflows";
import { signatureHeaders } from "./sign";

export type WebhookEvent = {
	id: string;
	type: string;
	data: unknown;
	createdAt: number;
};

export type Attempt = {
	eventId: string;
	eventType: string;
	attempt: number;
	status: number | null;
	error: string | null;
	durationMs: number;
	at: number;
};

// A failed delivery is retried 5 more times, 2s, 4s, 8s, 16s, and 32s apart,
// before the endpoint is disabled. Raise these numbers for production.
const MAX_RETRIES = 5;
const RETRY_BACKOFF_BASE_MS = 1000;
const RETRY_BACKOFF_MAX_MS = 60 * 60 * 1000;
const REQUEST_TIMEOUT_MS = 10_000;
// How many attempts the delivery log keeps.
const LOG_SIZE = 100;

// One workflow per customer endpoint, keyed by the endpoint id.
export const endpoint = workflow({
	state: {
		url: "",
		secret: "",
		disabled: false,
		// The id of the event being delivered, if any.
		pending: null as string | null,
		delivered: 0,
		attempts: [] as Attempt[],
	},
	// The backend passes the URL and signing secret when it creates the endpoint.
	onCreate: (c, input: { url: string; secret: string }) => {
		c.state.url = input.url;
		c.state.secret = input.secret;
	},
	queues: {
		events: queue<WebhookEvent>(),
		enable: queue<Record<string, never>>(),
	},
	run: async (ctx) => {
		await ctx.loop("delivery-loop", async (loopCtx) => {
			// The queue is the outbox. Events come off it one at a time, in order.
			const { body: event } = await loopCtx.queue.next("wait-event", {
				names: ["events"],
			});
			await loopCtx.step("mark-pending", async (step) => {
				step.state.pending = event.id;
			});

			// Keep at this event until it is delivered. Later events wait behind it.
			await loopCtx.loop("deliver-event", async (eventCtx) => {
				const delivery = await eventCtx.tryStep({
					name: "deliver",
					maxRetries: MAX_RETRIES,
					retryBackoffBase: RETRY_BACKOFF_BASE_MS,
					retryBackoffMax: RETRY_BACKOFF_MAX_MS,
					timeout: REQUEST_TIMEOUT_MS + 5_000,
					retryOnTimeout: true,
					run: async (step) => {
						const body = JSON.stringify(event);
						const startedAt = Date.now();
						let status: number | null = null;
						let error: string | null = null;
						try {
							const res = await fetch(step.state.url, {
								method: "POST",
								headers: {
									"content-type": "application/json",
									...signatureHeaders(step.state.secret, event.id, body),
								},
								body,
								signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS),
							});
							status = res.status;
						} catch (err) {
							error = err instanceof Error ? err.message : String(err);
						}

						// Record the attempt before throwing. A failed step keeps its
						// state changes, so every attempt is logged.
						const log = step.state.attempts;
						log.push({
							eventId: event.id,
							eventType: event.type,
							attempt: log.filter((a) => a.eventId === event.id).length + 1,
							status,
							error,
							durationMs: Date.now() - startedAt,
							at: startedAt,
						});
						if (log.length > LOG_SIZE) log.splice(0, log.length - LOG_SIZE);

						// Any 2xx answer counts as delivered. Anything else is retried.
						if (status === null || status < 200 || status >= 300) {
							throw new Error(
								`delivery failed: ${error ?? `status ${status}`}`,
							);
						}
					},
				});

				if (delivery.ok) {
					await eventCtx.step("record-delivery", async (step) => {
						step.state.pending = null;
						step.state.delivered += 1;
					});
					return Loop.break(undefined);
				}

				// Out of retries. Stop sending until the endpoint is turned back on.
				await eventCtx.step("disable", async (step) => {
					step.state.disabled = true;
					step.log.warn({ msg: "endpoint disabled", eventId: event.id });
				});
				await eventCtx.queue.nextBatch("wait-enable", {
					names: ["enable"],
					count: 10,
				});
				await eventCtx.step("enable", async (step) => {
					step.state.disabled = false;
				});
				// The loop runs again and retries the same event.
			});
		});
	},
	actions: {
		listAttempts: (c) => ({
			url: c.state.url,
			disabled: c.state.disabled,
			delivered: c.state.delivered,
			pending: c.state.pending,
			attempts: c.state.attempts.toReversed(),
		}),
		enable: async (c) => {
			if (c.state.disabled) await c.queue.send("enable", {});
		},
	},
});

On success, the event is cleared from pending and the loop breaks, so the workflow takes the next event off the queue.
On failure, the endpoint is marked disabled and the workflow waits on the enable queue. The failed event and every event behind it are kept until then, so nothing is lost.
While it waits, the Rivet Actor can sleep. No process is held for a disabled endpoint, however long it stays off.
After enable, the loop runs again and retries the same event with a fresh set of retries.

Emit events and serve the portal

Your backend creates endpoints, emits events, and gives your customer portal two routes:

import { randomUUID } from "node:crypto";
import { serve } from "@hono/node-server";
import { setup } from "@rivet-dev/workflows";
import { Hono } from "hono";
import { createClient } from "rivetkit/client";
import { endpoint, type WebhookEvent } from "./endpoint";
import { createSecret } from "./sign";

export const registry = setup({ use: { endpoint } });
const client = createClient<typeof registry>();

registry.start();

// Which endpoints get which event types. Keep this in your database.
const subscriptions = new Map<string, Set<string>>();

// Call this from your app whenever something happens that customers can subscribe to.
export async function emit(type: string, data: unknown): Promise<WebhookEvent> {
	const event: WebhookEvent = {
		id: `msg_${randomUUID()}`,
		type,
		data,
		createdAt: Date.now(),
	};
	for (const endpointId of subscriptions.get(type) ?? []) {
		await client.endpoint.get([endpointId]).send("events", event);
	}
	return event;
}

const app = new Hono();

// Your customer portal calls this when a customer adds an endpoint.
app.post("/endpoints", async (c) => {
	const { url, eventTypes } = await c.req.json<{
		url: string;
		eventTypes: string[];
	}>();
	const id = `ep_${randomUUID()}`;
	const secret = createSecret();
	await client.endpoint.create([id], { input: { url, secret } });
	for (const type of eventTypes) {
		if (!subscriptions.has(type)) subscriptions.set(type, new Set());
		subscriptions.get(type)?.add(id);
	}
	// Show the secret to the customer once so they can check signatures.
	return c.json({ id, secret });
});

app.get("/endpoints/:id/attempts", async (c) => {
	return c.json(await client.endpoint.get([c.req.param("id")]).listAttempts());
});

app.post("/endpoints/:id/enable", async (c) => {
	await client.endpoint.get([c.req.param("id")]).enable();
	return c.json({ enabled: true });
});

// Stands in for your app code, so you can emit test events over HTTP.
app.post("/events", async (c) => {
	const { type, data } = await c.req.json<{ type: string; data: unknown }>();
	return c.json(await emit(type, data));
});

serve({ fetch: app.fetch, port: Number(process.env.PORT ?? 3000) });

import { Loop, queue, workflow } from "@rivet-dev/workflows";
import { signatureHeaders } from "./sign";

export type WebhookEvent = {
	id: string;
	type: string;
	data: unknown;
	createdAt: number;
};

export type Attempt = {
	eventId: string;
	eventType: string;
	attempt: number;
	status: number | null;
	error: string | null;
	durationMs: number;
	at: number;
};

// A failed delivery is retried 5 more times, 2s, 4s, 8s, 16s, and 32s apart,
// before the endpoint is disabled. Raise these numbers for production.
const MAX_RETRIES = 5;
const RETRY_BACKOFF_BASE_MS = 1000;
const RETRY_BACKOFF_MAX_MS = 60 * 60 * 1000;
const REQUEST_TIMEOUT_MS = 10_000;
// How many attempts the delivery log keeps.
const LOG_SIZE = 100;

// One workflow per customer endpoint, keyed by the endpoint id.
export const endpoint = workflow({
	state: {
		url: "",
		secret: "",
		disabled: false,
		// The id of the event being delivered, if any.
		pending: null as string | null,
		delivered: 0,
		attempts: [] as Attempt[],
	},
	// The backend passes the URL and signing secret when it creates the endpoint.
	onCreate: (c, input: { url: string; secret: string }) => {
		c.state.url = input.url;
		c.state.secret = input.secret;
	},
	queues: {
		events: queue<WebhookEvent>(),
		enable: queue<Record<string, never>>(),
	},
	run: async (ctx) => {
		await ctx.loop("delivery-loop", async (loopCtx) => {
			// The queue is the outbox. Events come off it one at a time, in order.
			const { body: event } = await loopCtx.queue.next("wait-event", {
				names: ["events"],
			});
			await loopCtx.step("mark-pending", async (step) => {
				step.state.pending = event.id;
			});

			// Keep at this event until it is delivered. Later events wait behind it.
			await loopCtx.loop("deliver-event", async (eventCtx) => {
				const delivery = await eventCtx.tryStep({
					name: "deliver",
					maxRetries: MAX_RETRIES,
					retryBackoffBase: RETRY_BACKOFF_BASE_MS,
					retryBackoffMax: RETRY_BACKOFF_MAX_MS,
					timeout: REQUEST_TIMEOUT_MS + 5_000,
					retryOnTimeout: true,
					run: async (step) => {
						const body = JSON.stringify(event);
						const startedAt = Date.now();
						let status: number | null = null;
						let error: string | null = null;
						try {
							const res = await fetch(step.state.url, {
								method: "POST",
								headers: {
									"content-type": "application/json",
									...signatureHeaders(step.state.secret, event.id, body),
								},
								body,
								signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS),
							});
							status = res.status;
						} catch (err) {
							error = err instanceof Error ? err.message : String(err);
						}

						// Record the attempt before throwing. A failed step keeps its
						// state changes, so every attempt is logged.
						const log = step.state.attempts;
						log.push({
							eventId: event.id,
							eventType: event.type,
							attempt: log.filter((a) => a.eventId === event.id).length + 1,
							status,
							error,
							durationMs: Date.now() - startedAt,
							at: startedAt,
						});
						if (log.length > LOG_SIZE) log.splice(0, log.length - LOG_SIZE);

						// Any 2xx answer counts as delivered. Anything else is retried.
						if (status === null || status < 200 || status >= 300) {
							throw new Error(
								`delivery failed: ${error ?? `status ${status}`}`,
							);
						}
					},
				});

				if (delivery.ok) {
					await eventCtx.step("record-delivery", async (step) => {
						step.state.pending = null;
						step.state.delivered += 1;
					});
					return Loop.break(undefined);
				}

				// Out of retries. Stop sending until the endpoint is turned back on.
				await eventCtx.step("disable", async (step) => {
					step.state.disabled = true;
					step.log.warn({ msg: "endpoint disabled", eventId: event.id });
				});
				await eventCtx.queue.nextBatch("wait-enable", {
					names: ["enable"],
					count: 10,
				});
				await eventCtx.step("enable", async (step) => {
					step.state.disabled = false;
				});
				// The loop runs again and retries the same event.
			});
		});
	},
	actions: {
		listAttempts: (c) => ({
			url: c.state.url,
			disabled: c.state.disabled,
			delivered: c.state.delivered,
			pending: c.state.pending,
			attempts: c.state.attempts.toReversed(),
		}),
		enable: async (c) => {
			if (c.state.disabled) await c.queue.send("enable", {});
		},
	},
});

emit sends the event to the endpoint workflow of every endpoint subscribed to its type. send returns once the event is stored in the outbox, so your app never waits on a customer’s server.
POST /endpoints creates the workflow with the URL and a new secret, and returns the secret once so the customer can check signatures.
listAttempts returns the delivery log, newest first, along with whether the endpoint is disabled. It never returns the secret.
enable queues a message for the waiting workflow, and only when the endpoint is disabled.
The subscriptions map is in memory to keep the example short. Store subscriptions in your database.

These routes trust the endpoint id in the URL. In production, check that the signed-in customer owns the endpoint before you call listAttempts or enable.

Check signatures on the customer side

Your customers run the other half. This is what their check looks like, and the test receiver in the next step uses it:

import { createHmac, randomBytes, timingSafeEqual } from "node:crypto";

// Signatures follow the Standard Webhooks spec (standardwebhooks.com), so
// customers can check them with any Standard Webhooks library. A secret is
// `whsec_` followed by the base64 key bytes.
export function createSecret(): string {
	return `whsec_${randomBytes(24).toString("base64")}`;
}

function sign(
	secret: string,
	id: string,
	timestamp: number,
	body: string,
): string {
	const key = Buffer.from(secret.slice("whsec_".length), "base64");
	return createHmac("sha256", key)
		.update(`${id}.${timestamp}.${body}`)
		.digest("base64");
}

export function signatureHeaders(
	secret: string,
	id: string,
	body: string,
): Record<string, string> {
	const timestamp = Math.floor(Date.now() / 1000);
	return {
		"webhook-id": id,
		"webhook-timestamp": String(timestamp),
		"webhook-signature": `v1,${sign(secret, id, timestamp, body)}`,
	};
}

// Reject requests signed longer ago than this, so a captured request
// cannot be replayed later.
const TOLERANCE_SECONDS = 5 * 60;

// What your customers run on their side to check that a webhook came from you.
export function verifyWebhook(
	secret: string,
	headers: Headers,
	body: string,
): boolean {
	const id = headers.get("webhook-id");
	const timestamp = Number(headers.get("webhook-timestamp"));
	const header = headers.get("webhook-signature");
	if (!id || !header || !Number.isFinite(timestamp)) return false;
	if (Math.abs(Date.now() / 1000 - timestamp) > TOLERANCE_SECONDS) return false;

	const expected = Buffer.from(sign(secret, id, timestamp, body), "base64");
	// The header holds several space-separated signatures while a secret is
	// being rotated.
	return header.split(" ").some((entry) => {
		const [version, signature] = entry.split(",", 2);
		if (version !== "v1" || !signature) return false;
		const given = Buffer.from(signature, "base64");
		return given.length === expected.length && timingSafeEqual(given, expected);
	});
}

It rebuilds the signature from the raw body and the webhook-id and webhook-timestamp headers, then compares with timingSafeEqual.
Requests signed more than 5 minutes ago are rejected, so a captured request cannot be replayed later.
The header can hold several signatures separated by spaces, which lets you rotate an endpoint’s secret without breaking deliveries.

Try it

Start the server:

npx tsx server.ts

In another terminal, run the client. It adds an endpoint, starts a receiver that answers 503 to the first 3 requests, and emits two events:

import type { Attempt } from "./endpoint";
import { startReceiver } from "./receiver";

const api = process.env.API_URL ?? "http://localhost:3000";
const receiverPort = Number(process.env.RECEIVER_PORT ?? 4000);
// Set FAIL_FIRST=100 to watch the endpoint run out of retries and get disabled.
const failFirst = Number(process.env.FAIL_FIRST ?? 3);

async function call<T>(path: string, body?: unknown): Promise<T> {
	const res = await fetch(`${api}${path}`, {
		method: body === undefined ? "GET" : "POST",
		headers: { "content-type": "application/json" },
		body: body === undefined ? undefined : JSON.stringify(body),
	});
	return (await res.json()) as T;
}

// Add an endpoint the way a customer would in your portal, then start a
// receiver that checks signatures with its secret.
const { id, secret } = await call<{ id: string; secret: string }>(
	"/endpoints",
	{
		url: `http://localhost:${receiverPort}/webhooks`,
		eventTypes: ["order.created"],
	},
);
const receiver = startReceiver({ port: receiverPort, secret, failFirst });
console.log(`endpoint ${id}`);

// Emit two events. The second one waits until the first is delivered.
await call("/events", { type: "order.created", data: { orderId: "order_1" } });
await call("/events", { type: "order.created", data: { orderId: "order_2" } });

// Print the delivery log as a portal would, until both events are delivered.
let printed = 0;
for (;;) {
	await new Promise((resolve) => setTimeout(resolve, 1000));
	const log = await call<{
		disabled: boolean;
		delivered: number;
		attempts: Attempt[];
	}>(`/endpoints/${id}/attempts`);
	for (const a of log.attempts.toReversed().slice(printed)) {
		const time = new Date(a.at).toISOString().slice(11, 19);
		console.log(
			`${time} ${a.eventId.slice(0, 12)} attempt ${a.attempt}: ${a.status ?? a.error} (${a.durationMs}ms)`,
		);
	}
	printed = log.attempts.length;
	if (log.delivered === 2) break;
	if (log.disabled) {
		console.log(
			"endpoint disabled. Fixing the receiver and turning the endpoint back on.",
		);
		receiver.recover();
		await call(`/endpoints/${id}/enable`, {});
	}
}

console.log("both events delivered");
receiver.close();

import { serve } from "@hono/node-server";
import { Hono } from "hono";
import { verifyWebhook } from "./sign";

// A test endpoint that plays the customer. It answers 503 to the first
// `failFirst` requests, then 200.
export function startReceiver(opts: {
	port: number;
	secret: string;
	failFirst: number;
}) {
	let remainingFailures = opts.failFirst;
	const seen = new Set<string>();
	const app = new Hono();

	app.post("/webhooks", async (c) => {
		const body = await c.req.text();
		if (!verifyWebhook(opts.secret, c.req.raw.headers, body))
			return c.text("bad signature", 401);

		const id = c.req.header("webhook-id") ?? "";
		const { type } = JSON.parse(body) as { type: string };
		if (remainingFailures > 0) {
			remainingFailures -= 1;
			console.log(`receiver: ${id} ${type} -> 503`);
			return c.text("unavailable", 503);
		}
		// Skip ids already handled, since a delivery can arrive more than once.
		const duplicate = seen.has(id);
		seen.add(id);
		console.log(
			`receiver: ${id} ${type} -> 200${duplicate ? " (duplicate, skipped)" : ""}`,
		);
		return c.text("ok");
	});

	const server = serve({ fetch: app.fetch, port: opts.port });
	return {
		// Makes every later request succeed.
		recover: () => {
			remainingFailures = 0;
		},
		close: () => server.close(),
	};
}

npx tsx client.ts

The first event fails 3 times, 2, 4, and 8 seconds apart, then gets through. The second event waits behind it and is delivered right after:

endpoint ep_83460a4b-7b27-416c-acf1-856402c9b7c2
receiver: msg_e202e0a2-9126-4a90-8cb8-189d9f9b6d0e order.created -> 503
08:59:47 msg_e202e0a2 attempt 1: 503 (17ms)
receiver: msg_e202e0a2-9126-4a90-8cb8-189d9f9b6d0e order.created -> 503
08:59:49 msg_e202e0a2 attempt 2: 503 (3ms)
receiver: msg_e202e0a2-9126-4a90-8cb8-189d9f9b6d0e order.created -> 503
08:59:53 msg_e202e0a2 attempt 3: 503 (3ms)
receiver: msg_e202e0a2-9126-4a90-8cb8-189d9f9b6d0e order.created -> 200
receiver: msg_1e4e1568-28a1-498c-9d9f-bb95c83215e9 order.created -> 200
09:00:01 msg_e202e0a2 attempt 4: 200 (3ms)
09:00:01 msg_1e4e1568 attempt 1: 200 (2ms)
both events delivered

Run it with FAIL_FIRST=100 npx tsx client.ts to watch the endpoint run out of retries after 6 attempts and get disabled. The client then fixes the receiver and calls enable, and both events are delivered in order.

Next steps

Handle Incoming Webhooks: the receiving side, with signature checks and deduplication.
Failure & Recovery: alert your team or email the customer when an endpoint is disabled.
Patterns: broadcast each attempt so the portal updates live.

Steps: how tryStep returns failures as data.
Timers & Concurrency: step timeouts, retries, and backoff.
Queues & Messaging: waiting on a queue for a control message, like enable.
Reference: recommendations for replay-safe workflows.