Skip to content
Back to site

Retries and failed messages

Railhook retries a delivery on a fixed schedule, the retry ladder, until the endpoint answers 2xx or the ladder runs out. What it gives up on goes to Failed Messages.

Status Meaning
PENDING Created, or waiting for its next attempt to be due.
PROCESSING Claimed by a worker for one attempt. If the worker dies, another worker claims the delivery again once the claim times out (WEBHOOK_CLAIM_TIMEOUT_SECONDS, 300 by default).
SUCCESS The endpoint answered 2xx. Nothing more is sent.
FAILED Given up on because nobody can make another attempt help: the URL is refused, or the endpoint cannot receive it.
DLQ The ladder ran out, the endpoint answered with a status another attempt will not change, or the delivery outlived the age cap. Shown in the dashboard as Failed Messages.
Result of the attempt What happens
2xx SUCCESS.
408, 429, any 5xx Retried on the ladder. This is the default set. A subscription or an incoming destination can change it: see Which statuses are worth retrying.
A connection error, a timeout, or a host that does not resolve Retried on the ladder.
The transformation failed to apply Retried on the ladder. The raw payload is never sent instead.
Any other 4xx, or a 3xx DLQ on the first attempt, without the rest of the ladder. Fix the credentials or the URL, then retry from Failed Messages. Redirects are not followed.
The endpoint cannot receive it, for example it is not verified, or its URL is refused by the private address check FAILED without an attempt.

Each wait is jittered between 50% and 150% of its value, so retries do not arrive in a burst.

Direction Attempts Waits between attempts Nominal total
Outgoing (a delivery) 7 1 min, 5 min, 15 min, 1 h, 6 h, 24 h about 31 h 21 min
Incoming (a forward) 5 1 min, 5 min, 15 min, 1 h about 1 h 21 min

A subscription or an incoming destination can set its own maxAttempts (1 to 20) and retryDelays in seconds. When there are more attempts than delays, the last delay repeats.

A tenant or per-endpoint rate limit, full concurrency slots, an open circuit breaker, or an ordered delivery waiting for an earlier one stops the request before it is made. That is a deferral: it does not use an attempt or move the ladder, the delivery is just scheduled again.

One attempt waits 30 seconds for a response unless the subscription sets timeoutSeconds, which must be between 1 and 60. A timeout is a failed attempt.

By default a delivery is retried on 408, 429 and every 5xx, and nothing else. A subscription or an incoming destination can change this with retryableStatuses, a comma-separated list of terms:

Term Matches
429 Exactly that status.
500-599 An inclusive range.
5xx Shorthand for 500-599. Also 4xx, 3xx, 2xx, 1xx.
>=500, >500, <=408, <400 Open at one end.
!501 Excluded. Any term can be prefixed with !.

Exclusions win wherever they are written: 5xx,!501 and !501,5xx mean the same.

Examples
408,429,500-599 the default
5xx,!501 retry every server error except "not implemented"
>=500,408,429,409 a receiver that answers 409 while it is still catching up
408,429 a receiver whose 5xx means the payload was rejected for good

The API rejects a list it cannot read and names the bad term. A status the list does not cover is treated like a 4xx: the delivery goes to Failed Messages on the first attempt.

A receiver answering 429 or 503 can set Retry-After. Railhook accepts both HTTP forms: seconds or an HTTP date. Three rules bound it:

  • Only on 429 and 503. Only there does it mean “over capacity, wait”. On a 500 it is often a header a proxy left on.
  • It can only push an attempt later. A receiver cannot make Railhook retry harder than the subscription owner configured.
  • It is clamped, by WEBHOOK_RETRY_AFTER_MAX_SECONDS (default 21600, six hours). Otherwise one misconfigured response could park a delivery for a year.

Six hours is the largest step both default ladders share, and seven attempts at that maximum still fit inside the 96-hour age cap. A longer wait (Hookdeck allows seven days) would be cut short by the cap anyway. Set the variable to 0 to ignore the header.

An honoured Retry-After does not add an attempt. A 429 on the last attempt goes to Failed Messages like any other failure.

A delivery still outstanding 96 hours after it was created is moved to Failed Messages whatever its attempt count. A delivery retried or resent by hand is measured from that moment, so it gets its new attempts. For incoming forwards the cap is 24 hours. The cap catches deliveries that stopped moving; the ladder ends the ones that keep failing.

Variable Default Direction
DELIVERY_ESCALATION_HARD_CAP_HOURS 96 Outgoing
FORWARD_ESCALATION_HARD_CAP_HOURS 24 Incoming

The worker refuses to start if a default ladder, at the top of its jitter, would not fit inside its cap.

Failed Messages lists each abandoned delivery with its endpoint, event type, attempt count and last error. API: GET /api/v1/projects/{projectId}/dlq.

Retrying from Failed Messages puts the same delivery back to PENDING and gives it 3 more attempts. It keeps its delivery id, attempt history and Idempotency-Key, so a receiver that already processed it can detect the duplicate. The first new attempt is sent at once. Retry one delivery with POST /api/v1/projects/{projectId}/dlq/{deliveryId}/retry, or several with POST /api/v1/projects/{projectId}/dlq/retry.

For incoming webhooks the same screen is called Failed Forwards. Retrying one creates a new forward to the same destination, starting from attempt 1.