Observability

Railhook exports Prometheus metrics, ships alert rules and Grafana dashboards, and writes structured logs.

Where the metrics are

Prometheus scrapes /actuator/prometheus on a management port, separate from the application port:

Service	Application port	Management port
API	8080	8082
Worker	none	8081

Scrape the management port, not 8080

On the application port, /actuator/** requires a JWT or an API key, and Prometheus has neither. Scraping 8080 returns 401, and every alert rule then fires on missing data. The management port is not published to the host; only containers on the same network reach it.

On Kubernetes the Helm chart wires this up: its ServiceMonitor scrapes the port named management, and a PrometheusRule carries the alert rules. See Kubernetes.

Run the monitoring stack with Docker Compose

monitoring/ in the repository contains Grafana, Prometheus, Alertmanager, Loki and Promtail, node-exporter, cAdvisor and blackbox-exporter, preconfigured for Railhook.

1. Add a Grafana password to .env. There is no default, and Grafana refuses to start without one:

   echo "GRAFANA_ADMIN_PASSWORD=$(openssl rand -base64 24)" >> .env

2. Set a receiver for alerts: ALERTMANAGER_EMAIL_TO (sent through your SMTP_* relay), ALERTMANAGER_SLACK_WEBHOOK_URL, ALERTMANAGER_WEBHOOK_URL, or ALERTMANAGER_TELEGRAM_BOT_TOKEN and ALERTMANAGER_TELEGRAM_CHAT_ID. With none set, alerts show only in Grafana.

3. Start the stack: make monitoring-up in a clone, or on a server installed with install.sh the docker compose -p railhook-monitoring command in Monitoring.

Component	Address	Notes
Grafana	127.0.0.1:3001 (GRAFANA_PORT)	Overview, Alerts, Errors, Logs, Host, Containers, Uptime, Worker & Circuit Breaker, JVM & Micrometer
Prometheus	Inside the stack only	Keeps PROMETHEUS_RETENTION, default 15d
Alertmanager	Inside the stack only	CRITICAL alerts route faster than warnings; a firing critical alert suppresses the matching warning
Loki	Inside the stack only	Keeps logs for LOKI_RETENTION_PERIOD, default 168h (7 days)

Monitoring covers serving Grafana on its own domain and the host, container, uptime and log alerts.

The numbers that matter

Is work getting through?

Metric	Reading it
events_ingested_total	Events accepted, tagged direction (outgoing from /api/v1/events, incoming from /ingress)
deliveries_total	Deliveries created by fan-out. Divided by events, the average subscriptions per event.
webhook_delivery_attempts_total	Attempts. Far more attempts than deliveries means retries are doing a lot of work.
webhook_delivery_latency_ms	Time the endpoint took to answer. A rising p95 is the endpoint’s problem.
incoming_events_received_total, incoming_forward_attempts_total	The same pair for incoming webhooks

Is anything piling up?

Metric	Reading it
delivery_oldest_pending_age_seconds	The best single health signal: the age of the oldest unresolved delivery. A dead endpoint and a stuck worker both show up here. forward_oldest_pending_age_seconds is the incoming twin.
delivery_queue_depth, incoming_forward_queue_depth	Work waiting. Steady is fine; always rising is not.
webhook_dlq_depth, incoming_forward_dlq_depth	Failed messages waiting for a person
delivery_escalated_to_dlq_total	Deliveries moved to failed messages by the 96-hour hard cap rather than by running out of retries. Non-zero means something was degraded for days.

Is a protection interfering?

When a limiter or circuit breaker holds a delivery back, nothing is sent and no retry is used up. A high rate here is the platform protecting an endpoint, not an error.

Metric	Reading it
circuit_breaker_rejected_total	Attempts the breaker held back
circuit_breaker_state_transitions_total, circuit_breaker_slow_trips_total	How often it tripped, and whether on failures or slowness
circuit_breaker_degraded_total	Redis was unreachable and the breaker let calls through unprotected. Non-zero means a safety net is down.
webhook_concurrency_rejected_total, webhook_rate_limit_exceeded_total, webhook_project_rate_limit_exceeded_total	Which limit is holding deliveries back

Are retries and ordering coping?

Metric	Reading it
webhook_ordering_buffered_total	Deliveries waiting for earlier ones to the same endpoint
webhook_ordering_gap_timeout_total	An earlier delivery never resolved, so a later one was released to keep the endpoint moving. Order was knowingly broken. Rare is expected; frequent is not.
transform_failed_total	A transformation failed
events_duplicate_total, events_fanout_limited_total	Idempotency keys catching repeats; events refused for fanning out too far

If you alert on only three things

1. delivery_oldest_pending_age_seconds above a few hours. Catches almost everything.
2. circuit_breaker_degraded_total increasing. Catches protections that stopped protecting.
3. forward_oldest_pending_age_seconds above a few hours. Catches incoming forwards that stopped moving, which the delivery metric does not see.

Shipped alerts

The same rules ship in monitoring/prometheus/alerts.yml for Compose and in the Helm chart’s PrometheusRule for Kubernetes. A test in the build keeps the two identical.

What it means	Alerts
The backlog is growing	DeliveryPendingBacklogGrowing, DeliveryPendingBacklogHigh, DeliveryPendingBacklogCritical, IncomingForwardPendingBacklogHigh, OldestPendingDeliveryStale, OldestPendingDeliveryCritical, OldestPendingForwardStale
Deliveries are being abandoned	DlqDepthGrowing, DlqRateHigh, IncomingForwardFailureRateHigh, DlqActionableBacklog, IncomingForwardDlqActionableBacklog
A protection is engaging, or has failed	CircuitBreakerTripsHigh, CircuitBreakerRejectionsHigh, CircuitBreakerDegraded
The platform is struggling	ApiErrorRateHigh
A process is down	ApiDown, WorkerDown

DlqActionableBacklog and IncomingForwardDlqActionableBacklog are the ones that need a person: they clear when someone retries or purges the failed messages. CircuitBreakerDegraded is the odd one out among the breaker alerts: it means the breaker is not working.

Logs

In production the API and worker log single-line JSON that includes the correlation id and, for authenticated requests, the organization id. The worker’s delivery log lines name the delivery id, which is how you get from a Grafana panel to a delivery in the dashboard. The Compose monitoring stack ships only the api and worker logs to Loki, with level as the only label. Filter by the other ids with | json in Grafana; the Logs dashboard has variables for correlation id and organization id.

Tracing

There is none. Railhook has no OpenTelemetry export, so a slow delivery cannot be followed from ingestion to the attempt as one trace.