Monitoring

Health checks are on the public port, Prometheus metrics on internal ports. An optional monitoring stack (Grafana, Prometheus, Alertmanager, Loki, and exporters for the host, containers and uptime) starts next to Railhook with one Compose command.

Health checks

RAILHOOK_URL is your instance’s address: http://localhost after a default install, http://localhost:<port> if you passed --port, https://<your domain> with --domain.

curl -f "$RAILHOOK_URL/actuator/health/liveness"
curl -f "$RAILHOOK_URL/actuator/health/readiness"

nginx proxies only the health paths. Every other /actuator/ path answers 404 on the public port.

Metrics

Metrics are on management ports inside the Docker network only, with no credentials:

Service	Metrics URL
API	http://api:8082/actuator/prometheus
Worker	http://worker:8081/actuator/prometheus

They are not proxied to the public port because they would reveal endpoint names and traffic volume.

Both services always publish them, so any existing Prometheus can scrape these ports: HTTP latency buckets and counters for events ingested, deliveries created, rules matched and retention cleanup. Labels stay bounded: no organization, project or endpoint ids.

The bundled stack

monitoring/ in the repository is a second Compose project: Grafana, Prometheus, Alertmanager, Loki and Promtail, node-exporter, cAdvisor and blackbox-exporter. It is off until started; its memory limits add up to 1.5 GB.

Only Grafana publishes a port, 127.0.0.1:3001. Prometheus, Alertmanager and Loki stay inside the stack’s network; Grafana’s Explore and Alerting pages read all three.

On a server installed with install.sh

1. Set a Grafana password. There is no default: Grafana refuses to start without one, or with fewer than 16 characters.

   cd /opt/railhook
   echo "GRAFANA_ADMIN_PASSWORD=$(openssl rand -base64 24)" >> .env

2. Say where alerts go. Email uses the platform’s relay (EMAIL_FROM, SMTP_*) unless you set ALERTMANAGER_SMTP_*. Each mail links to Grafana on MONITORING_DOMAIN; without a domain it explains the SSH tunnel. Info alerts, such as a non-empty DLQ, are not mailed: they can stay up for days, so Grafana shows them instead.

   echo "ALERTMANAGER_EMAIL_TO=you@example.com" >> .env

   Slack (ALERTMANAGER_SLACK_WEBHOOK_URL), a generic webhook (ALERTMANAGER_WEBHOOK_URL) and Telegram (ALERTMANAGER_TELEGRAM_BOT_TOKEN, ALERTMANAGER_TELEGRAM_CHAT_ID) work the same way.

3. Download monitoring/ for the release the server runs, and start it. It is its own Compose project, named railhook-monitoring, reading the platform’s .env:

   v=$(grep '^API_IMAGE_TAG=' .env | cut -d= -f2)
   curl -fsSL "https://github.com/vadymkykalo/railhook/archive/refs/tags/v${v}.tar.gz" \
     | tar xz --strip-components=1 --wildcards '*/monitoring/*'
   docker compose -p railhook-monitoring --env-file .env -f monitoring/docker-compose.yml up -d

   It joins the platform’s network, railhook_webhook-network when the install directory is named railhook. For any other directory, set RAILHOOK_NETWORK in .env to the name docker network ls shows. Set MONITORING_NODENAME to the server’s name for the Host dashboard.

   After ./railhook upgrade, download monitoring/ again for the new release and run the same up -d with --force-recreate. The containers mount the files they started with, so without it they keep the old rules and dashboards.

4. Open Grafana through an SSH tunnel, and sign in as admin with the password from .env:

   ssh -L 3001:127.0.0.1:3001 root@your-server

   Then open http://localhost:3001.

The same command with ps, logs -f [service] or down in place of up -d lists the containers, follows the logs, or stops the stack. down keeps its metrics and logs in the railhook-monitoring_* volumes.

Grafana on its own domain

Set MONITORING_DOMAIN=grafana.example.com in .env, point that name at the server, and rewrite the Caddyfile with the installer’s --refresh:

curl -fsSL https://railhook.io/install.sh | bash -s -- --refresh --dir /opt/railhook

It adds the name to the Caddyfile and reloads Caddy, which gets a certificate as it did for the platform. ./railhook upgrade keeps the block. With --behind-proxy, point your own proxy at 127.0.0.1:3001 instead.

Caution

Put an identity-aware proxy such as Cloudflare Access in front of that name, so Grafana’s login is not the only barrier to your logs. If the proxy covers every path, exempt /.well-known/acme-challenge/, or Caddy cannot renew the certificate.

Cloudflare with Always Use HTTPS sends the Let’s Encrypt challenge to HTTPS, where there is no certificate yet. Give Caddy one instead (for example a Cloudflare origin certificate for that name): copy it and the key into the Caddy data volume, name them in .env, and run the --refresh above again:

docker exec webhook-caddy mkdir -p /data/certs
docker cp grafana.pem webhook-caddy:/data/certs/grafana.example.com.pem
docker cp grafana.key webhook-caddy:/data/certs/grafana.example.com.key
printf '%s\n' MONITORING_TLS_CERT=/data/certs/grafana.example.com.pem \
  MONITORING_TLS_KEY=/data/certs/grafana.example.com.key >> .env

From a clone

1. Start Railhook with make up, then add GRAFANA_ADMIN_PASSWORD to .env.

2. Start the stack:

   make monitoring-up

3. Open Grafana at http://localhost:3001.

make monitoring-down stops it; make monitoring-logs follows its logs.

Dashboards

Dashboard	What it shows
Railhook — Overview	Whether the public site is up, firing alerts, disk, memory, CPU and backup age, then events, deliveries, queues and the DLQ
Alerts	Every firing and pending alert, whether the alerting pipeline itself works, alert history, and links to silence one
Errors	Error and warning log lines by service, the most frequent error messages, API and nginx 4xx and 5xx, 401/403/429, failed deliveries and DLQ growth
Logs	Lines from the API, worker, UI’s nginx, Caddy, Postgres and backups, filtered by service, level, text, correlation ID or organization ID
Host	Node Exporter Full: CPU, memory, disks, network, file systems
Containers	CPU, memory against the Compose limit, network and restarts per container
Uptime	Probe status, uptime over the range, response time, TLS days left, backups
Worker & Circuit Breaker, JVM & Micrometer	The platform’s internals

Alerts it sends

The platform rules are the ones the Helm chart ships. Server rules on top:

Alert	Fires when
HostDiskAlmostFull, HostDiskCritical	A file system is over 85%, or over 95%
HostMemoryHigh, HostOomKill	Memory is over 90% for 10 minutes, or the kernel killed a process
HostLoadHigh	The 15-minute load is over 1.5 times the CPU count
ContainerRestarting, ContainerOomKilled, ContainerMemoryNearLimit	A container restarted twice in 15 minutes, hit its memory limit, or is above 90% of it
PublicEndpointDown, PublicEndpointSlow	A public URL fails, or takes over 3 seconds, from outside
UiDown, CaddyDown	The UI’s nginx or Caddy stops answering inside the network
TlsCertificateExpiringSoon, TlsCertificateExpiryImminent	A certificate expires in under 14 days, or under 3
ErrorLogSpike, EdgeServerErrorSpike	A service logs more than 10 errors a minute, or nginx answers more than 5 5xx a minute
BackupStale, BackupMissing	The newest scheduled backup is over 26 hours old, or there is none
Watchdog	Always. It is sent only to ALERTMANAGER_HEARTBEAT_URL

The probes request https://RAILHOOK_DOMAIN/, /docs/ and /actuator/health; set MONITORING_PROBE_URLS to probe other URLs.

Tip

Every other alert comes from the server it describes, so none can report that server down. Point ALERTMANAGER_HEARTBEAT_URL at a heartbeat service such as healthchecks.io: it gets the Watchdog every minute and alerts you when it stops.

The stack finds the platform’s Docker network by its Compose label; set RAILHOOK_NETWORK if it picks the wrong one. Every variable is in the configuration reference.

What to alert on

Metric	Why
delivery_oldest_pending_age_seconds	The age of the oldest unfinished delivery. Every kind of trouble shows up here. forward_oldest_pending_age_seconds is the incoming twin
webhook_dlq_depth, incoming_forward_dlq_depth	Deliveries and forwards that ran out of retries and need a person
circuit_breaker_degraded_total	Redis was unreachable and the breaker let calls through unprotected

Kubernetes

Turn on the chart’s integrations with kube-prometheus-stack:

values.yaml

monitoring:
  serviceMonitor:
    enabled: true
  prometheusRule:
    enabled: true
  grafanaDashboards:
    enabled: true

The ServiceMonitor scrapes the management port of the API and the worker.