Monitoring#

Requirements: OPS-001, OPS-002, OPS-003, OPS-004, OPS-011, OPS-012, OPS-015, SEC-015, AUD-011, AUD-012, AUD-013, AUD-014, EXT-005, EXT-015, EXT-017, DEP-009. Code: internal/observability/observability.go, internal/server/server.go (InternalHandler, refreshGauges), internal/httpapi/protocol.go (live, ready), internal/httpapi/middleware.go, internal/worker/worker.go (VerifyRecentAudit), internal/platform/tracing (span export).

Endpoints#

The internal listener (server.internal_listen) must be on a private network: /metrics and /status are protected only when server.metrics_token is set (if that reference cannot be resolved at startup, both answer 401 to every request), and /health/live never is. identity-worker has no listener unless workers.metrics_listen is set; it then serves the same /metrics, /status and /health/live on that private address, under the same token and metrics_enabled rules.

Operator status#

GET /status (operatorStatus in internal/server/server.go, TestMaintenanceAndStatus) always answers 200 with Cache-Control: no-store (or 401 without the token):

{
  "status": "degraded",
  "time": "2026-10-10T12:00:00Z",
  "capabilities": {"database": "ok", "schema": "ok", "signing": "ok", "outbox": "ok",
                   "email_delivery": "degraded", "webhooks": "ok", "jobs": "ok",
                   "audit_sealing": "ok"},
  "backlog_oldest_seconds": {"outbox": 0, "email_delivery": 412, "webhooks": 0, "jobs": 0,
                             "audit_sealing": 21},
  "clock_skew_ms": 3,
  "maintenance": {"mode": "off", "retry_after_seconds": 300},
  "key_provider": "kms_unwrapped",
  "residency": {"region": "eu-central", "destinations_restricted": true}
}

/status is meant for operators: it names internal capabilities and must not be exposed publicly. Anonymous clients only ever see the generic maintenance or unavailable problem responses.

Metrics#

All metrics are per process (identity-server, or identity-worker when workers.metrics_listen is set); aggregate across instances in your monitoring system. Labels never contain user IDs, emails or realm names: the route label (also in request logs and spans) is the registered route pattern without the method, for example /api/v1/realms/{realm}/users/{user}, and unmatched for requests that match no route (OPS-002), so path values such as slugs or external IDs never become labels. Each counter keeps at most 1000 label combinations and each histogram 500; further combinations are dropped.

Set or incremented only inside identity-worker, so meaningful only through its workers.metrics_listen listener (the server's /metrics never updates them; the help text of proudauth_jobs_failed_total says so):

• proudauth_worker_runs_total (task, outcome): every periodic task run, for example audit.seal, audit.archive, audit.verify_recent, retention.cleanup or groups.dynamic, with ok or error. lifecycle.expiry (every 30 seconds) removes expired role assignments (at most 500 per run), which ends just-in-time elevations at their deadline, and closes and applies access review campaigns past their due date (at most 20 per run; elevation-and-access-reviews.md). A retention.cleanup error names each failed step; the other steps still ran. The lifecycle workflow tasks are workflows.run (every second: advances due runs under leases) and workflows.schedule (every minute: fires schedule triggers); event-triggered runs start in outbox.dispatch (workflows.md).
• proudauth_security_events_total (type): outbox events whose type starts with security., including security.alert.
• proudauth_delivery_total (channel, outcome): final outcomes of deliveries: channel="webhook" (delivered, dead_letter), channel="notification" (email and SMS jobs: accepted by the relay or gateway, failed, expired) and channel="backchannel_logout" (delivered, failed). Retries are not counted, and neither are back-channel deliveries dropped because the client or its URI no longer exists or bounces reported later through POST /delivery-events.
• proudauth_audit_integrity_problems (gauge): problems found by the last run of audit.verify_recent (hourly; the newest 200 sealed batches: event content hashes, batch digests, links, and signatures where present). A non-zero run also increments proudauth_security_signals_total{signal="audit_failure"} and logs audit integrity problems.
• proudauth_jobs_failed_total (type): asynchronous jobs (exports, imports, bulk operations, organization deletion) that reached dead_letter, by job type. Retried failures are not counted. type="workflow" counts lifecycle workflow runs that ended failed, compensated or compensation_failed. There are no other workflow metrics: /status has no workflow backlog and proudauth_worker_active_leases does not count workflow runs; use GET /workflows/{id}/diagnostics.

The queue-age gauges are refreshed by whichever listener is scraped (the server's internal listener or the worker's workers.metrics_listen), so either one shows them. Not implemented from OPS-002: a metric for spans the trace exporter dropped.

Directory integration#

PROV-018, internal/directory/metrics.go (directory.md, outbound-provisioning.md). Labels are bounded enumerations; connector IDs, hosts and users never become label values.

Worker tasks (also in proudauth_worker_runs_total): directory.schedule (every 15 seconds, before jobs.run: claims due synchronization and reconciliation slots and queues directory.sync jobs, scans for changed accounts for outbound connectors, refreshes the two gauges) and directory.outbound.deliver (every 2 seconds). Runs are asynchronous jobs of type directory.sync, so a failed one also shows in proudauth_jobs_failed_total{type="directory.sync"}. Per connector, GET .../directory-connectors/{id}/health gives status, lag_seconds, consecutive_failures, last_error, endpoint health and delivery counts.

SAML has no metrics of its own: use the audit events saml.request.rejected and saml.response.rejected (with their reasons), saml.assertion.issued and proudauth_http_requests_total for the /realms/{id}/saml/... routes (saml.md).

Shared signals#

AUD-016, ADV-007 (shared-signals.md). Security Event Tokens have no metric of their own and are not counted in proudauth_delivery_total.

• Worker task ssf.deliver (every 2 seconds, up to 50 due push events per run) appears in proudauth_worker_runs_total{task="ssf.deliver"}; an error outcome means the queue query failed, not that a receiver refused a token. Failed and expiring events show in the per-stream view GET /api/v1/realms/{realm}/ssf-streams/{id}/events (state, attempts, last_status_code, last_error such as rejected_<err>, http_<status>, network_error, egress_denied, transmitter_disabled).
• retention.cleanup steps ssf_events (expires events undelivered after 7 days, deletes finished ones after 14) and ssf_received_events (deletes received-event records after 30 days).
• Inbound pushes: audit events ssf.event.received (with outcome) and ssf.event.replayed; per transmitter, GET /api/v1/realms/{realm}/ssf-transmitters/{id}/events. Rejected pushes (bad secret, signature, issuer, audience or age) are answered to the transmitter and leave no record; watch proudauth_http_requests_total for the receiver route's 4xx.

SELECT s.id, s.delivery_method, s.status, e.state, count(*)
  FROM ssf_events e JOIN ssf_streams s ON s.id = e.stream_id
 GROUP BY 1, 2, 3, 4;                                                    -- watch failed and expired
SELECT min(occurred_at) FROM ssf_events WHERE state = 'pending';         -- oldest undelivered event

Logs#

• Structured log/slog output to stderr, JSON by default (observability.log_format), level from observability.log_level (debug is refused in production). Records below that level are emitted only inside an operator debug window (identity-cli debug-window, SEC-015), optionally for one component, and still pass through redaction (support-bundle.md).
• Every record carries component (default server; http for request lines, audit for security signals) and event_code (an explicit code such as http.request or the audit event type, otherwise the log message).
• Redaction runs before serialization: attributes whose key matches password, secret, token, authorization, cookie, otp, code_verifier, code, seed, private, assertion, pepper or api_key are replaced with [REDACTED], and values that look like pa_* bearer secrets or JWTs are masked.
• One http line per request (component: http, event_code: http.request): outcome (success, client_error, rate_limited, server_error), request_id, trace_id, span_id, method, route (bounded label, not the raw path or query), status, duration_ms, network (client IP reduced to /24 or /48).
• X-Request-Id: accepted from the client when 1–64 printable ASCII characters, otherwise generated; echoed in the response and stored in audit events (request_id). Use it to join logs and audit.
• A valid incoming W3C traceparent continues that trace. Its sampled flag is honored only when the request comes directly from an address in server.trusted_proxies; otherwise, and for requests without one (which get a new trace ID), the request is sampled with probability observability.trace_sampling. Sampled requests emit an extra span log line (trace_id, span_id, name = method and route, status, start, duration_ms). Outbound HTTP calls made while serving a request (pre-action hooks, upstream identity providers' discovery, JWKS, token and user endpoints, challenge-provider verification, the identity-provider test call) carry a W3C traceparent with the request's trace ID, its span ID as the parent ID (or, when spans are exported, a new client span's ID) and its sampled flag, and never tracestate or baggage (API-012, egress.traceTransport, TestCorrelationPropagation). Calls the worker makes (webhook deliveries, back-channel logout, SMS gateway) carry a traceparent only during a sampled task run, and SMTP is not HTTP. Spans can be exported to an OpenTelemetry collector (Tracing).
• Webhook events carry correlation_id, the X-Request-Id of the request that caused them, equal to the request_id of its audit events; events from background work have none (api/events). Clients choose their own X-Request-Id, so it is a correlation aid, not evidence.
• Notable messages: request failed (5xx and unexpected errors), panic, worker task failed (with task), security event (warning, with event_type, logged by the worker for outbox events whose type starts with security.), security signal (warning, component: audit, event_code = audit event type, signal, outcome, realm_id, org_id, request_id), audit persistence failed (error), audit integrity problems (error, component: worker, event_code: audit.integrity_failed, problems), notification delivery failed, DEVELOPMENT MODE: not for production use.

Tracing#

OPS-003, internal/platform/tracing, TestOTLPExporter, TestTraceExport. identity-server records spans of sampled requests (see traceparent above) and identity-worker of sampled task runs; when observability.otlp_endpoint is set, both export them as OTLP/HTTP JSON with service.name proudauth-server or proudauth-worker (configuration.md):

• Spans carry no headers, bodies, query strings or tokens. http.route is the route label used for metrics: ID-shaped path segments and the realm name become {id}, but other path segments stay, for example external-ID values in /external-ids/{source}/<value> and organization slugs or aliases where a path accepts them instead of an ID. Treat the collector as holding operational metadata of your deployment.
• Unsampled requests export nothing and keep propagating traceparent as before. An incoming sampled flag counts only from server.trusted_proxies, so other clients cannot make the server export spans for their requests; an incoming "not sampled" flag does not prevent local sampling either.
• Export is asynchronous: batches of up to 256 spans every 5 seconds from a queue of 4096. When the queue is full, the collector does not accept a batch or the process is shutting down, spans are dropped (never delaying requests); drops are counted only inside the exporter, not as a metric.
• Worker runs started with identity-worker -once export no spans, and the worker writes no span log line; the server writes it whether or not spans are exported.
• The exporter uses its own HTTP client, not the egress policy: it ignores proxy environment variables, treats a redirect as a failed export, and its endpoint must be https in production (loopback excepted) and is checked against privacy.residency.allowed_destinations only at startup. Use an endpoint on your own network.
• Other tracing back ends can be attached in code through the tracing.Tracer interface (tracing.SetTracer).

Suggested alerts#

Audit queries#

Audit events are queried with GET /api/v1/realms/{realm}/audit-events (realm.audit.read) or, limited to one organization, GET /api/v1/realms/{realm}/organizations/{org}/audit-events (org.audit.read), or exported as NDJSON with POST /audit-exports. Results are newest first and paginated with limit and cursor. Filters (AUD-011, TestAuditSearchDimensions; all optional and combined with AND):

Audit sequence cursors are MAC-bound to the realm, authorized organization, route and URL-encoded filter parameters (API-004). Keep the filters when following next_cursor; only limit may change. A changed filter or route gets 400 invalid_request, instead of silently skipping evidence. Older development-build audit cursors must restart from the first page after upgrading. TestAuditTypePrefixIsLiteral and TestAuditCompoundFiltersAndCursors verify literal matching and exact compound pages, including newer inserts, foreign scope and cursor changes.

Events are always limited to the realm. Operator events written by identity-cli or at startup (operations.*, for example operations.migrations.applied; category operations or, for restore reconciliation, emergency) and credential.password.legacy_retired carry no realm and therefore never appear here; read them from the audit_events table. A correction of an event (audit.correction, AUD-007) is found with type=audit.correction&target_id=<event id> (data-inventory.md). Events with a security signal are pushed as security.alert webhook events (below); for anything else, poll the API or read the database.

Each event carries source: server for evidence ProudAuth recorded itself, application for activity reported by a service (below). Audit history imported from another system (MIG-016) is not in these results; it is listed by GET /imported-audit-events with source: "import" and verified: false (exit-and-export.md), and each import request appears here as one audit.history.imported event (category audit).

Security alerts#

AUD-012, internal/audit/audit.go (SecuritySignal), internal/runtime/deps.go, TestSecurityAlerts. Every audit event of a realm that carries a security signal is also written to the outbox as the webhook event security.alert, in the same transaction, so an alert exists exactly when its evidence committed. Operator events without a realm raise no alert. The alert holds identifiers only: signal, audit_event_id, event_type, actor_type, actor_id and, when the event has a subject, subject_type and subject_id; read the audit event for details. org_id is set when the event belongs to an organization, so organization-scoped subscriptions receive their organization's alerts.

Both payload forms carry every member of the alert, so the default minimal form is enough (api/events/README.md). Fetch the audit event with GET /audit-events?id=<audit_event_id> (or the organization route for organization-scoped subscriptions). The signals are those of proudauth_security_signals_total except audit_write_failed and audit_failure, which have no audit event. The ones added for AUD-012 are:

Reported application activity#

AUD-013, internal/httpapi/emergency.go (reportedActivityRoutes), TestReportedApplicationActivity. Services can file their own activity in the realm's audit log, apart from ProudAuth's evidence:

POST /api/v1/realms/{realm}/activity-events
{
  "type": "app.order.refunded",
  "occurred_at": "2026-10-11T09:30:00Z",
  "organization_id": "org_...",
  "subject": {"type": "user", "id": "usr_..."},
  "outcome": "success",
  "metadata": {"order": "ord_1"}
}

• The caller must be a service principal (a client-credentials token for the administration audience) holding realm.activity.report; user tokens get 403. Grant the permission through a custom realm role.
• type is app. followed by 1–96 lowercase letters, digits, dots and underscores, so reported events never share a type with server events. occurred_at (default: now) must be within the last 24 hours and at most 5 minutes ahead; organization_id must name an existing organization of the realm; subject.type is user or service_principal (the ID is not checked); outcome is success (default), failure or denied; metadata is at most 4 KiB as JSON and is redacted like server metadata.
• The event is recorded with source: "application", category application, the service principal as actor and reporting_service and client_id set to the caller's client ID; it is sealed into the hash chain like any other event. The response is 202 {"id", "source", "reporting_service"}.
• Query them with GET /audit-events?category=application or ?type_prefix=app.. They are claims by the reporting service, not evidence that ProudAuth observed the action.

Database queries for delivery health#

SELECT state, count(*) FROM notification_jobs GROUP BY state;
SELECT state, count(*) FROM webhook_deliveries GROUP BY state;            -- watch dead_letter
SELECT state, count(*) FROM logout_deliveries GROUP BY state;             -- watch failed
SELECT type, state, count(*) FROM jobs GROUP BY type, state;              -- watch dead_letter
SELECT min(occurred_at) FROM outbox_events WHERE state = 'pending';