Incident runbooks#

Runbooks required by docs/plan.md §31.3 and OPS-016. API paths are relative to /api/v1/realms/{realm} and need an admin-audience token from an MFA session (identity-cli login, then identity-cli api METHOD /path [json]). {user} is a user ID (usr_...); {client} is the client's internal id (cli_...) returned by POST /clients and GET /clients, not its OAuth client_id.

General rules:

• Preserve evidence before changing state: export audit for the window (POST /audit-exports {"from": "...", "to": "..."}, then download the job result), place a legal hold so retention cannot prune it (POST /audit-legal-holds {"reason": "...", "subject_id": "..."}, realm.audit.legal_hold; see data-inventory.md), and keep database and log snapshots.
• Prefer API actions over database edits: they are audited and publish outbox events. Where this page says a database edit is the only option, record it in your incident log, because it bypasses ProudAuth's audit trail.
• Offline-verified JWTs remain valid until expiry whatever you revoke; see revocation-exposure.md.

Runbook coverage (OPS-016)#

Rehearse the untested procedures in a staging environment before you need them.

Compromised signing key#

Signals: a signing private key or the key that wraps it (root key or keys.signing_wrap_keys) was exposed; tokens with valid signatures that the IdP did not issue (no matching token.issued audit event).

Impact while the key is trusted: anyone holding it can mint access tokens for any subject and audience that offline verifiers will accept, and ID tokens that clients will accept. The IdP's own account/admin APIs additionally require a live session ID and an active user and client, but session IDs appear in tokens (sid), so treat admin APIs as exposed too.

1. Rotate now: identity-cli api POST /signing-key-rotations '{"mode":"emergency","reason":"INC-..."}' (permission realm.keys.rotate, authentication within the last 5 minutes). The active key becomes compromised, leaves JWKS, and the IdP rejects its tokens within 5 seconds; a new key is active immediately on the node that handled the request. Other server nodes keep their cached key set for up to 5 seconds: until then they still accept the key's tokens and sign new tokens with it (those are rejected too once the window ends). Repeat per affected realm.
2. If the exposed key is not the active one (a retiring or pending key), the API cannot mark it. Database operator, last resort: UPDATE signing_keys SET state='compromised', compromised_at=now() WHERE kid='<kid>';
3. Make verifiers drop the key. Offline verifiers keep their cached JWKS. Restart them or force a JWKS refresh. The Go SDK verifier refreshes only on an unknown kid; a restart is the reliable option. Subscribed webhook consumers receive signing_key.rotated with previous_key_compromised: true.
4. Record the exposure window per relying service: from key exposure until that service dropped the key. Ordinary token expiry does not bound it.
5. Revoke what may have been abused. Sessions are not signed and need no action for the key itself, but if forged tokens reached admin APIs, review admin audit events for the window and follow "Compromised administrator account".
6. If the root key or wrap key leaked (not only one signing key): every encrypted secret is exposed. Rotate the root key (keys.md), rotate signing keys again so they are wrapped under the new key, rotate every webhook secret, and plan TOTP re-enrollment (assisted recovery with reset_mfa or user self-service).
7. Restores: the compromise is exported by identity-cli revocation-log as key_compromised; see the ordering rule in backup-restore.md.

Compromised root key, pepper or KMS access#

Signals: a root, pepper or signing-wrap key file or variable was exposed; the wrapped key files and credentials allowed to unwrap them with your KMS were exposed (with kms: references, either alone is not enough); the KMS's audit log shows unwraps you cannot attribute to a process start.

1. KMS access only (no unwrapped key left the hosts): revoke the exposed credentials or identity in the KMS and review its log. Running processes are unaffected; new starts need the remaining authorized identities.
2. An unwrapped root or signing-wrap key: every encrypted secret and every signing key wrapped under it is exposed. Generate a new version (wrapped in the KMS if you use one), distribute it, make it active (keys.md), then run an emergency signing-key rotation in every realm so new signing keys are wrapped under the new version (POST /signing-key-rotations {"mode":"emergency",...}), rotate every webhook secret (POST /webhooks/{id}/rotate-secret), re-create hooks, send identity-provider client secrets again (PUT /identity-providers/{id}) after rotating them at the provider, send organization mail-server passwords and connector credentials again, and plan TOTP re-enrollment (POST /me/totp/enroll and /activate replace the authenticator and the recovery codes). Drain the old version with the checks in keys.md (which also list SAML signing keys and shared-signals stream headers) before removing it.
3. A pepper key: add a new pepper version; verifiers are re-peppered at each user's next successful sign-in. Users who do not sign in keep the old version, so to retire it force resets (POST /users/{user}/credentials/password/compromised) for the accounts the query in keys.md still finds.
4. Record the exposure window and treat audit batch signatures made under the exposed version as forgeable from then on: rely on the audit archive and exports taken earlier.

Compromised administrator account#

1. Stop the account: POST /users/{user}/suspend {"reason":"INC-..."} (realm.users.suspend; administrators cannot change their own state). Suspension revokes all sessions, refresh families, offline grants and support grants where the user is actor or target, writes a user revocation record, and makes the user's personal access tokens inactive at introspection. If the compromised account is the only realm administrator, create a new one from the operator channel: identity-cli bootstrap -realm <r> -email ... -username ... -break-glass.
2. Revoke automation the account controls. List credentials: GET /api-keys?owner_type=user&owner_id={user} and API keys of service principals it owns (GET /service-principals, check owner_user_id); revoke with DELETE /api-keys/{id}. Disable service principals it created if in doubt (POST /service-principals/{id}/disable).
3. Remove privileges: GET /role-assignments?subject_type=user&subject_id={user}, then DELETE /role-assignments/{assignment}.
4. Reset credentials: POST /users/{user}/credentials/password/compromised (the password stops working; sessions and grants are revoked) and, if the second factor may be compromised, assisted recovery: POST /users/{user}/recovery-requests {"reason":"...","evidence_reference":"INC-...","reset_mfa":true} (revokes TOTP, passkeys and recovery codes and forces TOTP re-enrollment).
5. Investigate: GET /audit-events?actor_id={user} and ?subject_id={user}; look for client.*, role.*, authorization.role.assigned, webhook.* (exfiltration endpoints), signing_key.*, realm.policy.updated, job.result.downloaded (exports), support.impersonation.*, automation_credential.created, bootstrap.*. Undo unauthorized changes through the API.
6. Reactivate only after recovery: POST /users/{user}/reactivate.

The same steps apply to ordinary user accounts, using organization-scoped actions where the realm-wide account must stay active.

Compromised client secret#

1. Inspect the client: GET /clients/{client} returns its credentials (ID, display prefix, state, last use) and resource grants.
2. Issue a replacement and give the old secrets a short expiry: POST /clients/{client}/credentials {"retire_others_after_seconds": 300} (at most three active secrets). Deploy the new secret to the client.
3. Revoke the exposed secret: DELETE /clients/{client}/credentials/{id}.
4. If tokens may have been obtained with it: POST /clients/{client}/suspend {"reason":"INC-..."} revokes every refresh family, grant and opaque token of the client and makes introspection report its JWTs inactive. Reactivate with POST /clients/{client}/reactivate after the new secret is deployed. JWTs already issued remain valid at offline verifiers until expiry.
5. For private_key_jwt clients, add the new public key (POST /clients/{client}/credentials {"jwks": {...}}) and revoke the old key credential by ID.
6. Review client.authentication.failed and token.issued audit events for the client.

Compromised connector#

A system ProudAuth trusts or sends data to is compromised: an upstream identity provider (or its client secret), a pre-action hook endpoint, a webhook receiver, an organization's mail server, the SMS gateway or a challenge provider.

1. Switch it off without database edits (realm.emergency, reason required; emergency controls):

   shCopy

   identity-cli api POST /emergency/controls '{"target_type":"identity_provider","target_id":"<id>","action":"disable","reason":"INC-..."}'
   identity-cli api POST /emergency/controls '{"target_type":"hook","target_id":"<id>","action":"disable","reason":"INC-..."}'
   identity-cli api POST /emergency/controls '{"target_type":"webhook","target_id":"<id>","action":"disable","reason":"INC-..."}'

   Disabling a registration-eligibility, profile-validation or authorization hook lifts the restriction it enforced; disable it only if the endpoint's answers can no longer be trusted. An organization's mail server: DELETE /organizations/{org}/notification-settings/smtp (MFA session; mail falls back to the deployment relay). The deployment relay, the SMS gateway and challenge providers are operator configuration: change the file and restart, or turn off login.sms_otp and remove the realm's authentication_flow.challenge_provider (PUT /settings) meanwhile.

2. Upstream identity provider: find sign-ins through it in the window: GET /audit-events?type=federation.login.succeeded&from=...&to=... (metadata provider names the connection) and federation.identity.linked events. Revoke those users' sessions (POST /users/{user}/revoke-sessions), remove links created by the attacker (DELETE /users/{user}/linked-identities/{id}), and review accounts it created just in time (created: true in the sign-in event's metadata). Such an account has no other sign-in method, so its link cannot be removed (409 last_sign_in_method): suspend or delete the account instead (POST /users/{user}/suspend). Rotate the client secret at the provider and send it with PUT /identity-providers/{id} (If-Match) before enabling the connection again.

3. Webhook receiver: it received event payloads (identifiers; standard payloads also other data fields) and holds the signing secret. Rotate it (POST /webhooks/{id}/rotate-secret; the old secret stays valid for 24 hours, so disable the subscription if the receiver itself is hostile) and review what it subscribed to.

4. Hook endpoint: hooks have no secret rotation; delete the hook and create a new one (new secret) once the endpoint is trusted again. Review GET /hooks/{id}/diagnostics and the decisions it influenced in the window. A hook or subscription that a plugin manages (it shows a plugin_id) cannot be deleted or patched (409): switch it off with the emergency control or POST /plugins/{id}/disable, and replace it with a new plugin version. Installing or enabling the plugin again turns its parts back on.

5. Lifecycle workflow endpoint (the destination of a call_webhook step's subscription): disable the subscription with the emergency control (steps then fail and their runs compensate) or the workflow with POST /workflows/{id}/disable; the endpoint received subject and run IDs (workflows.md).

6. Plugin publisher key or plugin package: remove the key from extensions.plugin_publishers and restart, which stops uploads, installations and rollbacks signed with it; then disable or uninstall every plugin it signed (GET /plugins shows the publisher), because enabling does not re-check signatures. Review plugin.* audit events and the parts' hook.*, webhook.* and workflow.* events (plugins.md).

7. Record the window; the evidence is in the audit log (federation.provider.*, hook.*, webhook.*, notification.smtp.*, workflow.*, plugin.*).

Provisioning connector token compromise#

A SCIM connector token (pa_scim_...) leaked, or the directory that holds it is compromised (scim-provisioning.md). Paths are relative to /api/v1/realms/{realm}/organizations/{org}; each request needs org.integrations.write in the organization and an MFA session.

1. Stop the token. Either issue a new one (the old token stops working at once; give the new one to the directory only when the directory is trusted):

   shCopy

   identity-cli api POST /organizations/{org}/provisioning-connectors/{id}/token

   or disable the connector. PUT replaces the whole definition, so send the current members back with "state": "disabled" and the ETag of the GET as If-Match:

   shCopy

   identity-cli api GET /organizations/{org}/provisioning-connectors/{id}
   # PUT the returned name, allowed_resource_types, role_ceiling_ids,
   # group_role_mappings, default_role_id and config with "state": "disabled"
   # and the header If-Match: W/"v<N>"

   Maintenance mode restricted refuses both requests. If you cannot make them, suspending the organization with an emergency control (target_type: organization) also stops the connector, but stops the whole organization.

2. Know the limits of what the token could do. A connector acts only in its own organization and only on users and groups it provisioned: it could create accounts and memberships, adopt existing accounts whose verified email is in a domain the organization verified (never protected accounts or holders of realm-scoped roles), change the name, username and email of accounts it created that belong to no other organization, suspend or remove memberships in its organization, and grant roles only through default_role_id and group_role_mappings, within role_ceiling_ids. It cannot change credentials, sessions, the realm-wide account state or anything in another organization.

3. Review its actions in the window. Connector events carry the connector as actor (type connector):

   shCopy

   identity-cli api GET '/audit-events?actor_id=con_...&from=...&to=...'
   identity-cli api GET '/audit-events?type_prefix=scim.&organization_id=org_...&from=...'
   identity-cli api GET '/audit-events?type_prefix=provisioning.connector.&organization_id=org_...'

   Look for scim.user.provisioned with adopted: true (existing accounts brought into the organization), scim.user.updated with identity_changed: true (name, username or email changes), scim.user.deprovisioned, organization.membership.added / .suspended / .removed, group.member.added and role.assigned, and provisioning.connector.* for changes to the connector itself (a wider ceiling or new mappings). Organization administrators can run the same queries on GET /organizations/{org}/audit-events (org.audit.read).

4. Undo unauthorized changes through the API: memberships with PATCH /organizations/{org}/members/{membership} {"state": "active"} or DELETE /organizations/{org}/members/{membership}, roles with DELETE /organizations/{org}/members/{membership}/roles/{assignment} (and group role assignments with DELETE /role-assignments/{assignment}), and identifiers of affected accounts through the user administration API. If an email address was changed, treat the account as possibly taken over (Compromised administrator account, steps 1 and 4).

5. Review the connector's role_ceiling_ids and group_role_mappings before enabling it again, and rotate the token as the last step.

SAML key or counterparty compromise#

See saml.md. Paths are relative to /api/v1/realms/{realm}.

• The realm's SAML signing key leaked. Whoever holds it can sign assertions that every SAML application of the realm accepts, and AuthnRequests from the broker. Prepare a new key and activate it at once (category emergency, the reason is recorded):

  shCopy

  identity-cli api POST /saml/signing-keys '{"reason":"INC-..."}'
  identity-cli api POST /saml/signing-keys/{id}/activate '{"force":true,"reason":"INC-... SAML key compromise"}'

  The old key stays published for saml.retired_key_publication, and service providers keep trusting it until they drop its certificate: tell every SP's administrator to load the new metadata (GET /realms/{realm}/saml/metadata) and remove the old certificate. Until they do, suspend applications whose SP cannot be updated quickly (POST /emergency/controls with target_type: client and the application's backing client ID).

• A SAML connection's IdP, or its signing key, is compromised. Disable the connection (target_type: identity_provider); review sign-ins and links in the window (federation.login.succeeded with metadata provider, federation.identity.linked, accounts created just in time) as for any upstream provider (Compromised connector, step 2). Before enabling it again, replace its signing_certificates through PUT /saml/connections/{id}.

• A service provider is compromised. It received assertions (NameIDs and mapped attributes) and holds its own signing key: suspend its backing client (target_type: client), and replace its signing_certificates before reactivating it. Its users' sessions are not affected by suspending it; revoke them if the SP could have used its key to send forged logout or sign-in requests.

• Evidence: saml.assertion.issued, saml.request.rejected, saml.response.rejected, saml.signing_key.* and saml.application.* (GET /audit-events?type_prefix=saml.).

Directory connector compromise#

A directory server, the service account it uses, a downstream SCIM service or its bearer token is compromised (directory.md, outbound-provisioning.md). There is no emergency control for directory connectors.

1. Stop the connector. PUT .../directory-connectors/{id} with the current definition, "state": "disabled" and the ETag of the GET as If-Match. For an organization connector, suspending the organization (target_type: organization) also stops it. A disabled LDAP connector makes its users' password sign-ins fail closed (503); that is the intended effect if the directory itself is not trusted. Setting directory.enabled: false and restarting stops every connector at once, with the same effect for all directory-sourced accounts.
2. Rotate the credential with PUT .../credential once the directory or downstream is trusted again (change the service account's password, or issue a new downstream token, there first).
3. Know what it could do. A hostile directory sees the passwords of its users at sign-in and, for a realm connector with jit_provisioning, the passwords typed for identifiers unknown locally; it can create, update, suspend and (above the deletion threshold only with an operator override) deprovision the accounts and memberships it manages, and grant the roles its group mappings allow, within its ceiling. It cannot touch protected accounts, holders of privileged realm roles or other organizations. A downstream SCIM service received the pushed attributes and cannot change anything in ProudAuth.
4. Review its actions: GET /audit-events?actor_id=con_... and type_prefix=directory. (directory.user.provisioned, .linked, .updated, .deprovisioned, directory.sync.*), the connector's runs and findings, and operator_actions in .../health. Revoke the sessions of accounts it changed if their identity attributes or state were altered.

Abuse spikes#

Signals: proudauth_authentications_total{outcome="failure"}, proudauth_rate_limited_total, proudauth_load_shed_total rising; bursts of authentication.password.failed, authentication.password.temporarily_locked, registration.* or code sends; an SMS daily budget exhausted; mail provider complaints (monitoring.md).

1. Find the source: GET /audit-events?type_prefix=authentication.&network=<cidr>&from=... narrows by network (stored as /24 or /48), client and organization; proxy logs give full addresses.
2. Contain within the realm, without a restart (realm policy changes apply on every node within the 2-second realm cache):
   • authentication_flow rules: deny or challenge for the offending networks, clients or organizations, or for failed_attempts_at_least and risk_at_least with a risk hook (headless-authentication.md);
   • close sign-up (registration.mode: closed) or turn off code and link sign-in (login.email_otp, login.magic_link, login.sms_otp);
   • stop an abused client or grant class with an emergency control.
3. Tighten deployment budgets (security.rate_limits.*, delivery.sms.daily_budget; restart) and block the source at the proxy.
4. Watch the side effects: the temporary password lock also locks out the real owners of attacked accounts for security.rate_limits.temporary_lock (other sign-in methods keep working), and budgets that fail closed answer 429 or 503 to legitimate users of the same client or organization.
5. Afterwards, review accounts that signed in from the source and revoke their sessions if needed; record the rules you added and remove them when the spike is over (each policy change is audited as realm.policy.updated).

Mass session revocation#

After any mass revocation, JWT access tokens remain usable at offline verifiers for up to their remaining lifetime (default 5 minutes plus clock tolerance).

Database unavailable or degraded#

Behavior (docs/plan.md §31.2):

• /health/live stays 200 (it never touches dependencies) — do not let your orchestrator restart processes because the database is down.
• /health/ready returns 503 {"status":"unavailable"}; remove instances from load balancing.
• New grants, refresh, introspection, UserInfo and all custom APIs fail; the SDK Introspector returns ErrUnavailable and strict-online resources deny (TEST-035). Offline-verified JWTs keep working until expiry.
• Workers log worker task failed and retry on their next tick; queued work is not lost.
• A failure in the middle of a sensitive transition (refresh rotation, code exchange, password reset, invitation acceptance, ...) rolls the whole transition back: the client gets an error and no token, the refresh token, code or reset code it presented stays unconsumed (no reuse or replay revocation), and repeating the request once the database is back succeeds (TEST-059, TestTransactionFaultsLeaveNoPartialState).
• Slow database: queries are cancelled after storage.statement_timeout (10 s); requests are bounded at 30 s; excess concurrency is shed with 503 once limits.max_inflight_requests is reached.

Steps:

1. Check /status on the internal listener (capabilities, schema, backlog_oldest_seconds, clock_skew_ms; see monitoring.md) and PostgreSQL itself.
2. If failover to a standby lost recent transactions, treat it as a restore: reconcile with the revocation log (backup-restore.md).
3. If readiness fails with the database up, run identity-cli doctor: a schema behind the binary needs identity-cli migrate; an unusable signing key means wrong root keys or a missing active key.

Database loss#

The database (or its storage) is lost or corrupted beyond repair, or a failover lost committed transactions.

1. Put the service into maintenance (identity-cli maintenance -mode unavailable -message "...") or take the instances out of the load balancer, and stop identity-worker.
2. Follow the restore procedure: restore the newest usable backup or point in time, configure the same keys and compare the identity-cli doctor fingerprint, migrate if the binary is newer, mark the database with identity-cli restore-begin (it then stays unready until reconciled), collect the revocation logs exported by identity-cli revocation-log since the backup, mind the ordering rule for compromised signing keys, and run identity-cli restore-reconcile -log ... (add -invalidate-all-sessions when the log does not cover the whole gap).
3. Apply the manual remedies for what reconciliation does not cover (backup-restore.md), including audit evidence lost since the backup: it survives only in the audit archive and exports.
4. Verify (identity-cli doctor, identity-cli verify-audit, a synthetic sign-in and introspection), set maintenance off, then start the worker.

TestRestoreReconciliation exercises the export and reconciliation logic on a copied database; measure your own recovery time in restore drills.

Audit chain verification failure#

Run identity-cli verify-audit (operator). It verifies the whole audit_batches chain across all realms, including batch signatures, and prints batches_checked, events_checked, unsealed_events, batches_signature_verified, batches_pruned_by_retention, the problems list and ok, plus the current audit signing key (kid and Ed25519 public key) for independent verifiers; it exits non-zero when ok is false. Add -require-signatures once every batch has been sealed with signing, so unsigned batches count as problems. GET /audit-integrity (realm.audit.read) returns only a cached deployment-wide summary (ok, batches_checked, events_checked, the number of problems, scope, checked_at), recomputed at most every 10 minutes; use the CLI for details. The worker also verifies the newest 200 batches every hour (audit.verify_recent) and reports what it finds as proudauth_audit_integrity_problems and an audit integrity problems log line (monitoring.md); it names no details, so run the CLI.

Interpret problems:

Audit archive (AUD-008)#

With privacy.audit_archive_dir set, the worker task audit.archive copies every sealed batch, within about a minute, to a write-once file batch-<12-digit sequence>.ndjson in that directory: a header line (record: "batch", batch_id, sequence, previous_hash, events_digest, batch_hash, signature, signing_kid, event_count, and archived_events when the file holds fewer events than were sealed) followed by the batch's events as stored, one JSON object per line. Files are created exclusively and read-only (0440); an existing file is accepted only when it is byte-identical. Progress is kept in audit_archive_state (last_sequence and the current directory). Mount storage the operator controls independently of the database there (a separate volume, object storage with retention locks or a WORM device); ProudAuth never deletes archived files.

While an archive is configured, retention prunes only events of batches that are already archived (sequence at most audit_archive_state.last_sequence): nothing is pruned before the first archive run, and an archive that stops also holds back retention, so every archived file is complete. Only batches pruned before the archive was configured are archived without their pruned events; their header records the number of events written as archived_events.

After a database restore the restored database seals new batches under sequence numbers the archive already holds with other content (the lost period). The worker re-archives from the restored last_sequence, accepts files that are identical, and at the first sequence whose file differs it leaves the existing file in place, creates fork-<12-digit sequence>-<batch id>/ in the archive directory, writes the batch there and continues in that directory (migration 0039). It records the operator event audit.archive.forked (category audit, no realm; metadata sequence and directory). The main directory keeps the lost period's files as evidence; do not move or delete them.

identity-cli verify-audit-archive [-config <file>] [-dir <path>] (default directory: privacy.audit_archive_dir) needs no database connection; it loads the configured keys only to check signatures. It verifies the main directory and every fork-* directory: each event's content hash, each complete batch's events digest, each batch hash, a valid signature on every batch, and the links between files. It prints the report of verify-audit (batches_checked, events_checked, batches_signature_verified, batches_pruned_by_retention, problems, ok) plus first_sequence and last_sequence of the main directory and the forks it found, and exits non-zero when ok is false. Problems in a fork are prefixed with its directory name.

Limits: the verifier sees only files. It cannot tell that the newest files are missing, or that leading files are missing when the first file is not batch 1: check that first_sequence is 1 and compare the newest file (in the main directory, or in the newest fork after a restore) with SELECT last_sequence, directory FROM audit_archive_state and SELECT max(sequence_number) FROM audit_batches. A fork's first link is checked against the main directory only, so after a second restore that forks a fork, check the new fork's first previous_hash against the earlier fork by hand.

For suspected tampering:

1. Snapshot the database and PostgreSQL logs; restrict superuser access.
2. Compare affected events with the audit archive and with independently exported NDJSON audit exports (each archived or exported event includes its content_hash) to find the original values.
3. Identify who holds table-owner or superuser rights; the application role should not need them after migrations.
4. Remember the limit: an operator who controls the database, keys and backups can rewrite the entire chain consistently. Batches are hash-linked and signed with a key derived from the root key, so a rewrite also needs the keys; no external anchoring is built in, so only copies on storage the database operator cannot change (the audit archive, regular exports) and a recorded audit signing public key detect a full rewrite.

Email or webhook delivery outage#

Email:

1. Inspect GET /notification-jobs?reference_type=...&reference_id=... (state, attempts, last_error, masked recipient). Jobs retry with exponential backoff (10 s × 2^attempt) for 5 attempts, then failed; jobs past their expiry become expired.
2. Test the relay: POST /notification-settings/check connects, negotiates TLS and authenticates without sending anything and returns {"ok", "stage"}, where stage names the failing step (connect, greeting, starttls, credentials, authentication) or ok (not_applicable for the development sink); the server's error text stays in the operator log (smtp.check_failed). POST /notification-tests then sends a test message to the calling administrator's own verified address. Provider acceptance does not prove inbox delivery. Jobs in state suppressed were not sent on purpose: the recipient muted that optional notice, or the address is suppressed after a hard bounce or complaint reported through POST /delivery-events. Mandatory messages (codes, links, security notices) are never suppressed. An organization with its own mail server (TEN-022, GET /organizations/{org}/notification-settings/smtp) sends all its mail there, and /notification-settings/check does not test that server: if only one organization's mail fails, check its server with the organization's owner, or remove it (DELETE .../notification-settings/smtp, org.integrations.write) to fall back to the deployment relay. A server whose address the delivery egress policy refuses fails every attempt.
3. One-time codes expire after tokens.challenge_ttl (10 min); after the outage users request a new code (send budgets and cooldowns still apply). Do not mark addresses verified to work around an outage; POST /users/{user}/verification-overrides exists for documented exceptions only and is audited.

SMS: jobs to the SMS gateway are notification jobs like email; inspect them the same way. proudauth_delivery_total{channel="notification"} on the worker listener shows failed and expired outcomes for both. Users can sign in with other methods meanwhile; SMS codes are never a second factor.

Webhooks:

1. GET /webhooks/{id}/backlog summarizes one subscription (EXT-005): counts of pending, retrying, dead_letter and delivered deliveries, oldest_due_seconds, last_delivered_at and last_failure_category. GET /webhooks/{id}/deliveries shows state, attempts, last status and response category (network_error, egress_denied, http_500 and other status classes such as http_400, gone, ...). Retries use exponential backoff with jitter (5 s × 2^attempt, capped at 6 h) up to delivery.webhook_max_attempts (12), then dead_letter. HTTP 410 dead-letters immediately.
2. After the receiver is fixed, replay: POST /webhooks/{id}/deliveries/{delivery}/replay (keeps the original event_id).
3. Paused subscriptions (PATCH /webhooks/{id} {"state":"paused"}) do not receive events published while paused; those are not queued for later.
4. Back-channel logout deliveries retry up to 8 times and then stay failed; there is no replay API.

Organization owner recovery#

An organization has lost access to its owner: the only owner left, lost their credentials, or is suspended or compromised. Last-owner protection keeps the organization from losing its last owner role, so recover the account or add an owner.

1. The owner can be identified and reached: recover the account (account-recovery.md): lost factor with POST /users/{user}/recovery-requests (reset_mfa), lost contact address by adding, verifying (override) and promoting the new address first. For a compromised owner, follow Compromised administrator account first.

2. Ownership must move: a realm administrator (the built-in realm-admin role holds org.* in every organization) confirms the request through the organization's own channel, records the evidence, and transfers ownership to an active member:

   shCopy

   identity-cli api GET /organizations/{org}/members                     # find the new owner's membership ID
   identity-cli api POST /organizations/{org}/ownership-transfers '{"membership_id":"mem_..."}'

   The member receives the owner role (audit organization.ownership.transferred). release_current_owner releases only the caller's own ownership, so remove the former owner's owner role afterwards if needed (GET /role-assignments?subject_type=user&subject_id={user}, DELETE /role-assignments/{assignment}); it is no longer the last owner. Transfers take the exclusive organization lock directly and check current authority after waiting. A caller whose authority came only from the owner role loses it after self-release; a realm administrator retains their independently granted recovery authority. Concurrent retries cannot remove the last active owner.

3. Tell the organization's members through your own channel; notices are not sent automatically.

Privacy deletion errors#

An account deletion did not propagate, a scheduled deletion did not run, or a restore brought deleted data back.

1. Check the report: GET /users/{user}/deletion-status (data-inventory.md). 409 means the account is not deleted (its state is in the response).
2. Scheduled self-service deletion not done: accounts in pending_deletion are deleted by the worker task lifecycle.expiry once deletion_scheduled_at has passed. Check that identity-worker runs and look for worker task failed with task=lifecycle.expiry (one failing account stops the rest of that run). An administrator can delete at once with DELETE /users/{user}.
3. Pending stages:
   • sessions_and_grants should be 0 right after deletion; if not, revoke them (identity-cli revoke-sessions -realm <r> -user usr_...) and check again;
   • connectors: the user.deleted event has not reached every webhook receiver; fix the receiver and replay (GET /webhooks/{id}/backlog, POST /webhooks/{id}/deliveries/{delivery}/replay);
   • exports: purged by retention.cleanup after privacy.export_retention;
   • imported_audit_history: records left under a legal hold. Releasing the hold does not clear them later; once it is released, clear them with a reviewed database update mirroring account deletion and record it.
4. Deleted data back after a restore: run identity-cli restore-reconcile with revocation logs that contain the deletion's tombstone; it re-applies the deletion, including imported audit records. What it leaves is listed in backup-restore.md.
5. Backups: the backups stage dates when the last backup holding the account expires (privacy.backup_retention); ProudAuth does not delete backups.

Support impersonation abuse#

1. Find grants: GET /impersonation-grants?actor_user_id=...&state=active (support.impersonation.read; without it operators see only their own).
2. End access:
   • one grant: POST /impersonation-grants/{id}/revoke {"reason":"INC-..."} (support.impersonation.revoke, the actor, or the target customer);
   • all grants of an operator: remove their support-operator assignment (DELETE /role-assignments/{id}). Every introspection re-checks support.impersonation.start, so all their tokens stop working at the next request. Or suspend the operator (POST /users/{user}/suspend), which revokes their sessions and their grants.
3. Stop new access: set realm policy impersonation.enabled: false (PUT /settings) or deployment impersonation.enabled: false and restart. These switches block new requests, approvals and issuance, and every introspection re-checks them, so live support tokens become inactive at their next use. The grants themselves stay active until they expire (within impersonation.max_ttl, at most 5 minutes); revoke them as in step 2 to record the end explicitly.
4. Evidence: correlate by grant ID with GET /audit-events?impersonation_grant_id=imp_...: support.impersonation.requested, .approved, .started, .revoked, support.handoff.*, and support.token.introspected (one per introspection). Collect the resource servers' own SupportEvent records.
5. Notify affected customers if realm policy notify_customer was off; customers can view grants at GET /me/impersonation-grants.

Emergency controls (SEC-020)#

One audited endpoint switches off a grant class, client, identity provider, organization, service principal, pre-action hook or webhook subscription without database edits (internal/httpapi/emergency.go, TestEmergencyControls). It needs realm.emergency (held by the built-in realm-admin role) and a reason:

identity-cli api POST /emergency/controls \
  '{"target_type":"grant_type","target_id":"refresh_token","action":"disable","reason":"INC-42 refresh token theft"}'

• target_type: grant_type, client, identity_provider, organization, service_principal, hook or webhook. target_id is the target's internal ID (for clients the cli_... ID, not the OAuth client_id) or, for grant_type, one of authorization_code, refresh_token, client_credentials, urn:ietf:params:oauth:grant-type:device_code and urn:ietf:params:oauth:grant-type:token-exchange.
• action: disable or enable; reason: required, at most 500 characters.
• Response: {"target_type", "target_id", "state": "disabled" | "enabled"}.
• GET /emergency/controls (realm.emergency) shows what is switched off: realm_state, disabled_grant_types, and the IDs of suspended clients and organizations and of disabled identity providers, service principals, hooks and webhooks (up to 200 of each).

Every audit event recorded while a control runs is filed in the emergency category, and events without a reason of their own carry the operator's reason as their reason code. Client, organization and service-principal controls therefore produce the same event types as their ordinary routes, but in that category and with the reason. Events in the emergency category are kept beyond privacy.audit_retention and raise the emergency_action security signal (monitoring.md); list them with GET /audit-events?category=emergency.

The endpoint stays available in maintenance mode restricted but not in unavailable (see below). Signing keys are handled by emergency rotation (see "Compromised signing key"), a whole realm by POST /emergency/realm-state.

Other controls:

Maintenance mode (OPS-015)#

identity-cli maintenance (operator channel, internal/maintenance, TestMaintenanceAndStatus) refuses requests deployment-wide while you work on the database, keys or configuration:

identity-cli maintenance -mode restricted -message "Planned maintenance until 14:00 UTC" -retry-after 900
identity-cli maintenance              # print the current state
identity-cli maintenance -mode off

• The state is one row in system_state (migration 0025); every server reads it at most every 2 seconds, so a change applies to the whole deployment within that time. If the database cannot be read, a server keeps the last state it saw.
• Refusals are 503 problem documents with code maintenance, a Retry-After header (-retry-after, default 300 seconds, at most 86400) and, as detail, the -message (plain text, at most 200 characters, no <, > or line breaks). The message is shown to anonymous clients; do not put internal details in it. Running the command with -mode but without -message clears the message.
• Readiness is not affected: instances stay in load balancing and answer with the maintenance problem. /status on the internal listener reports the mode and turns its overall status into maintenance.
• restricted matches these requests by method and path only. It lets through POST /users/{user}/revoke-sessions, POST /users/{user}/suspend and /lock, client and organization suspension, and other POST routes ending in /suspend, so access can be taken away during maintenance; reactivation and other changes (including POST /service-principals/{id}/disable; use the emergency control instead) are refused.
• identity-worker is not paused: deliveries, jobs and other tasks keep running.
• Each change is audited as operations.maintenance.changed (actor local:<OS user>, metadata mode, retry_after_seconds and public_message) in the same transaction as the change, so a change is never stored without its audit event or the reverse.
• Server binaries that predate migration 0025 ignore the setting; finish a rolling upgrade before relying on it.