Skip to content

Internal alerts

Muster reports problems it can still report through its own pipeline as Internal alerts: Alerts of the built-in "Muster" Integration, which every Organization has from the first start. Each raise and each resolve is a synthetic Stored Snapshot of that Integration, listed with its other Stored Snapshots and processed in order with them. Muster raises and resolves them explicitly when their condition changes; they are never resolved as Gone or Stale. Besides alertname and severity, a configuration entity appears by its immutable id (integration, route, destination) with its current name in a *_name label; the fingerprint is computed from alertname and the id label only, so renaming the entity updates the open Internal alert instead of raising a new one. Every Internal alert carries the annotations summary, description and runbook_url: MUSTER_RUNBOOK_BASE_URL, the major.minor of the binary (latest for a development build) and the runbook below, joined by /. The capability is the one that raises it.

MusterDestinationBroken

critical · labels: destination, destination_name · C-11 · runbook: operations/runbooks/MusterDestinationBroken/

Raised while: A Destination is Broken: the messenger answered with a Fatal error, or with Transient errors until the budget ran out, so its deliveries wait. It resolves when a probe, a Destination check or a Destination test succeeds.

MusterHeartbeatLost

critical · labels: integration, integration_name, the Integration's Static labels · C-07 · runbook: operations/runbooks/MusterHeartbeatLost/

Raised while: An Integration is Heartbeat lost: no Heartbeat signal arrived within its Heartbeat timeout, so the path from its Alertmanager to Muster may be interrupted and Stale resolution is paused. It resolves with the next signal, when the Heartbeat is turned off or when the Integration is deleted.

MusterOIDCSecretExpiring

warning · no labels · C-19 · runbook: operations/runbooks/MusterOIDCSecretExpiring/

Raised while: The OIDC client secret expires within oidc.secret_expiry_lead, by the expiry date an Admin entered in the OIDC settings. It resolves when the date moves further out or is removed.

MusterSnapshotTruncated

warning · labels: integration, integration_name · C-06 · runbook: operations/runbooks/MusterSnapshotTruncated/

Raised while: An Integration has a truncated groupKey: Alertmanager cut a Snapshot with max_alerts, so Muster keeps the unlisted Alerts alive and cannot resolve them by absence. It resolves with the next untruncated Snapshot of every truncated groupKey.

MusterTemplateError

warning · labels: route, route_name, destination, destination_name, template · C-12 · runbook: operations/runbooks/MusterTemplateError/

Raised while: A template of a Route failed while rendering a message, so its messages use the Fallback template; or a request template of an outgoing webhook Destination failed, so that request was not sent. It resolves when the template renders again or, for a Route, a new template is saved.