Skip to content

MusterSnapshotTruncated

Meaning

Alertmanager truncates the Snapshots of an Integration: it left Alerts out of a Snapshot because of the max_alerts limit of its webhook receiver, or, for Grafana, the Max alerts limit of its webhook contact point. Muster treats each Snapshot as the whole Alertmanager group, so a Truncated Snapshot hides Alerts from it.

It is an Internal alert only, severity warning, raised by Muster as an Alert of the built-in "Muster" Integration when the first Alertmanager group (groupKey) of the Integration arrives truncated. Its labels are integration (the Integration's id) and integration_name. It is resolved when no group of the Integration is truncated any more: each group's next untruncated Snapshot ends its truncation, and so does a group that has sent nothing for as long as it takes an Alert to go Stale.

Impact

  • The Alerts that Alertmanager left out do not reach their Alert Groups: Muster may show fewer Alerts than fire, miss an Alert Group entirely, or keep an Alert open that Alertmanager no longer lists.
  • For Grafana, when a whole group resolves, its last Snapshot lists only the first alerts, up to Max alerts, and Grafana sends nothing more for the group until one of its alerts fires again. The Alerts it left out stay firing in Muster until they go Stale, which needs a live Heartbeat; the Heartbeat through Grafana has not been verified (see Connect Grafana).
  • Everything else of the Integration keeps working.

Diagnosis

  1. Open the Integration page (/integrations/<id>): the banner says "Alertmanager truncates Snapshots for N groups. Set max_alerts: 0 on the Alertmanager receiver." Its Alertmanager routes list shows the Truncated groups.
  2. System status (/admin/system-status) lists the Integration under Integrations needing attention with "Snapshots truncated".
  3. muster_ingest_truncated_snapshots_total by integration counts the truncated Snapshots; the log line internal_alert_raised (alertname, fingerprint, entity) marks when the alert was raised.
  4. Open a recent Stored Snapshot of the Integration (/integrations/<id>/snapshots/<id>): a Truncated Snapshot has truncatedAlerts above 0.
  5. In the Alertmanager configuration, find the webhook_configs entry of the receiver that sends to Muster and its max_alerts. For Grafana, open the webhook contact point that sends to Muster and check its Max alerts (maxAlerts in a provisioning file).

Fix

  • Set max_alerts: 0 on the Alertmanager receiver that sends to Muster, so that every Alert of the group is sent, and reload Alertmanager; see Connect Alertmanager.
  • For Grafana, set Max alerts to 0, the default, on the webhook contact point that sends to Muster; see Connect Grafana. Max alerts 0 does not end the Alerts that a wholly resolved group of Grafana's left firing: they go Stale while the Heartbeat is live, or end once the group fires again and sends untruncated Snapshots. Resolving their Alert Group by hand does not end them either: Alerts still firing when its Grace period ends start a new Alert Group.
  • The alert resolves once each truncated group has sent an untruncated Snapshot, at its next notification. A group of Grafana's that resolved wholly sends none until one of its alerts fires again, so its truncation ends as Meaning says.