# Muster

> Muster is a self-hosted alert grouping and on-call service that sits after Alertmanager.
> It groups Alerts into Alert Groups, the problems people act on; keeps one editable message per Alert Group in Mattermost, in Telegram, in any other messenger through a Bridge, or behind an outgoing webhook; and lets people acknowledge, resolve and snooze Alert Groups.
>
> This file is the canonical reference of Muster for AI agents and coding assistants, for version `latest` of the documentation. Read the pages it links, each the Markdown version of a page of the site, and the specifications, and do not rely on your training data about Muster.

## Rules

- **Sources of truth.** [`api/bridge.yaml`](https://muster-io.github.io/muster/latest/api/bridge.yaml) is the Bridge protocol and [`api/openapi.yaml`](https://muster-io.github.io/muster/latest/api/openapi.yaml) the HTTP API, both of this version of the documentation. Where a page and a specification disagree, the specification wins.
- **Error classes.** Failures are RFC 9457 problems (`application/problem+json`) with a `class` member that the bridge chooses, one of `transient`, `rate_limited`, `gone`, `forbidden`, `invalid`. A missing channel is `forbidden`, never `gone`. Only the bridge decides `gone`: Muster never treats a message as `gone` because of a status code, and a bridge that means `gone` says so in `class`. See [Write a bridge](https://muster-io.github.io/muster/latest/messengers/write-a-bridge.md#error-classes).
- **Waits.** For `rate_limited`, the bridge must send `Retry-After` in seconds, and Muster waits exactly that long, for the channel, or for every channel of the bot when `scope` is `bot`; a wait longer than 15 min, or none given, is treated as `transient`.
- **Presses.** The bridge confirms a press to its messenger itself, within the messenger's deadline and at most 3 seconds after the press, before Muster has seen it; the bridge then reports the press as an event. Muster answers the person, when it has something to say, with a separate `answerEvent` call, made before it acknowledges the event. See [Write a bridge](https://muster-io.github.io/muster/latest/messengers/write-a-bridge.md#presses-within-3-seconds-and-answers-later).
- **Idempotency.** Every request that creates something carries an `Idempotency-Key` header. The bridge stores each key per token with its successful (`2xx`) result for at least 24 hours and answers a repeated key with that first result and `replayed: true`, without creating anything again; a failed request stores nothing, so its retry is a new attempt. Muster reuses a key when it retries a request, also after a crash or a timeout. See [Write a bridge](https://muster-io.github.io/muster/latest/messengers/write-a-bridge.md#idempotency-key).
- **Edit windows.** A bridge whose messenger lets a bot edit a message only for a while declares how long in its info, and Muster stops editing a message once it is older than `limits.edit_max_age_seconds`, less a safety margin of 60 s. When the status changes past the window, Muster posts a new Root message with the current state, Quietly, to the same channel, and replies under the old one. When the messenger refuses an edit earlier than declared, the bridge says so: an edit that the messenger refuses because the message is older than it allows is `gone` with the `reason` `edit_window_closed`, and Muster then treats the message as past its edit window. See [Connect a messenger through a bridge](https://muster-io.github.io/muster/latest/messengers/bridge.md#edit-windows).
- **Finishing a bridge.** A bridge is finished only when [`muster bridge check`](https://muster-io.github.io/muster/latest/messengers/write-a-bridge.md#muster-bridge-check) passes against it: the check exits `0` when no step failed. Most bridge authors should not write the protocol again. A bridge in Go or Python uses that language's bridge library ([Start with a library](https://muster-io.github.io/muster/latest/messengers/write-a-bridge.md#start-with-a-library)); a bridge in another language follows [The protocol without a library](https://muster-io.github.io/muster/latest/messengers/write-a-bridge.md#the-protocol-without-a-library).

## Scenarios

### Write a bridge in Go

1. [Write a bridge](https://muster-io.github.io/muster/latest/messengers/write-a-bridge.md#the-go-library): The Go library, installed with `go get github.com/muster-io/muster-bridge-go`, and [The example bridge](https://muster-io.github.io/muster/latest/messengers/write-a-bridge.md#the-example-bridge), which is built on the Go library and lives in its repository, in `examples/chat`.
2. [Connect a messenger through a bridge](https://muster-io.github.io/muster/latest/messengers/bridge.md): the Bridge Connection and its Destinations, as an Admin sets them up.
3. [Bridge protocol](https://muster-io.github.io/muster/latest/reference/bridge-protocol.md) and its specification, [`api/bridge.yaml`](https://muster-io.github.io/muster/latest/api/bridge.yaml).
4. The repository [muster-io/muster-bridge-go](https://github.com/muster-io/muster-bridge-go): its README is the guide to the library, and its [AGENTS.md](https://github.com/muster-io/muster-bridge-go/blob/master/AGENTS.md) the guide for an agent that changes the library itself.
5. Finished when [`muster bridge check`](https://muster-io.github.io/muster/latest/messengers/write-a-bridge.md#muster-bridge-check) passes against the bridge.

### Write a bridge in Python

1. [Write a bridge](https://muster-io.github.io/muster/latest/messengers/write-a-bridge.md#the-python-library): The Python library. The libraries are released when Muster's first release is. Until then the package is not on PyPI: install it from a clone of its repository with `git clone https://github.com/muster-io/muster-bridge-python && cd muster-bridge-python`, then `python3 -m venv .venv && . .venv/bin/activate` and `pip install .`.
2. [Connect a messenger through a bridge](https://muster-io.github.io/muster/latest/messengers/bridge.md): the Bridge Connection and its Destinations, as an Admin sets them up.
3. [Bridge protocol](https://muster-io.github.io/muster/latest/reference/bridge-protocol.md) and its specification, [`api/bridge.yaml`](https://muster-io.github.io/muster/latest/api/bridge.yaml).
4. The repository [muster-io/muster-bridge-python](https://github.com/muster-io/muster-bridge-python): its README is the guide to the library, and its [AGENTS.md](https://github.com/muster-io/muster-bridge-python/blob/master/AGENTS.md) the guide for an agent that changes the library itself.
5. Finished when [`muster bridge check`](https://muster-io.github.io/muster/latest/messengers/write-a-bridge.md#muster-bridge-check) passes against the bridge.

### Connect Alertmanager

1. [Quick start](https://muster-io.github.io/muster/latest/quick-start.md): Muster with Docker Compose, an Integration and a test alert.
2. [Connect Alertmanager](https://muster-io.github.io/muster/latest/integrations/alertmanager.md): one Integration per Alertmanager cluster, its token, and the receiver and the route of the snippet.
3. [Heartbeat](https://muster-io.github.io/muster/latest/integrations/heartbeat.md), which is part of the recommended setup of every Integration.
4. [Snapshot semantics](https://muster-io.github.io/muster/latest/concepts/snapshot-semantics.md): how Muster reads what Alertmanager sends.
5. Finished when, after Alertmanager is reloaded, the Integration page shows the time of the last Snapshot ([Connect Alertmanager](https://muster-io.github.io/muster/latest/integrations/alertmanager.md#create-a-token-and-paste-the-snippet)) and the Heartbeat is live ([Heartbeat](https://muster-io.github.io/muster/latest/integrations/heartbeat.md#turn-it-on)).

### Connect Grafana

1. [Connect Grafana](https://muster-io.github.io/muster/latest/integrations/grafana.md), both ways: directly, from Grafana's built-in Alertmanager through a webhook contact point, with one Integration per Grafana cluster and organization, its token, the contact point and the notification policy ([Connect Grafana directly](https://muster-io.github.io/muster/latest/integrations/grafana.md#connect-grafana-directly)); or through an external Alertmanager, then [Connect Alertmanager](https://muster-io.github.io/muster/latest/integrations/alertmanager.md) ([Connect Grafana through an external Alertmanager](https://muster-io.github.io/muster/latest/integrations/grafana.md#connect-grafana-through-an-external-alertmanager)).
2. [Heartbeat](https://muster-io.github.io/muster/latest/integrations/heartbeat.md): its setup is for Alertmanager, so it serves Grafana through an external Alertmanager. For the direct way, the same setup through a webhook contact point has not been verified with Grafana, and without a Heartbeat Muster never resolves an Alert as Stale ([Heartbeat](https://muster-io.github.io/muster/latest/integrations/grafana.md#heartbeat)).
3. [Snapshot semantics](https://muster-io.github.io/muster/latest/concepts/snapshot-semantics.md): how Muster reads what Grafana's built-in Alertmanager or an external one sends, and the [Limits](https://muster-io.github.io/muster/latest/integrations/grafana.md#limits) of Grafana's way.
4. Finished when, after an alert fires in Grafana, the Integration page shows the time of the last Snapshot ([Connect Grafana](https://muster-io.github.io/muster/latest/integrations/grafana.md#send-the-alerts-with-a-notification-policy)).

### Automate through the API

1. [API](https://muster-io.github.io/muster/latest/reference/api.md): the conventions of the API and its tokens. Scripts send a Personal access token (`mstr_pat_`) or a Service account token (`mstr_sat_`) as a bearer token. [Sign-in](https://muster-io.github.io/muster/latest/sign-in/index.md#ways-to-sign-in) describes both.
2. [`api/openapi.yaml`](https://muster-io.github.io/muster/latest/api/openapi.yaml), the OpenAPI 3.1 document that the binary serves at `/api/v1/openapi.yaml`.
3. [API tokens and automation](https://muster-io.github.io/muster/latest/configure/api-tokens.md): Personal access tokens, Service accounts, the rate limit, `ETag` and `If-Match`, pagination.
4. [Statuses and commands](https://muster-io.github.io/muster/latest/concepts/statuses-and-commands.md): the statuses of an Alert Group and the commands that change them.
5. [The command line](https://muster-io.github.io/muster/latest/operations/cli.md): The commands are for emergencies, for setup and for diagnosis; everything else is in the web UI and the API.

## Document map

- [Home](https://muster-io.github.io/muster/latest/index.md): Muster is a self-hosted alert grouping and on-call service that sits after Alertmanager.
- [Quick start](https://muster-io.github.io/muster/latest/quick-start.md): Run Muster with Docker Compose on one machine, sign in, connect Alertmanager and send a test alert.

### Install

- [Install with Helm](https://muster-io.github.io/muster/latest/install/helm.md): The Helm chart runs Muster on Kubernetes 1.30 or newer.
- [The database](https://muster-io.github.io/muster/latest/install/database.md): Muster keeps everything in PostgreSQL 14 or newer, and nothing else stateful: no Redis and no queue.
- [Requirements and tested versions](https://muster-io.github.io/muster/latest/install/versions.md): What Muster needs, which versions it was checked against, and how to verify that a release you download is the one the project built.

### Configure

- [Messages and templates](https://muster-io.github.io/muster/latest/configure/messages-and-templates.md): Muster shows every Alert Group as one **Root message** in each Destination, and edits it as the Alert Group changes.
- [Organization settings](https://muster-io.github.io/muster/latest/configure/organization.md): The **Organization** is the owner of all Users, Routes and data in Muster; there is one.
- [API tokens and automation](https://muster-io.github.io/muster/latest/configure/api-tokens.md): Everything the web UI does, a script can do through the API, as a person or as a non-person identity.

### Integrations

- [Connect Alertmanager](https://muster-io.github.io/muster/latest/integrations/alertmanager.md): Muster receives alerts from Alertmanager through an Alertmanager receiver with a webhook.
- [Connect Grafana](https://muster-io.github.io/muster/latest/integrations/grafana.md): Grafana Alerting reaches Muster in one of two ways: directly, from Grafana's built-in Alertmanager through a webhook contact point, or through an external Alertmanager that Grafana forwards its alerts to.
- [Heartbeat](https://muster-io.github.io/muster/latest/integrations/heartbeat.md): A **Heartbeat** proves, for one Integration, that the path from your cluster to Muster works: Prometheus evaluates rules, Alertmanager sends notifications, and the network carries them to Muster.

### Messengers

- [Messengers](https://muster-io.github.io/muster/latest/messengers/index.md): Muster delivers Alert Groups to **Destinations**.
- [Connect Mattermost](https://muster-io.github.io/muster/latest/messengers/mattermost.md): Muster posts to Mattermost through the REST API with a **bot account**: one post per Alert Group, the **Root message**, which Muster edits as the Alert Group changes, and replies in its **Thread** for what happens later.
- [Connect Telegram](https://muster-io.github.io/muster/latest/messengers/telegram.md): Muster posts to a Telegram **channel** through the Bot API with a **bot**: one channel post per Alert Group, the **Root message**, which Muster edits as the Alert Group changes.
- [Telegram in restricted networks](https://muster-io.github.io/muster/latest/messengers/telegram-restricted-networks.md): Muster talks to Telegram only by calling out to the Bot API, `https://api.telegram.org` by default.
- [Proxies](https://muster-io.github.io/muster/latest/messengers/proxies.md): Muster has no proxy for the whole installation.
- [Connect a messenger through a bridge](https://muster-io.github.io/muster/latest/messengers/bridge.md): A **bridge** connects Muster to a messenger that Muster does not support natively.
- [Write a bridge](https://muster-io.github.io/muster/latest/messengers/write-a-bridge.md): A **bridge** connects Muster to a messenger that Muster does not support natively.
- [Link your messenger accounts](https://muster-io.github.io/muster/latest/messengers/account-links.md): An **Account link** ties your Telegram account, your account on a Mattermost server, or your account on a messenger that Muster reaches through a Bridge, to your Muster User.

### Outgoing webhooks

- [Outgoing webhook: events mode](https://muster-io.github.io/muster/latest/outgoing-webhooks/events.md): An outgoing webhook Destination in the **events mode** sends one HTTP `POST` for every lifecycle event of every Alert Group on its Routes: an ordered, signed stream of versioned JSON events that an automation can record, react to or forward.
- [Outgoing webhook: template mode](https://muster-io.github.io/muster/latest/outgoing-webhooks/templates.md): An outgoing webhook Destination in the **template mode** keeps one message per Alert Group in any chat that has a REST API, the way Muster keeps a message in Mattermost or Telegram: a **create** request posts it, an **update** request edits it to the current state, and optional **open thread** and **reply in thread** requests carry the follow-ups.
- [Recipes](https://muster-io.github.io/muster/latest/outgoing-webhooks/recipes.md): Four ways to use an outgoing webhook, built only from what it has: the events mode sends one signed event per lifecycle event; the template mode sends the requests you write and keeps their answers.

### Sign-in

- [Sign-in](https://muster-io.github.io/muster/latest/sign-in/index.md): Everything in the web UI and the API works only after sign-in.
- [Sign-in with Keycloak](https://muster-io.github.io/muster/latest/sign-in/oidc-keycloak.md): This page sets up sign-in through Keycloak with OpenID Connect (OIDC).
- [TOTP](https://muster-io.github.io/muster/latest/sign-in/totp.md): TOTP is a second step in the sign-in: a six-digit code from an authenticator app, which changes every 30 seconds (RFC 6238).
- [Closed networks](https://muster-io.github.io/muster/latest/sign-in/closed-networks.md): OpenID Connect needs a **back channel**: Muster itself calls the identity provider, to read its configuration and keys, to exchange the code for tokens, and to check the sessions again later.

### Concepts

- [Alert Groups](https://muster-io.github.io/muster/latest/concepts/alert-groups.md): An **Alert Group** is a set of Alerts that Muster treats as one problem.
- [Routes and Group keys](https://muster-io.github.io/muster/latest/concepts/routes-and-group-keys.md): A **Route** decides what happens to an Alert after Muster has read it: which Alert Group it joins, who is told, and how the Alert Group is handled afterwards.
- [Snapshot semantics](https://muster-io.github.io/muster/latest/concepts/snapshot-semantics.md): Alertmanager does not tell Muster "this alert started" and "this alert stopped" as separate events.
- [Loud and Quiet](https://muster-io.github.io/muster/latest/concepts/loud-and-quiet.md): Every change to an Alert Group is either **Loud** or **Quiet**.
- [Statuses and commands](https://muster-io.github.io/muster/latest/concepts/statuses-and-commands.md): An Alert Group is always in one of four statuses, and only a few things move it from one to another.
- [Broken Destinations](https://muster-io.github.io/muster/latest/concepts/broken-destinations.md): A Destination is **Broken** when Muster has decided that it cannot deliver there until something changes: the messenger refused the bot for good, or the messenger kept failing until Muster ran out of patience.

### Operations

- [Backup and restore](https://muster-io.github.io/muster/latest/operations/backup-and-restore.md): Muster keeps all its state in PostgreSQL, and encrypts its secrets with the master keys from the environment.
- [Upgrades](https://muster-io.github.io/muster/latest/operations/upgrades.md): Muster follows semantic versioning (`major.minor.patch`).
- [Key rotation](https://muster-io.github.io/muster/latest/operations/key-rotation.md): Muster encrypts its Secrets and signs the buttons of its messages with the **master keys** in `MUSTER_SECRET_KEYS`.
- [High availability](https://muster-io.github.io/muster/latest/operations/high-availability.md): Run more than one replica of Muster when you want the web UI, ingestion and delivery to survive the loss of a node or a restart.
- [Monitoring](https://muster-io.github.io/muster/latest/operations/monitoring.md): Muster exports Prometheus metrics, ships alert rules and a Grafana dashboard in its Helm chart, and raises Internal alerts about its own problems.
- [Outgoing heartbeat](https://muster-io.github.io/muster/latest/operations/outgoing-heartbeat.md): The metrics and the Internal alerts tell you about a Muster that runs.
- [The command line](https://muster-io.github.io/muster/latest/operations/cli.md): The `muster` binary is the server and its command line.
- [Request log](https://muster-io.github.io/muster/latest/operations/request-log.md): The **Request log** keeps the recent HTTP requests Muster sent to a messenger, a bridge or an outgoing webhook, and those that reached a Connection's addresses, each with its answer and with secrets masked.

#### Runbooks

- [Runbooks](https://muster-io.github.io/muster/latest/operations/runbooks/index.md): One page for every alert Muster can raise: the rules of the Helm chart, evaluated by Prometheus or VictoriaMetrics from Muster's metrics, and the Internal alerts that Muster raises itself.
- [MusterDown](https://muster-io.github.io/muster/latest/operations/runbooks/MusterDown.md): Prometheus cannot scrape a Muster replica: the target of the job `muster` is down.
- [MusterNoLeader](https://muster-io.github.io/muster/latest/operations/runbooks/MusterNoLeader.md): Every replica of Muster reports that it is not the Leader.
- [MusterDeliveryFailing](https://muster-io.github.io/muster/latest/operations/runbooks/MusterDeliveryFailing.md): A Destination has deliveries waiting and has not had a successful delivery for 10 minutes.
- [MusterDestinationBroken](https://muster-io.github.io/muster/latest/operations/runbooks/MusterDestinationBroken.md): A Destination is **Broken**: Muster stopped sending to it after a Fatal error — the messenger refused the bot or the request for good — or after its Transient budget ran out (`delivery.transient_budget`, 10 attempts within 30 minutes).
- [MusterHeartbeatLost](https://muster-io.github.io/muster/latest/operations/runbooks/MusterHeartbeatLost.md): An Integration is **Heartbeat lost**: its Heartbeat is on, and no signal arrived within its Heartbeat timeout (`integration.heartbeat_timeout`, 5 minutes by default).
- [MusterDeliverySlow](https://muster-io.github.io/muster/latest/operations/runbooks/MusterDeliverySlow.md): Deliveries are slow: the 95th percentile of the time from receiving a Snapshot to the API call that delivers the resulting change is above 5 seconds, the objective Muster keeps while its rate limiters have room.
- [MusterDeliveryQueueGrowing](https://muster-io.github.io/muster/latest/operations/runbooks/MusterDeliveryQueueGrowing.md): A Destination has more than 100 deliveries waiting, and the number keeps growing.
- [MusterIngestBacklog](https://muster-io.github.io/muster/latest/operations/runbooks/MusterIngestBacklog.md): Processing falls behind ingestion: more than 1,000 Stored Snapshots wait to be processed, or the 95th percentile of the time from receiving a Snapshot to the end of its processing is above 60 seconds.
- [MusterIngestRejected](https://muster-io.github.io/muster/latest/operations/runbooks/MusterIngestRejected.md): Muster refused ingestion requests within 15 minutes: an oversized one, or more than 10 without a valid Integration token, counted per value of the label `integration`.
- [MusterTemplateError](https://muster-io.github.io/muster/latest/operations/runbooks/MusterTemplateError.md): A template failed to render.
- [MusterClockSkew](https://muster-io.github.io/muster/latest/operations/runbooks/MusterClockSkew.md): A replica's clock differs from the database clock by more than 2 seconds.
- [MusterLoginFailures](https://muster-io.github.io/muster/latest/operations/runbooks/MusterLoginFailures.md): A burst of failed sign-ins: more than 20 for one method within 10 minutes.
- [MusterSnapshotTruncated](https://muster-io.github.io/muster/latest/operations/runbooks/MusterSnapshotTruncated.md): Alertmanager truncates the Snapshots of an Integration: it left Alerts out of a Snapshot because of the `max_alerts` limit of its webhook receiver, or, for Grafana, the Max alerts limit of its webhook contact point.
- [MusterOIDCSecretExpiring](https://muster-io.github.io/muster/latest/operations/runbooks/MusterOIDCSecretExpiring.md): The OIDC client secret expires soon: the date entered as **Client secret expires on** in the OIDC settings is within `oidc.secret_expiry_lead` (14 days).

### Reference

- [Reference](https://muster-io.github.io/muster/latest/reference/index.md): The pages of this section are generated from the code and the specification of this version of Muster.
- [Metrics](https://muster-io.github.io/muster/latest/reference/metrics.md): Muster serves these metrics in the Prometheus text format at `/metrics` on the internal listener and never pushes them; the endpoint also serves the process (`process_*`) and Go runtime (`go_*`) metrics.
- [Log events](https://muster-io.github.io/muster/latest/reference/log-events.md): Muster writes one JSON object per line on stdout, and every line is one of the events below.
- [Internal alerts](https://muster-io.github.io/muster/latest/reference/internal-alerts.md): Muster reports problems it can still report through its own pipeline as Internal alerts: Alerts of the built-in "Muster" Integration, which every Organization has from the first start.
- [Environment variables](https://muster-io.github.io/muster/latest/reference/environment.md): Muster reads its bootstrap settings from these `MUSTER_*` environment variables at start, and no others; every other setting lives in the database and changes through the UI or the API.
- [Chart values](https://muster-io.github.io/muster/latest/reference/chart-values.md): The values of the Helm chart `deploy/helm/muster`.
- [API](https://muster-io.github.io/muster/latest/reference/api.md): The API reference is rendered from `api/openapi.yaml`, the OpenAPI 3.1 document that the binary serves at `/api/v1/openapi.yaml`, so it describes this version of Muster.
- [Bridge protocol](https://muster-io.github.io/muster/latest/reference/bridge-protocol.md): The Bridge protocol is the HTTP protocol between Muster and a bridge, the separate service through which Muster delivers Alert Groups to a messenger it has no native support for: Muster calls the bridge to post, edit and reply, and the bridge reports presses, Thread messages, Account link requests and deleted messages back.

## Optional

- [Full text](https://muster-io.github.io/muster/latest/llms-full.txt): every page in one file, 609595 bytes, about 152398 tokens.