深色模式
告警分派 routing
摘要：分派错误比告警缺失更消耗信任——数据库告警打到前端值班人手里，等于没人处理。本文用"服务标签 + 环境 + 级别"三要素构造路由树，并给出批量验证分派结果的方法。
适用环境
bash
# 先在告警规则里统一打标签，分派才有依据
grep -rn "labels:" -A3 /etc/prometheus/rules/*.yml | head -40
kubectl version --short 2>/dev/null || echo "非 K8s 环境，按主机分组亦可"1
2
3
2
3
操作步骤
第 1 步：统一标签词汇表
分派的前提是标签一致。约定三个必填标签：service、env、severity。
yaml
# 告警规则示例：标签必须齐全
groups:
- name: pay.rules
rules:
- alert: PayLatencyHigh
expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{job="pay"}[5m])) > 1
for: 5m
labels:
service: pay
env: prod
severity: critical
annotations:
summary: 'pay 接口 P99 超过 1s'
runbook: 'ops/oncall/runbook/pay-latency.md'1
2
3
4
5
6
7
8
9
10
11
12
13
14
2
3
4
5
6
7
8
9
10
11
12
13
14
第 2 步：按服务建路由树
yaml
route:
receiver: 'catch-all'
group_by: ['alertname', 'service']
routes:
- matchers: ['service="pay"', 'env="prod"']
receiver: 'team-pay'
routes:
- matchers: ['severity="critical"']
receiver: 'team-pay-primary'
- matchers: ['service="order"', 'env="prod"']
receiver: 'team-order'
- matchers: ['env!="prod"']
receiver: 'low-priority-digest'
receivers:
- name: 'catch-all'
webhook_configs: [{ url: 'http://bot/catchall' }]1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
第 3 步：非生产环境降噪
yaml
# 预发/测试只发汇总，禁止呼叫值班人
- matchers: ['env="staging"']
receiver: 'staging-digest'
group_wait: 5m
group_interval: 30m
repeat_interval: 12h1
2
3
4
5
6
2
3
4
5
6
第 4 步：批量校验分派
bash
# 用 amtool 逐条测试路由落点
while read -r s e v; do
amtool config routes test --config.file=/etc/alertmanager/alertmanager.yml \
service="$s" env="$e" severity="$v" | sed "s/^/$s\/$e\/$v -> /"
done <<'EOF'
pay prod critical
pay prod warning
order prod critical
pay staging critical
unknown prod critical
EOF1
2
3
4
5
6
7
8
9
10
11
2
3
4
5
6
7
8
9
10
11
第 5 步：把分派表沉淀为文档
bash
# 生成分派对照表，随配置一起评审
cat > routing-table.md <<'EOF'
| service | env | severity | receiver |
| --- | --- | --- | --- |
| pay | prod | critical | team-pay-primary |
| order | prod | critical | team-order |
| * | staging | * | staging-digest |
| * | * | * | catch-all |
EOF1
2
3
4
5
6
7
8
9
2
3
4
5
6
7
8
9
验证
bash
# 1. 配置可加载
amtool check-config /etc/alertmanager/alertmanager.yml
# 2. 未匹配服务会落到 catch-all，不会静默丢失
amtool config routes test --config.file=/etc/alertmanager/alertmanager.yml service=unknown env=prod
# 3. 生产告警不会误入 staging digest
amtool config routes test --config.file=/etc/alertmanager/alertmanager.yml service=pay env=prod severity=critical1
2
3
4
5
6
7
8
2
3
4
5
6
7
8
常见坑
标签拼写不一致
service=pay 与 service=Pay 是两条路由。标签值统一小写，并在 CI 中校验告警规则必带标签。
默认 receiver 缺失导致丢告警
route 里没有 receiver 兜底时，未匹配的告警会被丢弃。永远保留一个 catch-all。
复制粘贴整棵路由树
大量重复配置会让修改漏改。按服务分文件后用 cat 合并，或用配置生成工具统一管理。