一、Alertmanager介绍Prometheus 内置了一个报警模块——Alertmanager它负责接收来自 Prometheus 的告警信息并支持对告警进行去重、降噪、分组、路由同时具备丰富的通知渠道集成能力是一款高效的告警通知系统。告警流程架构在 Prometheus 中告警功能分为两部分Prometheus定义告警规则并周期性计算触发告警后发送给 Alertmanager。Alertmanager接收告警并按照配置进行后续处理与通知。下图展示了告警流程架构二、部署AlertmanagerAlertmanager 使用 Go 编写无第三方依赖支持跨平台部署。1. Linux安装步骤1下载安装包bashwget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz2创建配置文件解压后默认配置文件为alertmanager.yml主要内容如下yamlroute: group_by: [alertname] group_wait: 30s group_interval: 5m repeat_interval: 1h receiver: web.hook receivers: - name: web.hook webhook_configs: - url: http://127.0.0.1:5001/ inhibit_rules: - source_match: severity: critical target_match: severity: warning equal: [alertname, dev, instance]3启动Alertmanagerbash./alertmanager支持启动参数--config.file指定配置文件路径--storage.path指定数据存储路径4查看运行状态访问http://IP:9093可进入Alertmanager控制台。2. Docker安装步骤bashmkdir -p /etc/alertmanager/ mkdir -p /etc/alertmanager/template vim /etc/alertmanager/alertmanager.yml # 内容同上 docker run -d -p 9093:9093 \ -v /etc/alertmanager:/etc/alertmanager \ -v /etc/localtime:/etc/localtime \ prom/alertmanager三、配置Receiver接收告警信息Alertmanager 支持多种通知方式包括邮件、Slack、Webhook、PagerDuty 等。集成QQ邮箱告警示例1. 申请QQ邮箱授权码参考QQ邮箱授权码申请指南2. 配置alertmanager.ymlyamlglobal: resolve_timeout: 5m smtp_smarthost: smtp.qq.com:465 smtp_from: 276****211qq.com smtp_auth_username: 276****211qq.com smtp_auth_password: xxxxxxxxx # 授权码非QQ密码 smtp_require_tls: false templates: - /etc/alertmanager/template/*.tmpl route: group_by: [service,alertname,cluster] group_wait: 30s group_interval: 10s repeat_interval: 5m receiver: email receivers: - name: email email_configs: - to: 276****211qq.com send_resolved: true html: {{ template email.html . }} inhibit_rules: - source_match: severity: critical target_match: severity: warning equal: [alertname, dev, instance]3. 创建自定义邮件模板bashvim /etc/alertmanager/template/email.tmpl模板内容如下支持Go模板语法go{{ define email.html }} {{ range $i, $alert :.Alerts }} 监控报警br 告警状态{{ .Status }}br 告警级别{{ $alert.Labels.severity }}br 告警类型{{ $alert.Labels.alertname }}br 告警应用{{ $alert.Annotations.summary }}br 告警主机{{ $alert.Labels.instance }}br 告警详情{{ $alert.Annotations.description }}br 触发阀值{{ $alert.Annotations.value }}br 告警时间{{ ($alert.StartsAt.Add 28800e9).Format 2006-01-02 15:04:05}}br endbr {{ end }} {{ end }}注意StartsAt为UTC时间需加28800e9转换为北京时间2006-01-02 15:04:05为Go语言固定时间格式四、启用Alertmanager模块与Rules告警规则1. 配置Prometheus连接Alertmanager编辑prometheus.yml添加以下内容yamlglobal: scrape_interval: 5s evaluation_interval: 5s alerting: alertmanagers: - static_configs: - targets: - 192.168.3.100:9093 rule_files: - /etc/prometheus/rules/*.rules重启Prometheus后访问http://IP:9090/config确认配置生效。2. 创建告警规则文件示例规则CPU与内存使用率告警bashmkdir -p /etc/prometheus/rules vim /etc/prometheus/rules/alerts.rules内容如下yamlgroups: - name: hostStatsAlert rules: - alert: hostCpuUsageAlert expr: (1 - avg(irate(node_cpu_seconds_total{modeidle}[5m])) by (instance))*100 85 for: 1m labels: severity: critical annotations: summary: Instance {{ $labels.instance }} CPU usage high description: {{ $labels.instance }} CPU usage above 85% (current value: {{ $value }}) - alert: hostMemUsageAlert expr: (1 - (node_memory_MemAvailable_bytes{} / (node_memory_MemTotal_bytes{})))* 100 70 for: 1m labels: severity: critical annotations: summary: Instance {{ $labels.instance }} MEM usage high description: {{ $labels.instance }} MEM usage above 70% (current value: {{ $value }})重启Prometheus后可通过以下地址查看规则与告警状态规则列表http://127.0.0.1:9090/rules告警状态http://127.0.0.1:9090/alerts3. 模拟告警触发测试创建CPU负载脚本bashvim load_cpu.sh #!/bin/bash while true; do : done chmod x load_cpu.sh ./load_cpu.sh运行后观察Prometheus 检测到CPU使用率 85%告警状态先变为PENDING持续1分钟后触发FIRING状态并发送至AlertmanagerAlertmanager 根据路由发送邮件通知总结本文完整介绍了从 Alertmanager 部署、配置邮件接收、编写Prometheus告警规则到模拟告警测试的全过程。通过合理配置路由、抑制规则与通知模板可构建一套稳定、易用的监控告警系统。后续可继续扩展集成企业微信、钉钉、Slack等通知配置告警分级与静默规则使用Webhook对接自愈系统