> For AI agents: the complete documentation index is available at https://www.tteam.icu/llms.txt, the full documentation bundle is available at https://www.tteam.icu/llms-full.txt. # prometheus [开源地址](https://github.com/prometheus/prometheus) [文档地址](https://prometheus.io/docs/introduction/overview/) ## 架构图 ![](https://img.tteam.icu/i/2025/07/31/voylzt-3.webp) ## prometheus-server **setup.sh** ```sh file="./prometheus/setup.sh" #!/bin/bash echo "Asia/Shanghai" > /etc/timezone mkdir -p $(pwd)/data chown -R 65534:65534 $(pwd)/data port=9090 docker stop prometheus docker rm prometheus docker run -d --net host \ --name prometheus \ -v /etc/timezone:/etc/timezone:ro \ -v /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime:ro \ -v $(pwd)/config/:/etc/prometheus/ \ -v $(pwd)/data/:/prometheus/ \ -v $(pwd)/groups/:/usr/local/prometheus/groups/ \ -v $(pwd)/rules/:/usr/local/prometheus/rules/ \ prom/prometheus:v3.11.0 \ --config.file=/etc/prometheus/prometheus.yml \ --web.console.libraries=/etc/prometheus/console_libraries \ --web.console.templates=/etc/prometheus/consoles \ --storage.tsdb.path=/prometheus \ --storage.tsdb.retention.time=60d \ --web.enable-admin-api ``` **config/prometheus.yml** ```yaml file="./prometheus/config/prometheus.yml" global: scrape_interval: 60s evaluation_interval: 60s rule_files: - "/usr/local/prometheus/rules/*.rules" scrape_configs: - job_name: 'prometheus' static_configs: - targets: ["127.0.0.1:9090"] ``` :::tip 如果在调试的时候需要查看请求的日志，可以在启动脚本中新增 `--log.level=debug` 配置。 ::: ## [exporter](https://www.tteam.icu/ops/monitor/prometheus-exporter.md) ## 集合不同节点数据 ### 主配置 ```yaml scrape_configs: - job_name: 'prometheus' static_configs: - targets: ["127.0.0.1:9090"] - job_name: 'federate' scrape_interval: 5s scrape_timeout: 3s metrics_path: '/federate' honor_labels: true params: 'match[]': - '{job!=""}' file_sd_configs: - files: ['/usr/local/prometheus/groups/federate/*.json','/usr/local/prometheus/groups/federate/*.yml'] ``` ### 组配置 ```yaml - targets: [ "127.0.0.1:9090"] labels: job_name: test instance: test comment: "test" ``` ## pushgetway ### 启动 ```sh file="./prometheus/pushgetway/setup.sh" title="setup.sh" #!/bin/bash docker kill pushgateway docker rm pushgateway docker run -d --net host \ --name pushgateway \ --restart=always \ -v /etc/localtime:/etc/localtime:ro \ -v /etc/tiemzone:/etc/timezone:ro \ prom/pushgateway:v1.7.0 \ --web.listen-address=0.0.0.0:9091 \ --web.enable-admin-api \ --web.telemetry-path=/metrics ``` ### prometheus 在`prometheus.yaml`中增加: ```yaml - job_name: 'aliyun-server-info' scrape_interval: 60s scrape_timeout: 35s static_configs: - targets: ['192.168.1.1:9091'] labels: instance: aliyun-server-info ``` ### 推送数据 一般uri格式为`/metrics/job//instance/` ```shell cat < [{{ .Name }}: {{ .Value | markdown | html }} ] {{- end }}
**故障时间:** {{ $alert.StartsAt.Local.Format "2006-01-02 15:04:05" }} ========= = end = ========= {{- end }} {{- end }} {{- if gt (len .Alerts.Resolved) 0 -}} {{- range $index, $alert := .Alerts -}} ========= 告警恢复 ========= **告警程序:** Alertmanager **告警主题:** {{ $alert.Annotations.summary }} **告警主机:** {{ .Labels.instance }} **告警类型:** {{ .Labels.alertname }} **告警级别:** {{ $alert.Labels.severity }} **告警状态:** {{ .Status }} **告警详情:** {{ $alert.Annotations.message }}{{ $alert.Annotations.description}} **告警图形:** [📈]({{ .GeneratorURL }}) **故障时间:** {{ $alert.StartsAt.Local.Format "2006-01-02 15:04:05" }} **恢复时间:** {{ $alert.EndsAt.Local.Format "2006-01-02 15:04:05" }} ========= = **end** = ========= {{- end }} {{- end }} {{- end }} ``` **setup.sh** ```sh file="./prometheus/prometheus-webhook-dingtalk/setup.sh" #!/bin/bash version="v2.1.0" docker stop prometheus-webhook-dingtalk docker rm prometheus-webhook-dingtalk docker run -d \ --net host \ --name prometheus-webhook-dingtalk \ -v /etc/timezone:/etc/timezone:ro \ -v /etc/localtime:/etc/localtime:ro \ -v $(pwd)/config/:/etc/prometheus-webhook-dingtalk/ \ -v $(pwd)/templates/:/prometheus-webhook-dingtalk/templates/ \ timonwong/prometheus-webhook-dingtalk:${version} \ --web.listen-address=:8060 \ --web.enable-ui \ --web.enable-lifecycle \ --config.file=/etc/prometheus-webhook-dingtalk/config.yml \ --log.level=info \ --log.format=logfmt ``` ### alertmanager **config/alertmanager.yml** ```yaml file="./prometheus/alertmanager/config/alertmanager.yml" global: # 每2分钟检查一次是否恢复 resolve_timeout: 2m # route用来设置报警的分发策略 route: receiver: 'ops-dingtalk' group_by: ['...'] group_wait: 3s group_interval: 1m repeat_interval: 5m routes: - receiver: 'ops-dingtalk' group_by: ['...'] # 等待时间，如果同一个组有新的告警会被合并到同一个消息内 group_wait: 3s # 异常持续报警间隔时间为 group_interval + repeat_interval 总和 # group_interval 相同的Group之间发送告警通知的时间间隔 group_interval: 1m # 一条成功发送的告警，在最终发送通知之前的等待时间 repeat_interval: 5m matchers: - severity=~"^信息$|^警告$|^一般严重$|^严重$|^灾难$|^测试模板$" receivers: - name: 'ops-dingtalk' webhook_configs: - send_resolved: true url: 'http://10.0.18.2:8060/dingtalk/webhook/send' max_alerts: 0 ``` **templates/alertmanager-email.tmpl** ```bash file="./prometheus/alertmanager/templates/alertmanager-email.tmpl" #########基于alertmanager官方模板修改，内容可删减######### {{ define "__alertmanager" }}Alertmanager{{ end }} {{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}{{ end }} {{ define "__description" }}{{ end }} {{ define "email.to.html" }} {{ template "__subject" . }}
发生 {{ .Alerts | len }} 个 {{ range .GroupLabels.SortedPairs }} {{ .Value }} {{ end }} 告警 ！！请尽快处理
{{ range .Alerts.Firing }} {{ end }}
告警名称 告警级别 实例 所属系统 厂商 触发时间 说明
{{ .Labels.severity }} {{ .Status }} {{ .Labels.instance }} {{ .Labels.ownningsystem }} {{ .Labels.company }} {{ .StartsAt.Format "2006-01-02 15:04:05" }} {{ .Annotations.description }}
Sent by {{ template "__alertmanager" . }}
{{ end }} ``` **setup.sh** ```sh file="./prometheus/alertmanager/setup.sh" #!/bin/bash echo "Asia/Shanghai" > /etc/timezone port=9093 docker stop alertmanager docker rm alertmanager docker run --name alertmanager -d \ --net host \ -v /etc/timezone:/etc/timezone:ro \ -v /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime:ro \ -v $(pwd)/config/alertmanager.yml:/etc/alertmanager/alertmanager.yml \ -v $(pwd)/templates/:/etc/alertmanager/templates/ \ prom/alertmanager:v0.26.0 ``` #### 配置增加 ```yaml alerting: alertmanagers: - static_configs: - targets: - '10.0.18.2:9093' ```