【Prometheus·Exporter 篇】Blackbox Exporter：黑盒监控与网络探测-平芜编程栈

前言

前面讲的 Exporter 都是"白盒"监控——需要在被监控方安装 Agent。但有些场景你无法在目标上安装任何东西：外部 API、第三方服务、网络设备。Blackbox Exporter 从外部探测目标，属于"黑盒"监控。

一、白盒 vs 黑盒监控

白盒监控： 目标内部 → 安装 Exporter → 暴露内部指标 → CPU、内存、QPS、错误率 → 回答"为什么出问题了" 黑盒监控： 外部 → 探测目标 → 验证可达性 → HTTP 状态码、响应时间、证书有效期 → 回答"是否出问题了"

维度	白盒	黑盒
部署位置	目标内部	外部
可见性	内部细节	外部行为
告警含义	即将出问题	已经出问题
适用	自己的服务	外部/不可控目标

二、安装 Blackbox Exporter

二进制

wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.25.0/blackbox_exporter-0.25.0.linux-amd64.tar.gz tar xzf blackbox_exporter-0.25.0.linux-amd64.tar.gz cd blackbox_exporter-0.25.0.linux-amd64 sudo cp blackbox_exporter /usr/local/bin/

配置文件

# /etc/blackbox/blackbox.yml modules: # HTTP 探测 http_2xx: prober: http timeout: 5s http: preferred_ip_protocol: ip4 ip_protocol_fallback: false valid_http_versions: [HTTP/1.1, HTTP/2.0] valid_status_codes: [200, 301, 302] method: GET follow_redirects: true headers: User-Agent: "Prometheus-Blackbox-Exporter" # TLS 配置 tls_config: insecure_skip_verify: false # HTTPS 探测（关注证书） http_2xx_check_tls: prober: http timeout: 5s http: valid_status_codes: [200] method: GET tls_config: insecure_skip_verify: false # POST 请求探测 http_post_2xx: prober: http timeout: 5s http: method: POST headers: Content-Type: application/json body: '{"key":"value"}' # TCP 端口探测 tcp_connect: prober: tcp timeout: 5s # SSH 端口探测（banner 检查） ssh_banner: prober: tcp timeout: 5s tcp: query_response: - expect: "^SSH-2.0-" # ICMP ping icmp: prober: icmp timeout: 5s icmp: preferred_ip_protocol: ip4 # DNS 解析 dns_check: prober: dns timeout: 5s dns: query_name: "example.com" query_type: A valid_rcodes: [NOERROR]

Systemd 服务

# /etc/systemd/system/blackbox_exporter.service [Unit] Description=Blackbox Exporter After=network.target [Service] Type=simple ExecStart=/usr/local/bin/blackbox_exporter \ --config.file=/etc/blackbox/blackbox.yml \ --web.listen-address=0.0.0.0:9115 \ --log.level=info Restart=on-failure [Install] WantedBy=multi-user.target

Docker 部署

docker run -d \ --name blackbox-exporter \ -p 9115:9115 \ -v /etc/blackbox/blackbox.yml:/etc/blackbox/blackbox.yml \ prom/blackbox-exporter:v0.25.0 \ --config.file=/etc/blackbox/blackbox.yml

三、HTTP 探测

手动探测

# 探测一个 URL curl "http://localhost:9115/probe?target=https://example.com&module=http_2xx" # 查看探测结果 probe_success 1 probe_duration_seconds 0.234 probe_http_status_code 200 probe_ssl_earliest_cert_expiry 1.738688e+09

Prometheus 配置

scrape_configs: - job_name: 'blackbox-http' metrics_path: /probe params: module: [http_2xx] static_configs: - targets: - https://www.example.com - https://api.example.com/health - http://internal-app:8080/health relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115

关键指标

# 1. 探测成功（0=失败，1=成功） probe_success # 2. 响应时间 probe_duration_seconds # 3. HTTP 状态码 probe_http_status_code # 4. SSL 证书剩余天数 (probe_ssl_earliest_cert_expiry - time()) / 86400 # 5. DNS 解析时间 probe_dns_lookup_time_seconds # 6. TLS 握手时间 probe_tls_handshake_seconds # 7. HTTP 版本 probe_http_version

四、TCP 端口探测

scrape_configs: - job_name: 'blackbox-tcp' metrics_path: /probe params: module: [tcp_connect] static_configs: - targets: - mysql:3306 - redis:6379 - kafka:9092 - postgres:5432 labels: probe_type: tcp relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115

五、ICMP Ping 探测

scrape_configs: - job_name: 'blackbox-icmp' metrics_path: /probe params: module: [icmp] static_configs: - targets: - 8.8.8.8 - 8.8.4.4 - 114.114.114.114 - 192.168.1.1 - 10.0.0.1 labels: probe_type: icmp relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115

⚠️踩坑提示：ICMP 探测需要 root 权限或setcap cap_net_raw+ep /usr/local/bin/blackbox_exporter。

# 设置 cap_net_raw 权限 sudo setcap cap_net_raw+ep /usr/local/bin/blackbox_exporter

六、DNS 解析探测

scrape_configs: - job_name: 'blackbox-dns' metrics_path: /probe params: module: [dns_check] static_configs: - targets: - 8.8.8.8 # DNS 服务器 labels: dns_query: 'example.com' relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115

七、SSL 证书监控

scrape_configs: - job_name: 'ssl-cert-expiry' metrics_path: /probe params: module: [http_2xx_check_tls] static_configs: - targets: - https://www.example.com - https://api.example.com - https://admin.example.com relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115

# 证书剩余天数 (probe_ssl_earliest_cert_expiry - time()) / 86400 # 证书过期告警（30 天） ((probe_ssl_earliest_cert_expiry - time()) / 86400) < 30

八、告警规则

groups: - name: blackbox-alerts rules: # 服务不可达 - alert: ServiceDown expr: probe_success == 0 for: 2m labels: severity: critical annotations: summary: "Service {{ $labels.instance }} is down" description: "Probe to {{ $labels.instance }} has been failing for 2 minutes" # 响应时间过长 - alert: SlowResponse expr: probe_duration_seconds > 5 for: 5m labels: severity: warning annotations: summary: "Slow response from {{ $labels.instance }}" description: "Response time is {{ $value }}s (threshold: 5s)" # SSL 证书即将过期 - alert: SSLCertExpiringSoon expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 30 for: 1h labels: severity: warning annotations: summary: "SSL cert for {{ $labels.instance }} expires soon" description: "Certificate expires in {{ $value | printf "%.0f" }} days" # SSL 证书即将过期（紧急） - alert: SSLCertExpiringCritical expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 7 for: 1h labels: severity: critical annotations: summary: "SSL cert for {{ $labels.instance }} expires in 7 days!" # HTTP 状态码异常 - alert: HTTPStatusError expr: probe_http_status_code >= 500 for: 2m labels: severity: critical annotations: summary: "HTTP {{ $labels.instance }} returning {{ $value }}" # DNS 解析失败 - alert: DNSResolutionFailed expr: probe_dns_rcode != 0 for: 5m labels: severity: warning annotations: summary: "DNS resolution failed for {{ $labels.instance }}"

九、动态目标配置

从 Consul 发现

scrape_configs: - job_name: 'blackbox-consul' metrics_path: /probe params: module: [http_2xx] consul_sd_configs: - server: 'consul:8500' services: ['web', 'api'] relabel_configs: - source_labels: [__address__] target_label: __param_target regex: '(.+):(?:\d+)' replacement: 'http://$1/' - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115

从文件发现

scrape_configs: - job_name: 'blackbox-file' metrics_path: /probe params: module: [http_2xx] file_sd_configs: - files: ['/etc/prometheus/targets/blackbox/*.yml'] relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115

# /etc/prometheus/targets/blackbox/services.yml - targets: - https://service1.example.com - https://service2.example.com labels: team: payments env: production - targets: - https://service3.example.com labels: team: orders env: production

要点回顾

探测类型	module	场景
HTTP	http_2xx	Web 服务可用性
HTTPS+TLS	http_2xx_check_tls	证书监控
TCP	tcp_connect	数据库/中间件端口
SSH	ssh_banner	SSH 服务检查
ICMP	icmp	主机存活检查
DNS	dns_check	DNS 解析监控