Kubernetes监控实战：Prometheus Operator部署与优化-平芜编程栈

1. 项目概述

在Kubernetes集群中实现高效可靠的监控一直是运维工作的重点和难点。Prometheus Operator作为Kubernetes原生的监控解决方案，通过自定义资源定义(CRD)简化了Prometheus及其相关组件的部署和管理。本文将基于Ubuntu 22.04 LTS系统，详细介绍如何优化和部署Prometheus Operator，构建完整的Kubernetes监控体系。

提示：本文所有操作均在Ubuntu 22.04 LTS (Jammy Jellyfish)上验证通过，内核版本5.15及以上，适用于生产环境和测试环境。

2. 环境准备与依赖安装

2.1 系统基础配置

首先确保Ubuntu系统已完成基础优化：

# 更新系统并安装必要工具 sudo apt update && sudo apt upgrade -y sudo apt install -y curl wget git vim net-tools lvm2 # 调整内核参数 echo "vm.swappiness = 10" | sudo tee -a /etc/sysctl.conf echo "vm.overcommit_memory = 1" | sudo tee -a /etc/sysctl.conf sudo sysctl -p # 设置时区同步 sudo timedatectl set-timezone Asia/Shanghai sudo systemctl enable --now chronyd

2.2 Kubernetes集群部署

使用kubeadm部署生产级Kubernetes集群：

# 安装容器运行时 sudo apt install -y containerd sudo mkdir -p /etc/containerd containerd config default | sudo tee /etc/containerd/config.toml sudo systemctl restart containerd # 安装kubeadm/kubelet/kubectl sudo apt install -y apt-transport-https ca-certificates curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list sudo apt update && sudo apt install -y kubelet kubeadm kubectl sudo apt-mark hold kubelet kubeadm kubectl # 初始化控制平面 sudo kubeadm init --pod-network-cidr=10.244.0.0/16 \ --apiserver-advertise-address=$(hostname -I | awk '{print $1}') \ --kubernetes-version=stable-1.28 # 配置kubectl mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config # 安装网络插件 kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

3. Prometheus Operator核心部署

3.1 Helm安装与配置

推荐使用Helm进行Operator部署：

# 安装Helm curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash # 添加Prometheus社区仓库 helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update

3.2 定制化安装Operator

创建values-custom.yaml配置文件：

# values-custom.yaml prometheus: prometheusSpec: scrapeInterval: 30s evaluationInterval: 30s retention: 15d resources: requests: memory: 2Gi cpu: 1 limits: memory: 4Gi cpu: 2 storageSpec: volumeClaimTemplate: spec: storageClassName: "standard" accessModes: ["ReadWriteOnce"] resources: requests: storage: 50Gi alertmanager: enabled: true alertmanagerSpec: replicas: 3 resources: requests: memory: 512Mi cpu: 500m limits: memory: 1Gi cpu: 1 grafana: enabled: true adminPassword: "StrongPassword@123" persistence: enabled: true size: 10Gi

执行安装命令：

helm install kube-prometheus prometheus-community/kube-prometheus-stack \ -n monitoring --create-namespace \ -f values-custom.yaml

4. 高级配置与优化

4.1 监控目标自动发现

配置ServiceMonitor监控自定义服务：

# custom-service-monitor.yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: custom-app-monitor namespace: monitoring labels: release: kube-prometheus spec: jobLabel: app selector: matchLabels: app: custom-app namespaceSelector: matchNames: - default endpoints: - port: web interval: 30s path: /metrics scheme: http

应用配置：

kubectl apply -f custom-service-monitor.yaml

4.2 告警规则配置

创建自定义告警规则：

# custom-alert-rules.yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: custom-alerts namespace: monitoring labels: app: kube-prometheus-stack release: kube-prometheus spec: groups: - name: node-alerts rules: - alert: HighNodeCPU expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80 for: 10m labels: severity: warning annotations: summary: "High CPU usage on {{ $labels.instance }}" description: "CPU usage is {{ $value }}% for more than 10 minutes"

应用告警规则：

kubectl apply -f custom-alert-rules.yaml

5. 性能优化实践

5.1 TSDB存储优化

调整Prometheus存储参数：

# prometheus-tsdb-optimization.yaml apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: kube-prometheus-prometheus namespace: monitoring spec: retention: 30d retentionSize: "100GB" walCompression: true query: lookbackDelta: 5m maxConcurrency: 20 timeout: 2m

5.2 资源限制与调度

优化资源分配策略：

# resource-optimization.yaml apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: kube-prometheus-prometheus namespace: monitoring spec: resources: requests: memory: "8Gi" cpu: "2" limits: memory: "16Gi" cpu: "4" affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - prometheus topologyKey: "kubernetes.io/hostname"

6. 监控数据可视化

6.1 Grafana仪表板配置

导入官方Kubernetes仪表板：

# 获取仪表板ID kubectl get secret -n monitoring kube-prometheus-grafana -o jsonpath='{.data.admin-password}' | base64 --decode # 访问Grafana (端口转发) kubectl port-forward -n monitoring svc/kube-prometheus-grafana 3000:80

推荐导入的仪表板ID：

Kubernetes集群监控：3119
Node Exporter：11074
Prometheus 2.0：3662

6.2 自定义仪表板开发

创建ConfigMap存储自定义仪表板：

# custom-dashboard.yaml apiVersion: v1 kind: ConfigMap metadata: name: custom-dashboard namespace: monitoring labels: grafana_dashboard: "1" data: custom-dashboard.json: | { "title": "Custom Application Metrics", "panels": [ { "title": "Request Rate", "type": "graph", "datasource": "Prometheus", "targets": [ { "expr": "sum(rate(http_requests_total[5m])) by (service)", "legendFormat": "{{service}}" } ] } ] }

7. 生产环境注意事项

7.1 高可用配置

实现Prometheus高可用部署：

# prometheus-ha.yaml apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: kube-prometheus-prometheus namespace: monitoring spec: replicas: 2 shards: 2 retention: 15d serviceMonitorSelector: matchLabels: release: kube-prometheus podMonitorSelector: matchLabels: release: kube-prometheus

7.2 安全加固措施

启用TLS和基础认证：

# prometheus-security.yaml apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: kube-prometheus-prometheus namespace: monitoring spec: enableAdminAPI: false web: tlsConfig: cert: secret: name: prometheus-tls key: tls.crt keySecret: name: prometheus-tls key: tls.key

8. 故障排查与维护

8.1 常见问题诊断

查看Operator日志：

kubectl logs -n monitoring -l app.kubernetes.io/name=prometheus-operator

检查Prometheus状态：

kubectl get prometheus -n monitoring -o wide kubectl describe prometheus kube-prometheus-prometheus -n monitoring

8.2 数据备份策略

定期备份Prometheus数据：

# 创建备份目录 mkdir -p /backup/prometheus # 执行备份 kubectl cp monitoring/prometheus-kube-prometheus-prometheus-0:/prometheus /backup/prometheus/$(date +%Y%m%d)

9. 扩展与集成

9.1 与Alertmanager集成

配置邮件告警通知：

# alertmanager-config.yaml apiVersion: v1 kind: Secret metadata: name: alertmanager-main namespace: monitoring type: Opaque stringData: alertmanager.yaml: | global: smtp_smarthost: 'smtp.example.com:587' smtp_from: 'alertmanager@example.com' smtp_auth_username: 'user@example.com' smtp_auth_password: 'password' route: group_by: ['alertname'] group_wait: 30s group_interval: 5m repeat_interval: 3h receiver: 'email' receivers: - name: 'email' email_configs: - to: 'alerts@example.com'

9.2 长期存储方案

集成Thanos实现长期存储：

# thanos-sidecar.yaml apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: kube-prometheus-prometheus namespace: monitoring spec: thanos: baseImage: quay.io/thanos/thanos version: v0.28.0 objectStorageConfig: key: thanos.yaml name: thanos-objstore-config

10. 性能监控指标解读

关键监控指标说明：

指标名称	正常范围	告警阈值	说明
`up`	1	<1	目标是否可达
`node_memory_MemAvailable_bytes`	>20%总内存	<10%总内存	可用内存
`node_cpu_seconds_total`	<70%	>80%持续5m	CPU使用率
`kube_pod_container_status_restarts_total`	0	>3	容器重启次数
`prometheus_tsdb_head_samples_appended_total`	-	突增50%	采样率变化

11. 版本升级与维护

执行Helm升级：

# 检查新版本 helm search repo prometheus-community/kube-prometheus-stack --versions # 执行升级 helm upgrade kube-prometheus prometheus-community/kube-prometheus-stack \ -n monitoring \ --version 45.7.1 \ -f values-custom.yaml

12. 资源清理

完整卸载监控栈：

helm uninstall kube-prometheus -n monitoring kubectl delete crd alertmanagerconfigs.monitoring.coreos.com kubectl delete crd alertmanagers.monitoring.coreos.com kubectl delete crd podmonitors.monitoring.coreos.com kubectl delete crd probes.monitoring.coreos.com kubectl delete crd prometheuses.monitoring.coreos.com kubectl delete crd prometheusrules.monitoring.coreos.com kubectl delete crd servicemonitors.monitoring.coreos.com kubectl delete crd thanosrulers.monitoring.coreos.com kubectl delete ns monitoring

Kubernetes监控实战：Prometheus Operator部署与优化