Kubernetes持久化存储实战
引言
Kubernetes的持久化存储是运行有状态应用的关键基础设施。与无状态服务不同，有状态应用需要持久化存储来保存数据，如数据库、消息队列、文件系统等。Kubernetes通过PersistentVolume（PV）和PersistentVolumeClaim（PVC）抽象提供了存储资源的声明式管理，支持多种存储后端包括本地存储、网络存储、分布式存储和云存储等。本文将全面介绍Kubernetes存储机制、存储类的使用、存储卷的配置以及有状态应用的部署实践。
一、Kubernetes存储架构
1.1 存储概念概述
Kubernetes的存储模型包含几个核心概念：PersistentVolume（持久卷）是集群层面的存储资源，由管理员或通过StorageClass动态创建；PersistentVolumeClaim（持久卷声明）是用户对存储资源的请求，类似于Pod消耗Node资源的方式；StorageClass提供了动态存储供应的能力，简化了PV的管理；Volume是Pod中使用的临时或持久存储。
┌─────────────────────────────────────────────────────────────┐ │ 用户 (Pod) │ │ 创建PVC │ └─────────────────────┬───────────────────────────────────────┘ │ ┌─────────────────────▼───────────────────────────────────────┐ │ PersistentVolumeClaim │ │ (请求: 10Gi, ReadWriteOnce) │ └─────────────────────┬───────────────────────────────────────┘ │ 绑定 ┌─────────────────────▼───────────────────────────────────────┐ │ PersistentVolume │ │ (容量: 20Gi, 存储类: fast) │ └─────────────────────┬───────────────────────────────────────┘ │ 供应 ┌─────────────────────▼───────────────────────────────────────┐ │ StorageClass │ │ (供应者: csi-driver, 参数: type=ssd) │ └─────────────────────┬───────────────────────────────────────┘ │ 供应 ┌─────────────────────▼───────────────────────────────────────┐ │ 存储后端 │ │ (NFS / Ceph / AWS EBS / GCE PD / Azure Disk) │ └─────────────────────────────────────────────────────────────┘1.2 Volume类型
Kubernetes支持多种Volume类型：emptyDir用于临时存储，Pod删除时会被清除；hostPath将Node上的文件或目录挂载到Pod；nfs用于网络文件系统存储；persistentVolumeClaim引用预先配置的PV；configMap和secret用于配置和敏感信息；cloud存储如awsElasticBlockStore、gcePersistentDisk等。
二、PersistentVolume详解
2.1 PV创建示例
apiVersion: v1 kind: PersistentVolume metadata: name: pv-mysql-data labels: type: local app: mysql spec: capacity: storage: 50Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /data/mysql nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node-12.2 访问模式
ReadWriteOnce（RWO）是最常用的模式，存储卷只能被单个节点以读写方式挂载；ReadOnlyMany（ROX）允许存储卷被多个节点以只读方式挂载；ReadWriteMany（RWX）允许存储卷被多个节点以读写方式挂载；ReadWriteOncePod（RWOP）允许单个Pod独占访问存储卷。
2.3 回收策略
Retain策略保留数据，手动处理回收；Delete策略自动删除存储资源；Recycle策略会删除数据并重新变成可用状态（已弃用）。
三、StorageClass动态供应
3.1 创建StorageClass
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: fast-ssd provisioner: kubernetes.io/gce-pd parameters: type: pd-ssd replication-type: regional-pd fstype: ext4 reclaimPolicy: Retain allowVolumeExpansion: true mountOptions: - debug - noatime volumeBindingMode: WaitForFirstConsumer3.2 CSI存储驱动
Container Storage Interface（CSI）是Kubernetes存储的标准接口。
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-rook-ceph-block provisioner: rook.io/block parameters: poolName: replicapool clusterNamespace: rook-ceph imagePrefix: rook/ imageName: ceph-block imageFormat: "2" imageFeatures: layering reclaimPolicy: Retain allowVolumeExpansion: true volumeBindingMode: Immediate四、PersistentVolumeClaim
4.1 PVC基本配置
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-pvc namespace: production spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi storageClassName: fast-ssd selector: matchLabels: type: database4.2 PVC在Pod中使用
apiVersion: apps/v1 kind: Deployment metadata: name: mysql namespace: production spec: selector: matchLabels: app: mysql template: spec: containers: - name: mysql image: mysql:8.0 ports: - containerPort: 3306 name: mysql env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: mysql-secret key: password volumeMounts: - name: mysql-data mountPath: /var/lib/mysql - name: mysql-config mountPath: /etc/mysql/conf.d livenessProbe: exec: command: ["mysqladmin", "ping", "-h", "localhost"] initialDelaySeconds: 30 periodSeconds: 10 readinessProbe: exec: command: ["mysql", "-h", "localhost", "-u", "root", "-p$MYSQL_ROOT_PASSWORD", "-e", "SELECT 1"] initialDelaySeconds: 20 periodSeconds: 5 volumes: - name: mysql-data persistentVolumeClaim: claimName: mysql-pvc - name: mysql-config configMap: name: mysql-config五、有状态应用部署
5.1 StatefulSet简介
StatefulSet是Kubernetes用于管理有状态应用的资源对象，提供了稳定的网络标识、稳定的持久存储和有序的部署和扩展。
apiVersion: apps/v1 kind: StatefulSet metadata: name: mongodb namespace: production spec: serviceName: mongodb-headless replicas: 3 selector: matchLabels: app: mongodb template: metadata: labels: app: mongodb spec: terminationGracePeriodSeconds: 30 containers: - name: mongodb image: mongo:6.0 ports: - containerPort: 27017 name: client - containerPort: 27018 name: replica command: - mongod - --replSet - rs0 - --keyFile - /etc/mongodb-secret/keyfile - --bind_ip_all volumeMounts: - name: mongodb-data mountPath: /data/db - name: mongodb-config mountPath: /data/configdb resources: requests: cpu: 500m memory: 1Gi limits: cpu: "2" memory: 4Gi readinessProbe: exec: command: - mongo - --eval - "db.adminCommand('ping')" initialDelaySeconds: 10 periodSeconds: 10 volumes: - name: mongodb-config emptyDir: {} volumeClaimTemplates: - metadata: name: mongodb-data spec: accessModes: [ "ReadWriteOnce" ] storageClassName: fast-ssd resources: requests: storage: 50Gi5.2 Headless Service
apiVersion: v1 kind: Service metadata: name: mongodb-headless namespace: production spec: clusterIP: None selector: app: mongodb ports: - port: 27017 targetPort: 27017 name: client六、数据备份和恢复
6.1 VolumeSnapshot
apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: mysql-backup-20240115 namespace: production spec: volumeSnapshotClassName: csi-aws-vsc source: persistentVolumeClaimName: mysql-pvc6.2 从快照恢复
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-restored-pvc namespace: production spec: dataSource: name: mysql-backup-20240115 kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io accessModes: - ReadWriteOnce resources: requests: storage: 50Gi storageClassName: fast-ssd6.3 定时备份Job
apiVersion: batch/v1 kind: CronJob metadata: name: mysql-backup namespace: production spec: schedule: "0 2 * * *" successfulJobsHistoryLimit: 7 failedJobsHistoryLimit: 3 jobTemplate: spec: template: spec: serviceAccountName: backup-sa containers: - name: backup image: mysql:8.0 command: - /bin/bash - -c - | mysqldump --all-databases -u root -p$MYSQL_ROOT_PASSWORD | gzip > /backups/backup-$(date +%Y%m%d%H%M%S).sql.gz find /backups -type f -mtime +7 -delete env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: mysql-secret key: password volumeMounts: - name: backup-storage mountPath: /backups - name: mysql-secret readOnly: true mountPath: /etc/mysql-secret volumes: - name: backup-storage persistentVolumeClaim: claimName: backup-pvc - name: mysql-secret secret: secretName: mysql-secret restartPolicy: OnFailure七、存储性能优化
7.1 存储QoS
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: high-performance-pvc namespace: production spec: accessModes: - ReadWriteOnce resources: requests: storage: 100Gi storageClassName: premium-storage selector: matchExpressions: - key: performance operator: In values: - high7.2 Volume扩展
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 100Gi # 从50Gi扩展到100Gi storageClassName: fast-ssd八、存储监控
8.1 VolumeMetrics
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: monitored-pvc annotations: volume.beta.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io8.2 Prometheus存储指标
# kube-state-metrics会暴露以下指标 # kube_persistentvolume_capacity_bytes # kube_persistentvolume_status_phase # kube_persistentvolumeclaim_info # kube_persistentvolumeclaim_status_phase # kube_persistentvolumeclaim_resource_requests_storage_bytes九、最佳实践
9.1 存储规划建议
对于生产环境，存储规划应考虑：使用本地SSD或高性能网络存储作为数据库存储；为不同类型的数据配置不同的存储类；启用存储卷的自动扩展功能；实施定期备份策略；监控存储使用量并设置告警。
9.2 安全配置
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: secure-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: secure-storage --- # StorageClass启用加密 apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: secure-storage provisioner: pd.csi.storage.gke.io parameters: type: pd-standard encrypted: "true" keyRotationEnabled: "true"9.3 故障排查清单
当存储出现问题时，检查：PVC状态是否为Pending或Lost；PV是否存在且状态正确；StorageClass配置是否正确；存储驱动的Pod是否正常运行；Node是否有足够的存储空间；PVC的accessMode是否与存储类型兼容。
总结
Kubernetes的持久化存储机制为有状态应用提供了强大的支持。通过合理使用PV、PVC和StorageClass，可以实现存储资源的灵活管理和动态供应。StatefulSet和有状态应用的结合使用，使得在Kubernetes上运行数据库和其他有状态服务成为可能。在实际生产环境中，需要关注存储的性能、可靠性、安全性和可扩展性，建立完善的监控和备份机制，确保数据的安全和可用。
