8.8 项目实战：生产级 Kubernetes 集群运维

预计阅读时间：24 分钟

第八编系统讲解了 Kubernetes 集群部署、Pod 管理、Service 网络、存储编排、调度策略、资源管理、监控与日志、etcd 备份与恢复等单项技术。本项目将这些知识点串联成一个完整的生产级集群运维流程，从零搭建多节点 K8s 集群，配置 HPA 自动伸缩、NetworkPolicy 网络策略、Prometheus+Grafana 监控、EFK 日志收集、etcd 备份恢复，最后进行故障演练验证集群自愈能力。

学习目标

学完本项目后，你将能够：

• 掌握生产级 K8s 集群的完整部署流程：kubeadm 初始化 → 节点加入 → 网络插件安装
• 能够独立完成 HPA 自动伸缩、NetworkPolicy 网络策略的配置与验证
• 理解 Prometheus+Grafana 监控栈和 EFK 日志收集栈的架构与部署
• 具备 etcd 备份恢复和故障演练的实战能力

前置知识

在开始本项目之前，建议先掌握以下知识：

• K8s 集群架构——控制面组件（kube-apiserver、etcd、scheduler、controller-manager）与工作节点组件（kubelet、kube-proxy）
• Docker 容器运行时基础——镜像、容器、Dockerfile 概念
• K8s 核心对象——Pod、Deployment、Service、ConfigMap、Secret 的基本用法
• K8s 网络模型——CNI 插件、Service 类型（ClusterIP、NodePort、LoadBalancer）
• K8s 资源管理——requests/limits、ResourceQuota、LimitRange
• Linux 系统管理基础——systemctl、防火墙、系统内核参数

kubeadm：Kubernetes 官方提供的集群初始化工具，用于快速搭建 K8s 集群。etcd：分布式键值存储系统，K8s 使用它存储所有集群状态数据。Calico：支持 NetworkPolicy 的 CNI（Container Network Interface）网络插件，提供网络策略和 IP 地址管理。HPA（Horizontal Pod Autoscaler）：水平 Pod 自动伸缩器，根据 CPU/内存使用率自动调整 Pod 副本数。NetworkPolicy：K8s 网络策略资源，用于控制 Pod 之间的网络流量，实现微分段隔离。

项目背景与架构设计

在生产环境中，Kubernetes 集群需要满足高可用、可观测、可恢复三大核心要求。本项目将构建一个 6 节点的 K8s 集群，完整覆盖生产级运维的核心能力。

集群拓扑

Control Plane (3 节点高可用)
    ├── kube-apiserver (负载均衡)
    ├── etcd 集群 (Raft 共识)
    ├── kube-scheduler
    └── kube-controller-manager

Worker Nodes (3 节点)
    ├── kubelet
    ├── kube-proxy
    └── 容器运行时 (containerd)

监控栈
    ├── Prometheus (指标采集)
    ├── Grafana (可视化)
    └── Alertmanager (告警)

日志栈
    ├── Elasticsearch (存储与索引)
    ├── Fluentd (日志收集)
    └── Kibana (查询界面)

技术选型

环境准备

在开始部署前，确保所有节点满足以下条件：

硬件要求

网络规划

# 节点 IP 规划
Control Plane:
  master-1: 192.168.1.10
  master-2: 192.168.1.11
  master-3: 192.168.1.12

Worker Nodes:
  worker-1: 192.168.1.20
  worker-2: 192.168.1.21
  worker-3: 192.168.1.22

Pod 网络: 10.244.0.0/16
Service 网络: 10.96.0.0/12
API Server 虚拟 IP: 192.168.1.100

系统初始化（所有节点执行）

# 1. 设置主机名
sudo hostnamectl set-hostname master-1   # master-1 节点
sudo hostnamectl set-hostname worker-1   # worker-1 节点（以此类推）

# 2. 配置 hosts 文件
sudo tee /etc/hosts <<'EOF'
192.168.1.10 master-1
192.168.1.11 master-2
192.168.1.12 master-3
192.168.1.20 worker-1
192.168.1.21 worker-2
192.168.1.22 worker-3
192.168.1.100 api-vip
EOF

# 3. 关闭 swap（K8s 强制要求）
sudo swapoff -a
sudo sed -i '/\sswap\s/s/^/#/' /etc/fstab

# 4. 加载内核模块
sudo tee /etc/modules-load.d/k8s.conf <<'EOF'
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter

# 5. 配置内核参数
sudo tee /etc/sysctl.d/k8s.conf <<'EOF'
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sudo sysctl --system

# 6. 配置防火墙（开放集群端口）
sudo firewall-cmd --permanent --add-port=6443/tcp    # API Server
sudo firewall-cmd --permanent --add-port=2379-2380/tcp  # etcd
sudo firewall-cmd --permanent --add-port=10250/tcp   # kubelet
sudo firewall-cmd --permanent --add-port=10251/tcp   # scheduler（K8s 1.24+ 已改为 10259）
sudo firewall-cmd --permanent --add-port=10252/tcp   # controller-manager（K8s 1.24+ 已改为 10257）
sudo firewall-cmd --permanent --add-port=10257/tcp   # controller-manager (secure)
sudo firewall-cmd --permanent --add-port=10259/tcp   # scheduler (secure)
sudo firewall-cmd --permanent --add-port=30000-32767/tcp  # NodePort
sudo firewall-cmd --reload

# 7. 配置 containerd
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install -y containerd.io

# 生成默认配置
sudo containerd config default | sudo tee /etc/containerd/config.toml
# 启用 SystemdCgroup
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
# 使用阿里云镜像源
sudo sed -i 's|registry.k8s.io/pause:3.8|registry.aliyuncs.com/google_containers/pause:3.9|' /etc/containerd/config.toml

sudo systemctl enable containerd
sudo systemctl restart containerd

第一步：部署 K8s 集群

使用 kubeadm 部署多节点高可用 K8s 集群，包括安装集群组件、初始化控制面和加入工作节点。

1.1 安装 kubeadm/kubelet/kubectl

# 配置 Kubernetes yum 仓库
sudo tee /etc/yum.repos.d/kubernetes.repo <<'EOF'
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

# 安装集群组件
sudo yum install -y kubelet-1.32.2 kubeadm-1.32.2 kubectl-1.32.2 --disableexcludes=kubernetes

# 设置 kubelet 自启动
sudo systemctl enable kubelet

# 验证安装
kubeadm version
kubelet --version
kubectl version --client

1.2 初始化 Control Plane（master-1 节点）

# 创建 kubeadm 配置文件
cat <<EOF | tee kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.32.2
controlPlaneEndpoint: "api-vip:6443"
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/12"
  dnsDomain: "cluster.local"
apiServer:
  extraArgs:
    enable-admission-plugins: "NodeRestriction"
etcd:
  local:
    extraArgs:
      quota-backend-bytes: "8589934592"  # 8GB etcd 存储配额
      auto-compaction-retention: "8"
controllerManager:
  extraArgs:
    terminated-pod-gc-threshold: "100"
scheduler:
  extraArgs:
    profiling: "false"
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
EOF

# 初始化集群
sudo kubeadm init --config kubeadm-config.yaml --upload-certs

# 输出示例（重要）：
# Your Kubernetes control plane has initialized successfully!
# To start using your cluster, you need to run the following as a regular user:
#   mkdir -p $HOME/.kube
#   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
#   sudo chown $(id -u):$(id -g) $HOME/.kube/config
#
# Then you can join any number of control-plane nodes by running:
#   kubeadm join api-vip:6443 --token xxx --discovery-token-ca-cert-hash sha256:xxx --control-plane --certificate-key xxx
#
# Then you can join any number of worker nodes by running:
#   kubeadm join api-vip:6443 --token xxx --discovery-token-ca-cert-hash sha256:xxx

1.3 配置 kubectl

# 配置当前用户 kubectl
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# 验证集群连接
kubectl get nodes
# 预期输出：
# NAME       STATUS     ROLES           AGE   VERSION
# master-1   NotReady   control-plane   2m    v1.32.2

# 查看所有系统组件
kubectl get pods -n kube-system
# 预期输出：
# NAME                               READY   STATUS    RESTARTS   AGE
# coredns-xxx                        0/1     Pending   0          2m
# etcd-master-1                      1/1     Running   0          3m
# kube-apiserver-master-1            1/1     Running   0          3m
# kube-controller-manager-master-1   1/1     Running   0          3m
# kube-proxy-xxx                     1/1     Running   0          2m
# kube-scheduler-master-1            1/1     Running   0          3m

1.4 加入其他 Control Plane 节点

# 在 master-2 和 master-3 上执行 kubeadm join（使用初始化输出的命令）
# 示例格式：
sudo kubeadm join api-vip:6443 \
  --token abc123.xyz456 \
  --discovery-token-ca-cert-hash sha256:xxxxx \
  --control-plane \
  --certificate-key xxxxx

# 验证控制面节点
kubectl get nodes
# 预期输出：
# NAME       STATUS     ROLES           AGE     VERSION
# master-1   NotReady   control-plane   10m     v1.32.2
# master-2   NotReady   control-plane   5m      v1.32.2
# master-3   NotReady   control-plane   3m      v1.32.2

1.5 安装网络插件（Calico）

# 安装 Calico 网络插件（支持 NetworkPolicy）
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/calico.yaml

# 等待所有 Pod 运行
watch kubectl get pods -n kube-system
# 预期输出（所有 Pod Running）：
# NAME                               READY   STATUS    RESTARTS   AGE
# calico-kube-controllers-xxx        1/1     Running   0          2m
# calico-node-xxx                    1/1     Running   0          2m
# coredns-xxx                        1/1     Running   0          15m
# ...

# 验证节点状态变为 Ready
kubectl get nodes
# 预期输出：
# NAME       STATUS   ROLES           AGE   VERSION
# master-1   Ready    control-plane   15m   v1.32.2
# master-2   Ready    control-plane   10m   v1.32.2
# master-3   Ready    control-plane   8m    v1.32.2

# 配置 Calico 检测网卡
kubectl -n kube-system set env daemonset/calico-node FELIX_IGNORELOOSEROUTE=0
# 或通过 ConfigMap 设置
kubectl -n kube-system patch configmap calico-config --type merge -p \
  '{"data":{"cni_network_config":"{...}"}}'

1.6 加入 Worker 节点

# 在 worker-1, worker-2, worker-3 上执行
sudo kubeadm join api-vip:6443 \
  --token abc123.xyz456 \
  --discovery-token-ca-cert-hash sha256:xxxxx

# 验证所有节点
kubectl get nodes -o wide
# 预期输出：
# NAME       STATUS   ROLES           AGE     VERSION   INTERNAL-IP     OS-IMAGE
# master-1   Ready    control-plane   20m     v1.32.2   192.168.1.10    CentOS Stream 9
# master-2   Ready    control-plane   15m     v1.32.2   192.168.1.11    CentOS Stream 9
# master-3   Ready    control-plane   13m     v1.32.2   192.168.1.12    CentOS Stream 9
# worker-1   Ready    <none>          5m      v1.32.2   192.168.1.20    CentOS Stream 9
# worker-2   Ready    <none>          3m      v1.32.2   192.168.1.21    CentOS Stream 9
# worker-3   Ready    <none>          1m      v1.32.2   192.168.1.22    CentOS Stream 9

# 为 Worker 节点添加标签
kubectl label node worker-1 node-role.kubernetes.io/worker=worker
kubectl label node worker-2 node-role.kubernetes.io/worker=worker
kubectl label node worker-3 node-role.kubernetes.io/worker=worker

第二步：部署应用

部署一个示例应用来验证集群功能，包括 Deployment、Service 和 Ingress。

2.1 创建 Deployment

# 创建示例应用 Deployment
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  labels:
    app: webapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      containers:
      - name: nginx
        image: nginx:1.25
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 256Mi
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 15
          periodSeconds: 20
EOF

# 查看 Deployment 状态
kubectl get deployment webapp
# 预期输出：
# NAME     READY   UP-TO-DATE   AVAILABLE   AGE
# webapp   3/3     3            3           30s

# 查看 Pod 状态
kubectl get pods -l app=webapp -o wide
# 预期输出：
# NAME                      READY   STATUS    RESTARTS   AGE   IP            NODE
# webapp-xxx-abc12          1/1     Running   0          30s   10.244.1.5    worker-1
# webapp-xxx-def34          1/1     Running   0          30s   10.244.2.8    worker-2
# webapp-xxx-ghi56          1/1     Running   0          30s   10.244.3.3    worker-3

2.2 创建 Service

# 创建 ClusterIP Service（集群内部访问）
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: webapp-svc
  labels:
    app: webapp
spec:
  type: ClusterIP
  selector:
    app: webapp
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
EOF

# 创建 NodePort Service（外部访问）
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: webapp-nodeport
  labels:
    app: webapp
spec:
  type: NodePort
  selector:
    app: webapp
  ports:
  - port: 80
    targetPort: 80
    nodePort: 30080
    protocol: TCP
EOF

# 验证 Service
kubectl get svc webapp-svc webapp-nodeport
# 预期输出：
# NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
# webapp-svc        ClusterIP   10.96.100.50    <none>        80/TCP         10s
# webapp-nodeport   NodePort    10.96.100.60    <none>        80:30080/TCP   10s

# 测试集群内部访问
kubectl run test-pod --rm -it --image=curlimages/curl -- curl -s http://webapp-svc
# 预期输出：Nginx 欢迎页面 HTML

2.3 创建 Ingress

# 安装 Nginx Ingress Controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.10.0/deploy/static/provider/cloud/deploy.yaml

# 等待 Ingress Controller 运行
kubectl get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx

# 创建 Ingress 规则
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: webapp-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/proxy-body-size: "10m"
spec:
  ingressClassName: nginx
  rules:
  - host: webapp.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: webapp-svc
            port:
              number: 80
  - host: api.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: webapp-svc
            port:
              number: 80
EOF

# 验证 Ingress
kubectl get ingress webapp-ingress
# 预期输出：
# NAME              CLASS   HOSTS                 ADDRESS         PORTS   AGE
# webapp-ingress    nginx   webapp.example.com    10.100.100.10   80      30s
#                   api.example.com

第三步：配置 HPA 自动伸缩

Horizontal Pod Autoscaler (HPA) 可以根据 CPU 或内存使用率自动调整 Pod 副本数。

3.1 安装 Metrics Server

# 安装 Metrics Server（HPA 依赖）
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# 如果使用自签名证书，需要添加 --kubelet-insecure-tls 参数
kubectl -n kube-system patch deployment metrics-server --type=json -p='[
  {"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}
]'

# 等待 Metrics Server 就绪
kubectl -n kube-system get deployment metrics-server
# 预期输出：
# NAME             READY   UP-TO-DATE   AVAILABLE   AGE
# metrics-server   1/1     1            1           2m

# 验证 Metrics Server
kubectl top nodes
# 预期输出：
# NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
# master-1   250m         12%    1024Mi          25%
# master-2   230m         11%    980Mi           24%
# worker-1   150m         7%     2048Mi          25%

3.2 创建 HPA 配置

# 为 webapp Deployment 创建 HPA
kubectl autoscale deployment webapp \
  --cpu-percent=70 \
  --min=3 \
  --max=20

# 验证 HPA 配置
kubectl get hpa
# 预期输出：
# NAME     REFERENCE          TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
# webapp   Deployment/webapp  5%/70%          3         20        3          30s

# 查看 HPA 详细信息
kubectl describe hpa webapp

# 也可以通过 YAML 文件创建（更灵活）
cat <<EOF | kubectl apply -f -
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: webapp-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: webapp
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
      - type: Pods
        value: 4
        periodSeconds: 60
      - type: Percent
        value: 100
        periodSeconds: 60
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Pods
        value: 2
        periodSeconds: 120
      selectPolicy: Min
EOF

3.3 测试自动伸缩

# 创建临时测试 Pod 生成 CPU 负载
kubectl run load-generator --rm -it --image=busybox -- /bin/sh

# 在 load-generator Pod 内执行（生成 CPU 压力）
while true; do wget -q -O- http://webapp-svc; done

# 另开终端观察 HPA 状态变化（每 10 秒刷新）
watch -n 10 kubectl get hpa webapp

# 预期输出（CPU 使用率上升后）：
# NAME     REFERENCE          TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
# webapp   Deployment/webapp  85%/70%         3         20        6          5m

# 观察 Pod 数量增加
kubectl get pods -l app=webapp
# 预期输出：Pod 数量从 3 个增加到更多

# 停止负载后观察 Pod 缩减
# Ctrl+C 停止 load-generator
kubectl get hpa webapp --watch
# 观察 REPLICAS 列逐渐缩减回 3

第四步：配置 NetworkPolicy

NetworkPolicy 用于控制 Pod 之间的网络流量，实现微分段隔离。Calico 原生支持 NetworkPolicy。

4.1 默认拒绝策略

# 为 default 命名空间设置默认拒绝所有入站流量
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: default
spec:
  podSelector: {}
  policyTypes:
  - Ingress
EOF

# 验证策略已应用
kubectl get networkpolicy
# 预期输出：
# NAME                  POD-SELECTOR   AGE
# default-deny-ingress  <none>         10s

# 测试：从 test-pod 访问 webapp（应该超时）
kubectl run test-blocked --rm -it --image=curlimages/curl -- curl -s --connect-timeout 5 http://webapp-svc
# 预期输出：连接超时（被 NetworkPolicy 阻止）

4.2 允许特定流量

# 允许 webapp 命名空间内的 Pod 互相访问
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-webapp-ingress
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: webapp
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: default
    - podSelector:
        matchLabels:
          app: webapp
    ports:
    - port: 80
      protocol: TCP
EOF

# 允许 Ingress Controller 访问 webapp
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-controller
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: webapp
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: ingress-nginx
    ports:
    - port: 80
      protocol: TCP
EOF

# 允许出站 DNS 解析
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: default
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector: {}
    ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
EOF

4.3 测试网络策略

# 从 webapp Pod 内部测试 DNS 解析
kubectl exec -it deploy/webapp -- nslookup kubernetes.default
# 预期输出：成功解析（DNS egress 允许）

# 从 webapp Pod 内部测试访问其他 webapp Pod
kubectl exec -it deploy/webapp -- wget -q -O- http://webapp-svc
# 预期输出：成功返回（ingress 规则允许）

# 测试外部访问（通过 Ingress）
curl -H "Host: webapp.example.com" http://<INGRESS_IP>
# 预期输出：成功返回（ingress-nginx 命名空间允许访问）

# 查看 NetworkPolicy 效果（Calico）
kubectl get networkpolicy -o wide
# 查看 Calico 策略
kubectl get networkpolicies.crd.projectcalico.org -A

第五步：配置监控

使用 kube-prometheus-stack 部署完整的监控栈，包括 Prometheus、Grafana 和 Alertmanager。

5.1 部署 Prometheus Stack

# 添加 Helm 仓库
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# 创建监控命名空间
kubectl create namespace monitoring

# 创建 Prometheus values 文件
cat <<EOF | tee prometheus-values.yaml
# Prometheus 配置
prometheus:
  prometheusSpec:
    retention: 15d
    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 50Gi
    resources:
      requests:
        cpu: 500m
        memory: 1Gi
      limits:
        cpu: 2
        memory: 4Gi
    nodeSelector:
      node-role.kubernetes.io/control-plane: ""
    tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule

# Grafana 配置
grafana:
  adminPassword: "admin123"
  persistence:
    enabled: true
    size: 10Gi
  dashboards:
    default:
      kubernetes-cluster:
        gnetId: 7249
        revision: 1
        datasource: Prometheus
      node-exporter:
        gnetId: 1860
        revision: 33
        datasource: Prometheus
  ingress:
    enabled: true
    ingressClassName: nginx
    hosts:
    - grafana.example.com

# Alertmanager 配置
alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi
EOF

# 安装 Prometheus Stack
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --values prometheus-values.yaml

# 等待部署完成
kubectl -n monitoring get pods
# 预期输出：
# NAME                                                     READY   STATUS    RESTARTS   AGE
# alertmanager-prometheus-kube-prom-alertmanager-0          2/2     Running   0          5m
# grafana-xxx                                              3/3     Running   0          5m
# kube-prometheus-kube-prom-operator-xxx                   1/1     Running   0          5m
# prometheus-kube-prom-prometheus-0                         2/2     Running   0          5m
# prometheus-node-exporter-xxx                              1/1     Running   0          5m

5.2 部署 Grafana 仪表盘

# 获取 Grafana 端口
kubectl -n monitoring get svc grafana
# 预期输出：
# NAME      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
# grafana   ClusterIP   10.96.200.100   <none>        80/TCP    5m

# 通过端口转发访问 Grafana
kubectl -n monitoring port-forward svc/grafana 3000:80 &

# 在浏览器访问 http://localhost:3000
# 用户名: admin  密码: admin123

# 导入常用仪表盘
# 仪表盘 ID: 7249 (Kubernetes Cluster Monitoring)
# 仪表盘 ID: 1860 (Node Exporter Full)
# 仪表盘 ID: 315 (Kubernetes Apps)

# 通过 API 导入仪表盘（自动化）
GRAFANA_POD=$(kubectl -n monitoring get pods -l app.kubernetes.io/name=grafana -o jsonpath='{.items[0].metadata.name}')
kubectl -n monitoring exec $GRAFANA_POD -- wget -q -O- \
  "https://grafana.com/api/dashboards/7249/revisions/latest/download" | \
  kubectl -n monitoring exec -i $GRAFANA_POD -- tee /tmp/dashboard.json > /dev/null

5.3 配置告警规则

# 创建自定义告警规则
cat <<EOF | kubectl apply -f -
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: k8s-custom-alerts
  namespace: monitoring
  labels:
    release: prometheus
spec:
  groups:
  - name: k8s-node-alerts
    rules:
    - alert: NodeNotReady
      expr: kube_node_status_condition{condition="Ready",status="true"} == 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "节点 {{ \$labels.node }} 不可用"
        description: "节点 {{ \$labels.node }} 已超过 5 分钟处于 NotReady 状态"

    - alert: PodCrashLooping
      expr: rate(kube_pod_container_status_restarts_total[15m]) * 60 * 5 > 0
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Pod {{ \$labels.namespace }}/{{ \$labels.pod }} 频繁重启"
        description: "Pod 在过去 15 分钟内重启超过 5 次"

    - alert: HighMemoryUsage
      expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "节点 {{ \$labels.instance }} 内存使用率过高"
        description: "节点内存使用率超过 90%，当前值: {{ \$value }}%"

    - alert: HighCPUUsage
      expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "节点 {{ \$labels.instance }} CPU 使用率过高"
        description: "节点 CPU 使用率超过 85%，持续 10 分钟"

  - name: k8s-pvc-alerts
    rules:
    - alert: PVCNearFull
      expr: (kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes) * 100 > 85
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "PVC {{ \$labels.namespace }}/{{ \$labels.persistentvolumeclaim }} 即将满"
        description: "PVC 使用率超过 85%，当前值: {{ \$value }}%"

  - name: k8s-deployment-alerts
    rules:
    - alert: DeploymentReplicasMismatch
      expr: kube_deployment_spec_replicas != kube_deployment_status_ready_replicas
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "Deployment {{ \$labels.namespace }}/{{ \$labels.deployment }} 副本数不匹配"
        description: "期望副本数 {{ \$value }} 与就绪副本数不一致"

    - alert: DeploymentReplicasUnavailable
      expr: kube_deployment_status_unavailable_replicas > 0
      for: 10m
      labels:
        severity: critical
      annotations:
        summary: "Deployment {{ \$labels.namespace }}/{{ \$labels.deployment }} 有不可用副本"
        description: "存在 {{ \$value }} 个不可用副本"
EOF

# 查看告警规则
kubectl -n monitoring get prometheusrules
# 预期输出：
# NAME                       AGE
# k8s-custom-alerts          10s
# prometheus-kube-prom-...   5m

5.4 配置 Alertmanager 告警接收

# 创建 Alertmanager 配置
cat <<EOF | kubectl apply -f -
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: custom-config
  namespace: monitoring
  labels:
    release: prometheus
spec:
  receivers:
  - name: default
    webhookConfigs:
    - url: 'http://alertmanager-webhook:9095/webhook'
  - name: critical
    webhookConfigs:
    - url: 'http://alertmanager-webhook:9095/webhook'
  route:
    receiver: default
    groupBy:
    - alertname
    - namespace
    - severity
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 4h
    routes:
    - receiver: critical
      match:
        severity: critical
      groupWait: 10s
      repeatInterval: 1h
EOF

# 通过 PrometheusRule 配置告警路由（更简单的方式）
cat <<EOF | kubectl apply -f -
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: monitoring
spec:
  ruleSelector:
    matchLabels:
      release: prometheus
  alerting:
    alertmanagers:
    - namespace: monitoring
      name: prometheus-kube-prom-alertmanager
      port: web
EOF

第六步：配置日志收集

使用 EFK（Elasticsearch + Fluentd + Kibana）栈收集和查询集群日志。

6.1 部署 EFK 栈

# 创建日志命名空间
kubectl create namespace logging

# 部署 Elasticsearch
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: logging
spec:
  serviceName: elasticsearch
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      initContainers:
      - name: sysctl
        image: busybox
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      containers:
      - name: elasticsearch
        image: elasticsearch:8.11.0
        env:
        - name: discovery.seed_hosts
          value: "elasticsearch"
        - name: cluster.initial_master_nodes
          value: "elasticsearch-0,elasticsearch-1,elasticsearch-2"
        - name: ES_JAVA_OPTS
          value: "-Xms1g -Xmx1g"
        - name: xpack.security.enabled
          value: "false"
        ports:
        - containerPort: 9200
          name: rest
        - containerPort: 9300
          name: inter-node
        resources:
          requests:
            cpu: 500m
            memory: 2Gi
          limits:
            cpu: 1
            memory: 4Gi
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 30Gi
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: logging
spec:
  selector:
    app: elasticsearch
  ports:
  - port: 9200
    name: rest
  - port: 9300
    name: inter-node
  clusterIP: None
EOF

# 部署 Fluentd DaemonSet
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: logging
data:
  fluent.conf: |
    <source>
      @type tail
      @id in_tail_container_logs
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      exclude_path ["/var/log/containers/fluentd*"]
      read_from_head true
      <parse>
        @type multi_format
        <pattern>
          format json
          time_key time
          time_format %Y-%m-%dT%H:%M:%S.%NZ
          keep_time_key true
        </pattern>
        <pattern>
          format regexp
          expression /^(?.time) (??.stream) (??.log)$/
          time_format %Y-%m-%dT%H:%M:%S.%NZ
        </pattern>
      </parse>
    </source>

    <filter kubernetes.**>
      @type kubernetes_metadata
      @id filter_kube_metadata
    </filter>

    <match **>
      @type elasticsearch
      @id out_es
      @log_level info
      include_tag_key true
      host elasticsearch.logging.svc.cluster.local
      port 9200
      logstash_format true
      logstash_prefix k8s-logs
      logstash_dateformat %Y.%m.%d
      <buffer>
        @type file
        path /var/log/fluentd-buffers/kubernetes.system.buffer
        flush_mode interval
        flush_thread_count 2
        flush_interval 5s
        retry_type exponential_backoff
        retry_forever true
        retry_max_interval 30
        chunk_limit_size 2M
        queue_limit_length 8
        overflow_action block
      </buffer>
    </match>
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      serviceAccountName: fluentd
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1.16-debian-elasticsearch8-1
        env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: "elasticsearch.logging.svc.cluster.local"
        - name: FLUENT_ELASTICSEARCH_PORT
          value: "9200"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: containers
          mountPath: /var/log/containers
          readOnly: true
        - name: config
          mountPath: /fluentd/etc/
        resources:
          requests:
            cpu: 100m
            memory: 200Mi
          limits:
            cpu: 500m
            memory: 500Mi
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: containers
        hostPath:
          path: /var/log/containers
      - name: config
        configMap:
          name: fluentd-config
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentd
rules:
- apiGroups: [""]
  resources: ["pods", "namespaces"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluentd
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluentd
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: logging
EOF

# 部署 Kibana
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: logging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: kibana:8.11.0
        env:
        - name: ELASTICSEARCH_HOSTS
          value: http://elasticsearch:9200
        ports:
        - containerPort: 5601
        resources:
          requests:
            cpu: 200m
            memory: 512Mi
          limits:
            cpu: 1
            memory: 2Gi
---
apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: logging
spec:
  selector:
    app: kibana
  ports:
  - port: 5601
    targetPort: 5601
  type: NodePort
EOF

# 验证 EFK 部署
kubectl -n logging get pods
# 预期输出：
# NAME                            READY   STATUS    RESTARTS   AGE
# elasticsearch-0                 1/1     Running   0          5m
# elasticsearch-1                 1/1     Running   0          5m
# elasticsearch-2                 1/1     Running   0          5m
# fluentd-xxx                     1/1     Running   0          3m
# kibana-xxx                      1/1     Running   0          3m

6.2 查询日志

# 通过 Kibana 查询日志
# 访问 http://<NODE_IP>:<NodePort> (Kibana 服务端口)

# 通过 Elasticsearch API 直接查询
kubectl -n logging exec -it elasticsearch-0 -- \
  curl -s "http://localhost:9200/_cat/indices?v"
# 预期输出：
# health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
# green  open   k8s-logs-2026.07.30             xxx                    1   1      12345            0       15mb           7.5mb

# 查询最近日志
kubectl -n logging exec -it elasticsearch-0 -- \
  curl -s "http://localhost:9200/k8s-logs-*/_search?pretty" -H 'Content-Type: application/json' -d '{
    "query": {
      "match_all": {}
    },
    "sort": [{"@timestamp": {"order": "desc"}}],
    "size": 10
  }' | head -50

# Fluentd 状态检查
kubectl -n logging logs daemonset/fluentd --tail=20
# 预期输出：显示日志收集状态

第七步：备份与恢复

定期备份 etcd 数据，确保集群状态可恢复。

7.1 备份 etcd

# 创建备份目录
sudo mkdir -p /backup/etcd

# 手动备份 etcd（在控制面节点执行）
sudo ETCDCTL_API=3 etcdctl snapshot save /backup/etcd/snapshot-$(date +%F-%H%M).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# 验证快照
sudo ETCDCTL_API=3 etcdctl snapshot status /backup/etcd/snapshot-$(date +%F-%H%M).db -w table
# 预期输出：
# +----------+----------+------------+------------+
# |   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
# +----------+----------+------------+------------+
# | xxxx     |  123456  |     5678   |    120 MB  |
# +----------+----------+------------+------------+

# 创建自动化备份 CronJob
cat <<EOF | kubectl apply -f -
apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-backup
  namespace: kube-system
spec:
  schedule: "0 */6 * * *"  # 每 6 小时执行一次
  jobTemplate:
    spec:
      template:
        spec:
          nodeName: master-1
          hostNetwork: true
          containers:
          - name: etcdctl
            image: bitnami/etcd:3.5
            env:
            - name: ETCDCTL_API
              value: "3"
            - name: ETCDCTL_ENDPOINTS
              value: https://127.0.0.1:2379
            - name: ETCDCTL_CACERT
              value: /certs/ca.crt
            - name: ETCDCTL_CERT
              value: /certs/server.crt
            - name: ETCDCTL_KEY
              value: /certs/server.key
            command:
            - /bin/sh
            - -c
            - |
              BACKUP_FILE="/backup/etcd-\$(date +%F-%H%M).db"
              etcdctl snapshot save \$BACKUP_FILE
              etcdctl snapshot status \$BACKUP_FILE -w table
              # 清理 7 天前的备份
              find /backup -name "etcd-*.db" -mtime +7 -delete
            volumeMounts:
            - name: etcd-certs
              mountPath: /certs
              readOnly: true
            - name: backup
              mountPath: /backup
          restartPolicy: OnFailure
          volumes:
          - name: etcd-certs
            hostPath:
              path: /etc/kubernetes/pki/etcd
              type: Directory
          - name: backup
            hostPath:
              path: /backup/etcd
              type: DirectoryOrCreate
EOF

7.2 恢复测试

# 1. 停止控制面组件
sudo systemctl stop kubelet
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
sudo mv /etc/kubernetes/manifests/kube-controller-manager.yaml /tmp/
sudo mv /etc/kubernetes/manifests/kube-scheduler.yaml /tmp/

# 2. 备份当前 etcd 数据
sudo mv /var/lib/etcd /var/lib/etcd.bak

# 3. 恢复快照
sudo ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd/snapshot-2026-07-30-0600.db \
  --data-dir=/var/lib/etcd \
  --name=master-1 \
  --initial-cluster="master-1=https://192.168.1.10:2380,master-2=https://192.168.1.11:2380,master-3=https://192.168.1.12:2380" \
  --initial-advertise-peer-urls=https://192.168.1.10:2380 \
  --initial-cluster-token=k8s-etcd-cluster

# 4. 重启控制面组件
sudo mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/
sudo mv /tmp/kube-controller-manager.yaml /etc/kubernetes/manifests/
sudo mv /tmp/kube-scheduler.yaml /etc/kubernetes/manifests/
sudo systemctl start kubelet

# 5. 验证恢复
kubectl get nodes
kubectl get pods -A

# 6. 清理备份目录
sudo rm -rf /var/lib/etcd.bak

第八步：故障演练

通过模拟故障验证集群的自愈能力和高可用性。

8.1 模拟节点故障

# 模拟 Worker 节点故障（关闭 kubelet）
ssh worker-1 "sudo systemctl stop kubelet"

# 观察 Pod 状态变化
watch kubectl get pods -l app=webapp -o wide
# 预期输出：Pod 状态变为 Terminating，其他节点创建新 Pod

# 查看节点状态
kubectl get nodes
# 预期输出：worker-1 状态变为 NotReady

# 观察节点状况事件
kubectl describe node worker-1 | grep -A 5 "Conditions:"
# 预期输出：Ready 状态变为 False

# 恢复节点
ssh worker-1 "sudo systemctl start kubelet"
kubectl get nodes
# 预期输出：worker-1 状态恢复为 Ready

8.2 模拟 Pod 故障

# 强制删除一个 Pod
kubectl delete pod -l app=webapp --grace-period=0 --force

# 观察 Pod 重建
watch kubectl get pods -l app=webapp
# 预期输出：Deployment controller 自动创建新 Pod，保持 3 个副本

# 模拟应用 OOM（内存溢出）
kubectl run oom-test --image=nginx --limits=memory=64Mi -- /bin/sh -c "dd if=/dev/zero of=/tmp/test bs=1M count=100"

# 观察 Pod 重启
kubectl get pods -l run=oom-test -w
# 预期输出：Pod 状态变为 OOMKilled → Running（重启）

# 清理测试 Pod
kubectl delete pod oom-test --grace-period=0 --force

8.3 模拟 etcd 故障

# 模拟 etcd 节点故障（停止 etcd）
ssh master-3 "sudo crictl stop \$(sudo crictl ps -q --name etcd)"

# 验证集群仍可用（剩余 2 个 etcd 节点仍满足多数派）
kubectl get nodes
kubectl get pods -A

# 恢复 etcd
ssh master-3 "sudo crictl start \$(sudo crictl ps -q --name etcd)"

# 验证 etcd 集群健康
kubectl -n kube-system exec etcd-master-1 -- \
  etcdctl endpoint health --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

8.4 模拟网络分区

# 使用 iptables 模拟网络分区（阻断特定 Pod 的流量）
sudo iptables -A INPUT -s 10.244.1.0/24 -j DROP
sudo iptables -A OUTPUT -d 10.244.1.0/24 -j DROP

# 观察 Pod 状态（可能变为 Pending）
kubectl get pods -o wide

# 恢复网络
sudo iptables -D INPUT -s 10.244.1.0/24 -j DROP
sudo iptables -D OUTPUT -d 10.244.1.0/24 -j DROP

验证集群状态

完成所有步骤后，执行以下命令验证集群健康状态：

# 1. 检查所有节点状态
echo "===== 节点状态 ====="
kubectl get nodes -o wide

# 2. 检查所有系统 Pod
echo "===== 系统 Pod ====="
kubectl get pods -A | grep -E "kube-system|monitoring|logging|ingress-nginx"

# 3. 检查应用 Pod
echo "===== 应用 Pod ====="
kubectl get pods -l app=webapp -o wide

# 4. 检查 HPA 状态
echo "===== HPA ====="
kubectl get hpa

# 5. 检查 NetworkPolicy
echo "===== NetworkPolicy ====="
kubectl get networkpolicy

# 6. 检查 Service 和 Ingress
echo "===== Service ====="
kubectl get svc -A | grep -E "webapp|grafana|kibana|elasticsearch"

# 7. 检查 etcd 健康
echo "===== etcd 健康 ====="
kubectl -n kube-system exec etcd-master-1 -- \
  etcdctl endpoint health --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# 8. 检查资源使用
echo "===== 资源使用 ====="
kubectl top nodes
kubectl top pods -A --sort-by=memory | head -15

常见错误

最佳实践

练习题

1. （基础）按照本项目步骤搭建完整集群，记录每步输出结果。
2. （进阶）配置 Pod 反亲和，确保 webapp 的 3 个副本分布在不同 Worker 节点。
3. （进阶）在 Grafana 中创建自定义仪表盘，展示 HPA 伸缩历史和集群资源使用趋势。
4. （高级）将整个集群部署过程改写为 Ansible Playbook，实现一键部署。
5. （高级）配置 PodDisruptionBudget，确保滚动更新时至少保持 2 个可用副本。
6. （探究）研究 K8s Secret 加密（EncryptionConfiguration），思考如何保护 etcd 中的敏感数据。
7. （探究）研究 K8s 节点维护流程（cordon/drain/uncordon），设计生产环境的滚动维护方案。
8. （探究）对比 Calico 与 Flannel 的 NetworkPolicy 支持差异，说明为什么生产环境推荐 Calico。

学习检查点

学完本章后，请检验自己是否掌握以下内容：

本章总结

本项目从零搭建了生产级 K8s 集群，覆盖了集群部署、应用部署、自动伸缩、网络策略、监控日志、备份恢复和故障演练的完整流程。以下是核心要点回顾：

掌握这套集群运维流程，是进入 Kubernetes 生产环境运维的重要一步。实际生产中还需考虑多集群管理、GitOps、服务网格等进阶话题。

延伸阅读