4.23 项目实战：从零到生产级运维平台——整合全部知识

预计阅读时间：18 分钟

学习目标

完成本项目后，你将能够：

• 从零搭建完整的企业级 Linux 运维平台
• 整合服务器初始化、安全加固、Web环境搭建
• 配置 Prometheus + Grafana 监控告警体系
• 搭建集中日志收集与分析系统
• 使用 GitHub Actions 实现 CI/CD 流水线
• 使用 Ansible 实现自动化批量部署
• 建立数据库备份恢复和异地容灾机制
• 执行故障演练验证系统韧性

前置知识

开始本项目前，你需要掌握以下知识：

• Linux 基础命令和文件系统操作
• Shell 脚本编程基础
• 用户与权限管理
• 网络基础（IP、端口、防火墙）
• Nginx、MySQL、Redis 基础
• 系统监控的基本概念
• CI/CD 基本流程
• Ansible 基础操作

Ansible：无代理（Agentless）的自动化运维工具，通过 SSH 直连目标服务器执行任务。Playbook：Ansible 的配置和部署脚本，使用 YAML 格式定义任务编排。Role：Ansible 的任务复用单元，将 Playbook 按职责拆分为可复用的模块。Alertmanager：Prometheus 的告警管理组件，负责接收、去重、分组和路由告警通知。

项目背景与需求分析

业务场景

假设你是一家初创公司的运维工程师，公司需要为新上线的 Web 应用搭建完整的运维基础设施。应用采用经典的 LNMP 架构（Linux + Nginx + MySQL + PHP），需要具备高可用、可观测、可恢复的能力。

技术选型

架构设计

                    ┌─────────────────────────────────────┐
                    │         运维平台架构               │
                    └─────────────────────────────────────┘
                                    │
        ┌───────────────────────────┼───────────────────────────┐
        │                           │                           │
   ┌────┴────┐                 ┌────┴────┐                 ┌────┴────┐
   │ Web 层  │                 │ 监控层  │                 │ 自动化层│
   └────┬────┘                 └────┬────┘                 └────┬────┘
        │                           │                           │
   Nginx + PHP-FPM             Prometheus + Grafana         Ansible + CI/CD
        │                           │                           │
   MySQL + Redis              Alertmanager + 告警规则        GitHub Actions
        │                           │                           │
   SSL/TLS 证书               日志收集 + 告警            自动部署 + 回滚

环境准备

# 假设两台服务器：web-server(192.168.1.10) 和 monitor-server(192.168.1.20)
# 在两台服务器上执行

# 1. 更新系统
sudo apt update && sudo apt upgrade -y

# 2. 设置主机名
sudo hostnamectl set-hostname web-server    # Web 服务器
sudo hostnamectl set-hostname monitor-server # 监控服务器

# 3. 配置 hosts
cat > /etc/hosts <<EOF
192.168.1.10 web-server
192.168.1.20 monitor-server
EOF

# 4. 创建运维用户
sudo useradd -m -s /bin/bash ops
sudo usermod -aG sudo ops

# 5. 配置 SSH 密钥
sudo -u ops ssh-keygen -t ed25519 -C "ops@ops-platform"
sudo -u ops ssh-copy-id ops@192.168.1.10
sudo -u ops ssh-copy-id ops@192.168.1.20

第一阶段：服务器初始化

系统配置与安全加固

# 1. 设置时区和 NTP
sudo timedatectl set-timezone Asia/Shanghai
sudo timedatectl set-ntp true

# 2. 优化内核参数
cat > /etc/sysctl.d/99-ops-platform.conf <<EOF
net.ipv4.tcp_max_syn_backlog = 65536
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 30
net.core.somaxconn = 65536
fs.file-max = 2097152
vm.swappiness = 10
EOF
sudo sysctl -p /etc/sysctl.d/99-ops-platform.conf

# 3. 配置 SSH 安全
cat > /etc/ssh/sshd_config.d/ops-hardening.conf <<EOF
Port 22222
PermitRootLogin no
PasswordAuthentication no
MaxAuthTries 3
AllowUsers ops
EOF
sudo systemctl restart sshd

# 4. 配置防火墙
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22222/tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable

# 5. 安装 fail2ban
sudo apt install -y fail2ban
cat > /etc/fail2ban/jail.local <<EOF
[sshd]
enabled = true
port = 22222
maxretry = 3
bantime = 86400
EOF
sudo systemctl enable fail2ban

# 6. 安装 Node Exporter（监控代理）
wget https://github.com/prometheus/node_exporter/releases/download/v1.9.0/node_exporter-1.9.0.linux-amd64.tar.gz
tar xzf node_exporter-1.9.0.linux-amd64.tar.gz
sudo useradd --no-create-home --shell /bin/false node_exporter
sudo mv node_exporter-1.9.0.linux-amd64/node_exporter /usr/local/bin/

# 创建 systemd 服务
cat > /etc/systemd/system/node_exporter.service <<EOF
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter --web.listen-address=:9100
Restart=always

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

# 验证监控代理
curl -s http://localhost:9100/metrics | head -3
# 输出：# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.

# 7. 配置系统日志
cat > /etc/rsyslog.d/50-ops-platform.conf <<EOF
# 运维平台日志配置
*.* @@192.168.1.20:514
EOF
sudo systemctl restart rsyslog

# 8. 验证服务器状态
echo "=== 服务器状态验证 ==="
echo "主机名: $(hostname)"
echo "IP 地址: $(hostname -I | awk '{print $1}')"
echo "系统版本: $(cat /etc/os-release | grep PRETTY_NAME | cut -d'"' -f2)"
echo "内核版本: $(uname -r)"
echo "内存使用: $(free -h | grep Mem | awk '{print $3"/"$2}')"
echo "磁盘使用: $(df -h / | tail -1 | awk '{print $3"/"$2" ("$5")"}')"
systemctl status node_exporter | grep Active

第二阶段：Web 环境搭建

软件源配置

# 1. 添加 PHP 官方源（Ubuntu）
sudo add-apt-repository ppa:ondrej/php -y
sudo apt update

# 2. 添加 MySQL 官方源
wget https://dev.mysql.com/get/mysql-apt-config_0.8.30-1_all.deb
sudo dpkg -i mysql-apt-config_0.8.30-1_all.deb

# 3. 添加 Redis 官方源
sudo apt install -y lsb-release
curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/redis.list

# 4. 更新软件包索引
sudo apt update

MySQL 安装与配置

# 1. 安装 MySQL
sudo apt install -y mysql-server

# 2. 安全初始化
sudo mysql_secure_installation

# 3. 创建应用数据库
sudo mysql -u root -p <<EOSQL
CREATE DATABASE ops_platform CHARACTER SET utf8mb4;
CREATE USER 'ops_app'@'localhost' IDENTIFIED BY 'StrongPassword123!';
GRANT ALL PRIVILEGES ON ops_platform.* TO 'ops_app'@'localhost';
FLUSH PRIVILEGES;
EOSQL

# 4. 优化配置
cat > /etc/mysql/mysql.conf.d/ops-platform.cnf <<EOF
[mysqld]
server-id = 1
log-bin = mysql-bin
innodb_buffer_pool_size = 1G
max_connections = 500
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
long_query_time = 2
EOF
sudo systemctl restart mysql

Redis 安装与配置

# 1. 安装 Redis
sudo apt install -y redis-server

# 2. 优化配置
cat > /etc/redis/redis.conf.d/ops-platform.conf <<EOF
bind 127.0.0.1
requirepass RedisStrongPass123!
maxmemory 512mb
maxmemory-policy allkeys-lru
save 900 1
save 300 10
save 60 10000
EOF
sudo systemctl restart redis-server

PHP-FPM 安装与配置

# 1. 安装 PHP-FPM 和扩展
sudo apt install -y php8.1-fpm php8.1-mysql php8.1-redis php8.1-mbstring php8.1-xml php8.1-curl

# 2. 优化配置
cat > /etc/php/8.1/fpm/pool.d/ops-platform.conf <<EOF
[ops-platform]
user = www-data
listen = /run/php/php8.1-fpm-ops.sock
pm = dynamic
pm.max_children = 50
pm.start_servers = 10
pm.min_spare_servers = 5
pm.max_spare_servers = 20
slowlog = /var/log/php8.1-fpm/ops-platform-slow.log
request_slowlog_timeout = 5s
EOF
sudo systemctl restart php8.1-fpm

Nginx 安装与配置

# 1. 安装 Nginx
sudo apt install -y nginx

# 2. 创建站点配置
cat > /etc/nginx/sites-available/ops-platform <<EOF
server {
    listen 80;
    server_name ops.example.com;
    return 301 https://\$host\$request_uri;
}

server {
    listen 443 ssl http2;
    server_name ops.example.com;
    ssl_certificate /etc/ssl/ops-platform/cert.pem;
    ssl_certificate_key /etc/ssl/ops-platform/key.pem;
    root /var/www/ops-platform;
    index index.php index.html;

    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;

    location ~ \.php\$ {
        fastcgi_pass unix:/run/php/php8.1-fpm-ops.sock;
        fastcgi_param SCRIPT_FILENAME \$document_root\$fastcgi_script_name;
        include fastcgi_params;
    }

    location ~* \.(jpg|jpeg|png|css|js)$ {
        expires 30d;
    }

    location /health {
        access_log off;
        return 200 "OK";
    }
}
EOF

sudo ln -s /etc/nginx/sites-available/ops-platform /etc/nginx/sites-enabled/
sudo rm -f /etc/nginx/sites-enabled/default
sudo mkdir -p /var/www/ops-platform
echo "<?php phpinfo(); ?>" | sudo tee /var/www/ops-platform/index.php
sudo chown -R www-data:www-data /var/www/ops-platform
sudo nginx -t && sudo systemctl restart nginx

SSL/TLS 证书配置

# 自签名证书（测试环境）
sudo mkdir -p /etc/ssl/ops-platform
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
    -keyout /etc/ssl/ops-platform/key.pem \
    -out /etc/ssl/ops-platform/cert.pem \
    -subj "/C=CN/ST=Beijing/O=OpsPlatform/CN=ops.example.com"

# 生产环境使用 Let's Encrypt
# sudo apt install -y certbot python3-certbot-nginx
# sudo certbot --nginx -d ops.example.com

第三阶段：监控告警系统

Prometheus 安装与配置

# 在监控服务器（192.168.1.20）上执行

# 1. 安装 Prometheus
sudo useradd --no-create-home --shell /bin/false prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.49.1/prometheus-2.49.1.linux-amd64.tar.gz
tar xzf prometheus-2.49.1.linux-amd64.tar.gz
sudo mv prometheus-2.49.1.linux-amd64/{prometheus,promtool} /usr/local/bin/
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

# 2. 配置抓取目标
cat > /etc/prometheus/prometheus.yml <<EOF
global:
  scrape_interval: 15s

rule_files:
  - "rules/*.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'web-server'
    static_configs:
      - targets: ['192.168.1.10:9100']
  - job_name: 'mysql'
    static_configs:
      - targets: ['192.168.1.10:9104']
  - job_name: 'redis'
    static_configs:
      - targets: ['192.168.1.10:9121']
EOF

# 3. 创建告警规则
mkdir -p /etc/prometheus/rules
cat > /etc/prometheus/rules/ops-platform.yml <<EOF
groups:
  - name: ops-platform-alerts
    rules:
      - alert: HighCpuUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "CPU 使用率过高 ({{ \$labels.instance }})"
      - alert: HighMemoryUsage
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "内存使用率过高 ({{ \$labels.instance }})"
      - alert: DiskSpaceLow
        expr: (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 85
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "磁盘空间不足 ({{ \$labels.instance }})"
EOF

# 4. 启动服务
cat > /etc/systemd/system/prometheus.service <<EOF
[Unit]
Description=Prometheus
After=network.target

[Service]
User=prometheus
ExecStart=/usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --storage.tsdb.retention.time=30d
Restart=always

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus

Grafana 安装与配置

# 1. 安装 Grafana
sudo apt-get install -y apt-transport-https software-properties-common wget
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update && sudo apt-get install -y grafana

# 2. 启动服务
sudo systemctl enable grafana-server
sudo systemctl start grafana-server

# 3. 访问地址：http://localhost:3000
# 默认用户: admin / admin（首次登录需修改）

# 4. 添加 Prometheus 数据源（通过 API）
curl -X POST http://admin:admin@localhost:3000/api/datasources \
    -H "Content-Type: application/json" \
    -d '{"name":"Prometheus","type":"prometheus","url":"http://localhost:9090","access":"proxy","isDefault":true}'

Alertmanager 配置

# 1. 安装 Alertmanager
wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
tar xzf alertmanager-0.27.0.linux-amd64.tar.gz
sudo mv alertmanager-0.27.0.linux-amd64/{alertmanager,amtool} /usr/local/bin/
sudo useradd --no-create-home --shell /bin/false alertmanager

# 2. 配置告警接收
cat > /etc/alertmanager/alertmanager.yml <<EOF
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'ops-team'

receivers:
  - name: 'ops-team'
    email_configs:
      - to: 'ops@example.com'
        from: 'alertmanager@example.com'
        smarthost: 'smtp.example.com:587'
        auth_username: 'alertmanager@example.com'
        auth_password: 'email-password'
EOF

# 3. 启动服务
cat > /etc/systemd/system/alertmanager.service <<EOF
[Unit]
Description=Alertmanager
After=network.target

[Service]
User=alertmanager
ExecStart=/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml
Restart=always

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable alertmanager
sudo systemctl start alertmanager

第四阶段：日志收集系统

rsyslog 与 logrotate 配置

# 在 Web 服务器上执行

# 1. 配置 rsyslog 发送日志
cat > /etc/rsyslog.d/50-ops-platform.conf <<EOF
# 转发日志到监控服务器
*.* @@192.168.1.20:514
EOF
sudo systemctl restart rsyslog

# 2. 配置日志轮转
cat > /etc/logrotate.d/ops-platform <<EOF
/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
    endscript
}

/var/log/mysql/*.log {
    daily
    rotate 7
    compress
    notifempty
    create 0640 mysql adm
    postrotate
        /usr/bin/mysqladmin flush-logs
    endscript
}
EOF

# 3. 在监控服务器上配置日志接收
cat > /etc/rsyslog.d/50-ops-receive.conf <<EOF
module(load="imtcp")
input(type="imtcp" port="514")

template(name="OpsLog" type="string"
    string="/var/log/ops-platform/%HOSTNAME%/%PROGRAMNAME%.log")

if $hostname == 'web-server' then {
    action(type="omfile" dynaFile="OpsLog")
    stop
}
EOF
sudo mkdir -p /var/log/ops-platform/web-server
sudo systemctl restart rsyslog

日志分析脚本

#!/bin/bash
# log-analysis.sh - 日志分析脚本

LOG_DIR="/var/log/nginx"
REPORT="/tmp/log-report-$(date +%Y%m%d).txt"

echo "=== 日志分析报告 ===" > "$REPORT"
echo "总访问量: $(wc -l < ${LOG_DIR}/ops-platform-access.log)" >> "$REPORT"
echo "唯一IP数: $(awk '{print $1}' ${LOG_DIR}/ops-platform-access.log | sort -u | wc -l)" >> "$REPORT"
echo "" >> "$REPORT"

echo "--- 状态码分布 ---" >> "$REPORT"
awk '{print $9}' ${LOG_DIR}/ops-platform-access.log | sort | uniq -c | sort -rn >> "$REPORT"

echo "--- 404 页面 TOP 10 ---" >> "$REPORT"
awk '$9 == 404 {print $7}' ${LOG_DIR}/ops-platform-access.log | sort | uniq -c | sort -rn | head -10 >> "$REPORT"

cat "$REPORT"

第五阶段：CI/CD 流水线

GitHub Actions 工作流

# .github/workflows/ci-cd.yml
name: CI/CD Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: shivammathur/setup-php@v2
        with:
          php-version: '8.1'
      - run: composer install --prefer-dist
      - run: vendor/bin/phpunit

  build:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@v5
        with:
          push: ${{ github.event_name != 'pull_request' }}
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}

  deploy:
    needs: build
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: appleboy/ssh-action@v1
        with:
          host: ${{ secrets.SERVER_HOST }}
          username: ${{ secrets.SERVER_USER }}
          key: ${{ secrets.SERVER_SSH_KEY }}
          script: |
            cd /opt/ops-platform
            git pull origin main
            docker-compose pull
            docker-compose up -d

Docker 容器化

# Dockerfile
FROM php:8.1-fpm-alpine
RUN apk add --no-cache nginx supervisor
RUN docker-php-ext-install pdo_mysql mbstring opcache
COPY src/ /var/www/html/
COPY deploy/nginx.conf /etc/nginx/http.d/default.conf
COPY deploy/supervisord.conf /etc/supervisor/conf.d/
RUN chown -R nginx:nginx /var/www/html
EXPOSE 80
CMD ["/usr/bin/supervisord"]

# docker-compose.yml
version: '3.8'
services:
  app:
    build: .
    ports:
      - "8080:80"
    depends_on:
      - mysql
      - redis
  mysql:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: RootPassword123!
      MYSQL_DATABASE: ops_platform
    volumes:
      - mysql-data:/var/lib/mysql
  redis:
    image: redis:7-alpine
    command: redis-server --requirepass RedisStrongPass123!
volumes:
  mysql-data:

第六阶段：自动化部署

Ansible Playbook

# inventory/hosts.ini
[webservers]
web-server ansible_host=192.168.1.10 ansible_user=ops

[monitoring]
monitor-server ansible_host=192.168.1.20 ansible_user=ops

# playbooks/deploy.yml
---
- name: Deploy ops-platform
  hosts: webservers
  become: yes
  tasks:
    - name: Install packages
      apt:
        name: [docker.io, docker-compose, git]
        state: present
        update_cache: yes

    - name: Ensure Docker running
      systemd:
        name: docker
        state: started
        enabled: yes

    - name: Clone application
      git:
        repo: "https://github.com/yourusername/ops-platform.git"
        dest: /opt/ops-platform
        version: main
        force: yes

    - name: Deploy application
      command: docker-compose up -d
      args:
        chdir: /opt/ops-platform

# playbooks/configure.yml
---
- name: Configure servers
  hosts: all
  become: yes
  tasks:
    - name: Set timezone
      timezone:
        name: Asia/Shanghai

    - name: SSH hardening
      lineinfile:
        path: /etc/ssh/sshd_config
        regexp: "{{ item.regexp }}"
        line: "{{ item.line }}"
      loop:
        - { regexp: '^#?PermitRootLogin', line: 'PermitRootLogin no' }
        - { regexp: '^#?PasswordAuthentication', line: 'PasswordAuthentication no' }
      notify: restart sshd

    - name: Configure UFW
      ufw:
        rule: allow
        port: "{{ item }}"
        proto: tcp
      loop: ['22', '80', '443']

  handlers:
    - name: restart sshd
      systemd:
        name: sshd
        state: restarted

Ansible Role 设计

# 创建 Role 目录结构
mkdir -p ansible/roles/webserver/{tasks,handlers,templates,files,defaults,vars}
mkdir -p ansible/roles/mysql/{tasks,handlers,templates,defaults}
mkdir -p ansible/roles/redis/{tasks,defaults}

# Role: webserver
cat > ansible/roles/webserver/tasks/main.yml <<EOF
---
- name: Install Nginx and PHP-FPM
  apt:
    name:
      - nginx
      - php8.1-fpm
      - php8.1-mysql
      - php8.1-redis
      - php8.1-mbstring
      - php8.1-xml
    state: present
    update_cache: yes

- name: Deploy Nginx configuration
  template:
    src: nginx.conf.j2
    dest: /etc/nginx/sites-available/ops-platform
  notify: restart nginx

- name: Enable site
  file:
    src: /etc/nginx/sites-available/ops-platform
    dest: /etc/nginx/sites-enabled/ops-platform
    state: link
  notify: restart nginx

- name: Start services
  systemd:
    name: "{{ item }}"
    state: started
    enabled: yes
  loop:
    - nginx
    - php8.1-fpm
EOF

cat > ansible/roles/webserver/handlers/main.yml <<EOF
---
- name: restart nginx
  systemd:
    name: nginx
    state: restarted

- name: restart php-fpm
  systemd:
    name: php8.1-fpm
    state: restarted
EOF

cat > ansible/roles/webserver/defaults/main.yml <<EOF
---
nginx_worker_processes: auto
nginx_worker_connections: 1024
nginx_listen_port: 80
nginx_server_name: ops.example.com
php_max_children: 50
php_start_servers: 10
EOF

# Role: mysql
cat > ansible/roles/mysql/tasks/main.yml <<EOF
---
- name: Install MySQL
  apt:
    name:
      - mysql-server
      - mysql-client
    state: present
    update_cache: yes

- name: Deploy MySQL configuration
  template:
    src: mysql.cnf.j2
    dest: /etc/mysql/mysql.conf.d/ops-platform.cnf
  notify: restart mysql

- name: Create application database
  mysql_db:
    name: "{{ db_name }}"
    state: present
    collation: utf8mb4_unicode_ci

- name: Create application user
  mysql_user:
    name: "{{ db_user }}"
    password: "{{ db_pass }}"
    priv: "{{ db_name }}.*:ALL"
    state: present
EOF

cat > ansible/roles/mysql/handlers/main.yml <<EOF
---
- name: restart mysql
  systemd:
    name: mysql
    state: restarted
EOF

执行部署

# 执行部署
cd ansible
ansible-playbook -i inventory/hosts.ini playbooks/deploy.yml

# 输出示例：
# PLAY [Deploy ops-platform] ***************************************************
# TASK [Gathering Facts] *******************************************************
# ok: [web-server]
# TASK [Install packages] ******************************************************
# changed: [web-server]
# PLAY RECAP *******************************************************************
# web-server : ok=5    changed=3    unreachable=0    failed=0

第七阶段：备份恢复

数据库备份

#!/bin/bash
# mysql-backup.sh - MySQL 备份脚本

BACKUP_DIR="/var/backups/mysql"
DATE=$(date +%Y%m%d_%H%M%S)
RETENTION_DAYS=7

mkdir -p "${BACKUP_DIR}/${DATE}"

# 备份所有数据库
for DB in $(mysql -u ops_app -e 'SHOW DATABASES;' | grep -v 'Database\|information_schema\|performance_schema\|mysql\|sys'); do
    mysqldump -u ops_app --single-transaction "${DB}" | gzip > "${BACKUP_DIR}/${DATE}/${DB}.sql.gz"
done

# 清理过期备份
find "${BACKUP_DIR}" -maxdepth 1 -type d -mtime +${RETENTION_DAYS} -exec rm -rf {} \;
echo "备份完成: ${BACKUP_DIR}/${DATE}"

异地备份与恢复

# 1. 异地备份（rsync）
#!/bin/bash
# remote-backup.sh
rsync -avz -e "ssh -p 22222" /var/backups/ backup@backup-server:/backup/web-server/

# 1.5 配置备份（config-backup.sh）
#!/bin/bash
# config-backup.sh —— 备份关键配置目录
BACKUP_DIR="/var/backups/config"
mkdir -p "$BACKUP_DIR"
tar czf "$BACKUP_DIR/config-$(date +%F).tar.gz" \
    /etc/nginx /etc/mysql /etc/prometheus /etc/alertmanager /etc/grafana 2>/dev/null
find "$BACKUP_DIR" -name "*.tar.gz" -mtime +30 -delete

# 2. 恢复数据库
#!/bin/bash
# mysql-restore.sh
BACKUP_FILE="$1"
if [ -z "$BACKUP_FILE" ]; then
    echo "用法: $0 <备份文件.sql.gz>"
    exit 1
fi
zcat "$BACKUP_FILE" | mysql -u root
echo "恢复完成"

# 4. 恢复配置
#!/bin/bash
# config-restore.sh
BACKUP_FILE="$1"
if [ -z "$BACKUP_FILE" ]; then
    echo "用法: $0 <备份文件.tar.gz>"
    exit 1
fi
tar xzf "$BACKUP_FILE" -C /
echo "恢复完成，请重启相关服务"

# 4. 验证备份完整性
#!/bin/bash
# verify-backup.sh
BACKUP_FILE="$1"
echo "验证备份文件: $BACKUP_FILE"
if gzip -t "$BACKUP_FILE" 2>/dev/null; then
    echo "备份文件完整"
else
    echo "备份文件损坏"
    exit 1
fi

定时备份

# 设置定时任务
# 每天凌晨 2 点备份数据库
0 2 * * * /usr/local/bin/mysql-backup.sh >> /var/log/mysql-backup.log 2>&1
# 每天凌晨 4 点异地备份
0 4 * * * /usr/local/bin/remote-backup.sh >> /var/log/remote-backup.log 2>&1
# 每周日凌晨 3 点备份配置
0 3 * * 0 /usr/local/bin/config-backup.sh >> /var/log/config-backup.log 2>&1

第八阶段：故障演练

故障注入与恢复

# 1. 模拟 Nginx 宕机
echo "=== 故障演练：Nginx 宕机 ==="
sudo systemctl stop nginx
sleep 5
# 观察 Grafana 仪表盘和告警
sudo systemctl start nginx

# 2. 模拟磁盘空间不足
echo "=== 故障演练：磁盘空间不足 ==="
dd if=/dev/zero of=/tmp/fill-disk bs=1M count=10240
sleep 5
rm -f /tmp/fill-disk

# 3. 模拟高 CPU 使用率
echo "=== 故障演练：高 CPU ==="
sudo apt install -y stress    # 先安装（apt 可能报 stress 不在发行版仓库，若如此用 htop 观察即可）
stress --cpu 4 --timeout 60 &
sleep 5

# 4. 验证监控告警
curl -s http://localhost:9093/api/v2/alerts | jq '.[].labels.alertname'

监控验证与告警测试

# 1. 检查 Prometheus 告警状态
curl -s http://localhost:9093/api/v2/alerts | jq '.[] | {labels: .labels, status: .status.state}'
# 输出示例：
# {
#   "labels": { "alertname": "ServiceDown", "instance": "web-server" },
#   "status": { "state": "active" }
# }

# 2. 检查 Grafana 告警
curl -s http://admin:admin@localhost:3000/api/alerts | jq '.[].state'

# 3. 恢复服务并验证
echo "=== 恢复服务 ==="
sudo systemctl start nginx
rm -f /tmp/fill-disk
echo "服务已恢复"

# 4. 验证告警恢复
sleep 60
curl -s http://localhost:9093/api/v2/alerts | jq '.[] | .labels.alertname'
# 输出：应该为空（告警已恢复）

恢复流程文档

# 故障级别定义
# P0: 服务完全不可用（30 分钟内响应）
# P1: 核心功能受损（2 小时内响应）
# P2: 非核心功能受损（24 小时内响应）

# 恢复步骤
# 1. 确认故障现象
# 2. 检查监控告警
# 3. 定位故障原因
# 4. 执行恢复操作
# 5. 验证服务恢复
# 6. 记录事件报告

# 常用恢复命令
systemctl restart nginx      # 重启 Nginx
systemctl restart mysql      # 重启 MySQL
mysql-restore.sh backup.sql  # 恢复数据库
config-restore.sh backup.tar.gz  # 恢复配置

# 事件报告模板
mkdir -p /docs   # 报告目录（一次性准备）
cat > /docs/incident-report-$(date +%Y%m%d).md <<EOF
# 故障事件报告

## 基本信息
- 事件编号: INC-$(date +%Y%m%d)
- 发现时间: $(date)
- 影响范围: [描述]

## 故障描述
[详细描述故障现象]

## 根本原因
[分析故障原因]

## 影响评估
- 业务影响: [描述]
- 持续时间: [时间]

## 恢复措施
1. [步骤 1]
2. [步骤 2]

## 改进建议
- [建议 1]
- [建议 2]
EOF

测试验证

# 完整验证流程
echo "=== 运维平台验证 ==="

# 1. Web 服务
curl -s -o /dev/null -w "%{http_code}" https://ops.example.com/health
# 输出：200

# 2. 数据库
mysql -u ops_app -e "SELECT 1;" ops_platform
# 输出：1

# 3. Redis
redis-cli -a 'RedisStrongPass123!' ping
# 输出：PONG

# 4. 监控
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets | length'
# 输出：4

# 5. 备份
ls -la /var/backups/mysql/ | tail -3
# 输出：最近的备份文件

常见错误

最佳实践

• 安全第一：所有服务使用非 root 用户运行，密码使用密钥管理
• 监控先行：部署服务前先配置监控，确保可观测性
• 备份验证：定期测试备份恢复流程，确保数据可恢复
• 自动化：所有重复性操作都应该自动化，减少人为错误
• 文档化：所有配置和流程都应该有文档记录
• 版本控制：所有配置文件使用 Git 管理，记录变更历史
• 告警优化：避免告警疲劳，设置合理的阈值和通知方式
• 定期演练：定期进行故障演练，验证系统韧性

练习题

1. 为运维平台添加 Nginx 监控，使用 nginx-prometheus-exporter 收集指标
2. 编写 Ansible Role 实现 MySQL 主从复制配置
3. 配置 Prometheus 告警发送到钉钉或企业微信
4. 实现数据库自动备份到云存储（如 S3、OSS）
5. 设计并实施完整的故障演练方案，包括网络分区、服务降级等场景

学习检查点

学完本章后，请检验自己是否掌握以下内容：

本章总结

本项目从零搭建了完整的企业级运维平台，整合了全部 6 编知识：

掌握这套运维平台的搭建和运维，是成为合格运维工程师的重要一步。实际生产中还需根据业务需求进行定制和扩展。

延伸阅读