手动部署k8s(七)-kube-apiserver高可用之Nginx代理

April 7, 2019

984views

5676字数

> 原教程来自 [github/opsnull](https://github.com/opsnull/follow-me-install-kubernetes-cluster), 现在此基础上记录自己搭建遇到的问题

基于 nginx 代理的 kube-apiserver 高可用方案
- 控制节点的 kube-controller-manager、kube-scheduler 是多实例部署，所以只要有一个实例正常，就可以保证高可用；
- 集群内的 Pod 使用 K8S 服务域名 kubernetes 访问 kube-apiserver， kube-dns 会自动解析出多个 kube-apiserver 节点的 IP，所以也是高可用的；
- 在每个节点起一个 nginx 进程，后端对接多个 apiserver 实例，nginx 对它们做健康检查和负载均衡；
- kubelet、kube-proxy、controller-manager、scheduler 通过本地的 nginx（监听 127.0.0.1）访问 kube-apiserver，从而实现 kube-apiserver 的高可用；

#### 下载和编译 nginx
下载源码：
```
cd /opt/k8s/work
wget http://nginx.org/download/nginx-1.15.3.tar.gz
tar -xzvf nginx-1.15.3.tar.gz
```

配置编译参数：
```
cd /opt/k8s/work/nginx-1.15.3
mkdir nginx-prefix
./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module
```

- --with-stream：开启 4 层透明转发(TCP Proxy)功能；
- --without-xxx：关闭所有其他功能，这样生成的动态链接二进制程序依赖最小；

输出：
```
Configuration summary
  + PCRE library is not used
  + OpenSSL library is not used
  + zlib library is not used

nginx path prefix: "/opt/k8s/work/nginx-1.15.3/nginx-prefix"
  nginx binary file: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx"
  nginx modules path: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/modules"
  nginx configuration prefix: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/conf"
  nginx configuration file: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/conf/nginx.conf"
  nginx pid file: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/logs/nginx.pid"
  nginx error log file: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/logs/error.log"
  nginx http access log file: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/logs/access.log"
  nginx http client request body temporary files: "client_body_temp"
  nginx http proxy temporary files: "proxy_temp"
```

编译和安装：
```
cd /opt/k8s/work/nginx-1.15.3
make && make install
```

#### 验证编译的 nginx
```
cd /opt/k8s/work/nginx-1.15.3
./nginx-prefix/sbin/nginx -v
```

输出：
```
[root@ _14_ /opt/k8s/work/nginx-1.15.3]# ./nginx-prefix/sbin/nginx -v
nginx version: nginx/1.15.3

```

查看 nginx 动态链接的库：
```
[root@ _15_ /opt/k8s/work/nginx-1.15.3]# ldd ./nginx-prefix/sbin/nginx
```
输出
```
	linux-vdso.so.1 =>  (0x00007ffe3ed70000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fe8876fe000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe8874e2000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fe887121000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fe887902000)
```
由于只开启了 4 层透明转发功能，所以除了依赖 libc 等操作系统核心 lib 库外，没有对其它 lib 的依赖(如 libz、libssl 等)，这样可以方便部署到各版本操作系统中；

#### 安装和部署 nginx
创建目录结构：
```
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}
  done
```
拷贝二进制程序：
```
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
    scp /opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx  root@${node_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx
    ssh root@${node_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*"
  done
```

- 重命名二进制文件为 kube-nginx

配置 nginx，开启 4 层透明转发功能
```
cd /opt/k8s/work
cat > kube-nginx.conf <<EOF
worker_processes 1;

events {
    worker_connections  1024;
}

stream {
    upstream backend {
        hash $remote_addr consistent;
        server 192.168.1.31:6443        max_fails=3 fail_timeout=30s;
        server 192.168.1.32:6443        max_fails=3 fail_timeout=30s;
        server 192.168.1.32:6443        max_fails=3 fail_timeout=30s;
    }

server {
        listen 127.0.0.1:8443;
        proxy_connect_timeout 1s;
        proxy_pass backend;
    }
}
EOF
```

- 需要根据集群 kube-apiserver 的实际情况，替换 backend 中 server 列表

分发配置文件：
```
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-nginx.conf  root@${node_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.conf
  done
```

#### 配置 systemd unit 文件，启动服务
配置 kube-nginx systemd unit 文件：
```
cd /opt/k8s/work
cat > kube-nginx.service <<EOF
[Unit]
Description=kube-apiserver nginx proxy
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=forking
ExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t
ExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx
ExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload
PrivateTmp=true
Restart=always
RestartSec=5
StartLimitInterval=0
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF
```

分发 systemd unit 文件：
```
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-nginx.service  root@${node_ip}:/etc/systemd/system/
  done
```
启动 kube-nginx 服务：
```
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-nginx && systemctl restart kube-nginx"
  done
```

#### 检查 kube-nginx 服务运行状态
```
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status kube-nginx |grep 'Active:'"
  done
```

确保状态为 active (running)，否则查看日志，确认原因：
```
journalctl -u kube-nginx
```

手动部署k8s(七)-kube-apiserver高可用之Nginx代理