> 原教程来自 [github/opsnull](https://github.com/opsnull/follow-me-install-kubernetes-cluster), 现在此基础上记录自己搭建遇到的问题 基于 nginx 代理的 kube-apiserver 高可用方案 - 控制节点的 kube-controller-manager、kube-scheduler 是多实例部署，所以只要有一个实例正常，就可以保证高可用； - 集群内的 Pod 使用 K8S 服务域名 kubernetes 访问 kube-apiserver， kube-dns 会自动解析出多个 kube-apiserver 节点的 IP，所以也是高可用的； - 在每个节点起一个 nginx 进程，后端对接多个 apiserver 实例，nginx 对它们做健康检查和负载均衡； - kubelet、kube-proxy、controller-manager、scheduler 通过本地的 nginx（监听 127.0.0.1）访问 kube-apiserver，从而实现 kube-apiserver 的高可用； #### 下载和编译 nginx 下载源码： ``` cd /opt/k8s/work wget http://nginx.org/download/nginx-1.15.3.tar.gz tar -xzvf nginx-1.15.3.tar.gz ``` 配置编译参数： ``` cd /opt/k8s/work/nginx-1.15.3 mkdir nginx-prefix ./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module ``` - --with-stream：开启 4 层透明转发(TCP Proxy)功能； - --without-xxx：关闭所有其他功能，这样生成的动态链接二进制程序依赖最小； 输出： ``` Configuration summary + PCRE library is not used + OpenSSL library is not used + zlib library is not used nginx path prefix: "/opt/k8s/work/nginx-1.15.3/nginx-prefix" nginx binary file: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx" nginx modules path: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/modules" nginx configuration prefix: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/conf" nginx configuration file: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/conf/nginx.conf" nginx pid file: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/logs/nginx.pid" nginx error log file: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/logs/error.log" nginx http access log file: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/logs/access.log" nginx http client request body temporary files: "client_body_temp" nginx http proxy temporary files: "proxy_temp" ``` 编译和安装： ``` cd /opt/k8s/work/nginx-1.15.3 make && make install ``` #### 验证编译的 nginx ``` cd /opt/k8s/work/nginx-1.15.3 ./nginx-prefix/sbin/nginx -v ``` 输出： ``` [root@ _14_ /opt/k8s/work/nginx-1.15.3]# ./nginx-prefix/sbin/nginx -v nginx version: nginx/1.15.3 ``` 查看 nginx 动态链接的库： ``` [root@ _15_ /opt/k8s/work/nginx-1.15.3]# ldd ./nginx-prefix/sbin/nginx ``` 输出 ``` linux-vdso.so.1 => (0x00007ffe3ed70000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fe8876fe000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe8874e2000) libc.so.6 => /lib64/libc.so.6 (0x00007fe887121000) /lib64/ld-linux-x86-64.so.2 (0x00007fe887902000) ``` 由于只开启了 4 层透明转发功能，所以除了依赖 libc 等操作系统核心 lib 库外，没有对其它 lib 的依赖(如 libz、libssl 等)，这样可以方便部署到各版本操作系统中； #### 安装和部署 nginx 创建目录结构： ``` cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin} done ``` 拷贝二进制程序： ``` cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" ssh root@${node_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}" scp /opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx root@${node_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx ssh root@${node_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*" done ``` - 重命名二进制文件为 kube-nginx 配置 nginx，开启 4 层透明转发功能 ``` cd /opt/k8s/work cat > kube-nginx.conf <<EOF worker_processes 1; events { worker_connections 1024; } stream { upstream backend { hash $remote_addr consistent; server 192.168.1.31:6443 max_fails=3 fail_timeout=30s; server 192.168.1.32:6443 max_fails=3 fail_timeout=30s; server 192.168.1.32:6443 max_fails=3 fail_timeout=30s; } server { listen 127.0.0.1:8443; proxy_connect_timeout 1s; proxy_pass backend; } } EOF ``` - 需要根据集群 kube-apiserver 的实际情况，替换 backend 中 server 列表 分发配置文件： ``` cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" scp kube-nginx.conf root@${node_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.conf done ``` #### 配置 systemd unit 文件，启动服务 配置 kube-nginx systemd unit 文件： ``` cd /opt/k8s/work cat > kube-nginx.service <<EOF [Unit] Description=kube-apiserver nginx proxy After=network.target After=network-online.target Wants=network-online.target [Service] Type=forking ExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t ExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx ExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload PrivateTmp=true Restart=always RestartSec=5 StartLimitInterval=0 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF ``` 分发 systemd unit 文件： ``` cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" scp kube-nginx.service root@${node_ip}:/etc/systemd/system/ done ``` 启动 kube-nginx 服务： ``` cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-nginx && systemctl restart kube-nginx" done ``` #### 检查 kube-nginx 服务运行状态 ``` cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" ssh root@${node_ip} "systemctl status kube-nginx |grep 'Active:'" done ``` 确保状态为 active (running)，否则查看日志，确认原因： ``` journalctl -u kube-nginx ``` Last modification：August 5th, 2019 at 05:07 pm © 允许规范转载 Support 如果觉得我的文章对你有用 ×Close Appreciate the author Sweeping payments