> 原教程来自 [github/opsnull](https://github.com/opsnull/follow-me-install-kubernetes-cluster), 现在此基础上记录自己搭建遇到的问题 基于 nginx 代理的 kube-apiserver 高可用方案 - 控制节点的 kube-controller-manager、kube-scheduler 是多实例部署,所以只要有一个实例正常,就可以保证高可用; - 集群内的 Pod 使用 K8S 服务域名 kubernetes 访问 kube-apiserver, kube-dns 会自动解析出多个 kube-apiserver 节点的 IP,所以也是高可用的; - 在每个节点起一个 nginx 进程,后端对接多个 apiserver 实例,nginx 对它们做健康检查和负载均衡; - kubelet、kube-proxy、controller-manager、scheduler 通过本地的 nginx(监听 127.0.0.1)访问 kube-apiserver,从而实现 kube-apiserver 的高可用; #### 下载和编译 nginx 下载源码: ``` cd /opt/k8s/work wget http://nginx.org/download/nginx-1.15.3.tar.gz tar -xzvf nginx-1.15.3.tar.gz ``` 配置编译参数: ``` cd /opt/k8s/work/nginx-1.15.3 mkdir nginx-prefix ./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module ``` - --with-stream:开启 4 层透明转发(TCP Proxy)功能; - --without-xxx:关闭所有其他功能,这样生成的动态链接二进制程序依赖最小; 输出: ``` Configuration summary + PCRE library is not used + OpenSSL library is not used + zlib library is not used nginx path prefix: "/opt/k8s/work/nginx-1.15.3/nginx-prefix" nginx binary file: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx" nginx modules path: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/modules" nginx configuration prefix: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/conf" nginx configuration file: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/conf/nginx.conf" nginx pid file: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/logs/nginx.pid" nginx error log file: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/logs/error.log" nginx http access log file: "/opt/k8s/work/nginx-1.15.3/nginx-prefix/logs/access.log" nginx http client request body temporary files: "client_body_temp" nginx http proxy temporary files: "proxy_temp" ``` 编译和安装: ``` cd /opt/k8s/work/nginx-1.15.3 make && make install ``` #### 验证编译的 nginx ``` cd /opt/k8s/work/nginx-1.15.3 ./nginx-prefix/sbin/nginx -v ``` 输出: ``` [root@ _14_ /opt/k8s/work/nginx-1.15.3]# ./nginx-prefix/sbin/nginx -v nginx version: nginx/1.15.3 ``` 查看 nginx 动态链接的库: ``` [root@ _15_ /opt/k8s/work/nginx-1.15.3]# ldd ./nginx-prefix/sbin/nginx ``` 输出 ``` linux-vdso.so.1 => (0x00007ffe3ed70000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fe8876fe000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe8874e2000) libc.so.6 => /lib64/libc.so.6 (0x00007fe887121000) /lib64/ld-linux-x86-64.so.2 (0x00007fe887902000) ``` 由于只开启了 4 层透明转发功能,所以除了依赖 libc 等操作系统核心 lib 库外,没有对其它 lib 的依赖(如 libz、libssl 等),这样可以方便部署到各版本操作系统中; #### 安装和部署 nginx 创建目录结构: ``` cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin} done ``` 拷贝二进制程序: ``` cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" ssh root@${node_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}" scp /opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx root@${node_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx ssh root@${node_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*" done ``` - 重命名二进制文件为 kube-nginx 配置 nginx,开启 4 层透明转发功能 ``` cd /opt/k8s/work cat > kube-nginx.conf <>> ${node_ip}" scp kube-nginx.conf root@${node_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.conf done ``` #### 配置 systemd unit 文件,启动服务 配置 kube-nginx systemd unit 文件: ``` cd /opt/k8s/work cat > kube-nginx.service <>> ${node_ip}" scp kube-nginx.service root@${node_ip}:/etc/systemd/system/ done ``` 启动 kube-nginx 服务: ``` cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-nginx && systemctl restart kube-nginx" done ``` #### 检查 kube-nginx 服务运行状态 ``` cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" ssh root@${node_ip} "systemctl status kube-nginx |grep 'Active:'" done ``` 确保状态为 active (running),否则查看日志,确认原因: ``` journalctl -u kube-nginx ``` Last modification:August 5th, 2019 at 05:07 pm © 允许规范转载 Support 如果觉得我的文章对你有用 ×Close Appreciate the author Sweeping payments