kubernetes 要求集群内各节点(包括 master 节点)能通过 Pod 网段互联互通。flannel 使用 vxlan 技术为各节点创建一个可以互通的 Pod 网络。
flaneel 第一次启动时,从 etcd 获取 Pod 网段信息,为本节点分配一个未使用的地址段,然后创建 flannedl.1
(也可能是其它名称,如 flannel1 等) 接口。
flannel 将分配的 Pod 网段信息写入 /run/flannel/docker
文件,docker 后续使用这个文件中的环境变量设置 docker0
网桥。
下载和分发 flanneld 二进制文件
到 https://github.com/coreos/flannel/releases 页面下载最新版本的发布包:
cd /opt/k8s/work mkdir flannel wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz tar -xzvf flannel-v0.10.0-linux-amd64.tar.gz -C flannel
分发 flanneld 二进制文件到集群所有节点:
cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" scp flannel/{flanneld,mk-docker-opts.sh} root@${node_ip}:/opt/k8s/bin/ ssh root@${node_ip} "chmod +x /opt/k8s/bin/*" done
创建 flannel 证书和私钥
flannel 从 etcd 集群存取网段分配信息,而 etcd 集群启用了双向 x509 证书认证,所以需要为 flanneld 生成证书和私钥。
创建证书签名请求:
cd /opt/k8s/work cat > flanneld-csr.json <<EOF { "CN": "flanneld", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "4Paradigm" } ] } EOF
- 该证书只会被 kubectl 当做 client 证书使用,所以 hosts 字段为空;
生成证书和私钥:
cfssl gencert -ca=/opt/k8s/work/ca.pem \ -ca-key=/opt/k8s/work/ca-key.pem \ -config=/opt/k8s/work/ca-config.json \ -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld ls flanneld*pem
将生成的证书和私钥分发到所有节点(master 和 worker):
cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" ssh root@${node_ip} "mkdir -p /etc/flanneld/cert" scp flanneld*.pem root@${node_ip}:/etc/flanneld/cert done
向 etcd 写入集群 Pod 网段信息
注意:本步骤只需执行一次。
cd /opt/k8s/work source /opt/k8s/bin/environment.sh etcdctl \ --endpoints=${ETCD_ENDPOINTS} \ --ca-file=/opt/k8s/work/ca.pem \ --cert-file=/opt/k8s/work/flanneld.pem \ --key-file=/opt/k8s/work/flanneld-key.pem \ set ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'", "SubnetLen": 21, "Backend": {"Type": "vxlan"}}'
- flanneld 当前版本 (v0.10.0) 不支持 etcd v3,故使用 etcd v2 API 写入配置 key 和网段数据;
- SubnetLen 设置为 21,而 ${CLUSTER_CIDR} 是 /16 段地址,故支持 32 个节点,每个节点最多 2048 个 Pods;
创建 flanneld 的 systemd unit 文件
cd /opt/k8s/work source /opt/k8s/bin/environment.sh export IFACE=eno1 cat > flanneld.service << EOF [Unit] Description=Flanneld overlay address etcd agent After=network.target After=network-online.target Wants=network-online.target After=etcd.service Before=docker.service [Service] Type=notify ExecStart=/opt/k8s/bin/flanneld \\ -etcd-cafile=/etc/kubernetes/cert/ca.pem \\ -etcd-certfile=/etc/flanneld/cert/flanneld.pem \\ -etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \\ -etcd-endpoints=${ETCD_ENDPOINTS} \\ -etcd-prefix=${FLANNEL_ETCD_PREFIX} \\ -iface=${IFACE} ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker Restart=on-failure [Install] WantedBy=multi-user.target RequiredBy=docker.service EOF
mk-docker-opts.sh
脚本将分配给 flanneld 的 Pod 子网网段信息写入/run/flannel/docker
文件,后续 docker 启动时使用这个文件中的环境变量配置 docker0 网桥;- flanneld 使用系统缺省路由所在的接口与其它节点通信,对于有多个网络接口(如内网和公网)的节点,可以用
-iface
参数指定通信接口,如上面的 eno1 接口; - flanneld 运行时需要 root 权限;
完整 unit 见 flanneld.service
分发 flanneld systemd unit 文件到所有节点
source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" scp flanneld.service root@${node_ip}:/etc/systemd/system/ done
启动 flanneld 服务
source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" ssh root@${node_ip} "systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld" done
检查启动结果
source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" ssh root@${node_ip} "systemctl status flanneld|grep Active" done
确保状态为 active (running)
,否则查看日志,确认原因:
$ journalctl -u flanneld
检查分配给各 flanneld 的 Pod 网段信息
查看集群 Pod 网段(/16):
source /opt/k8s/bin/environment.sh etcdctl \ --endpoints=${ETCD_ENDPOINTS} \ --ca-file=/etc/kubernetes/cert/ca.pem \ --cert-file=/etc/flanneld/cert/flanneld.pem \ --key-file=/etc/flanneld/cert/flanneld-key.pem \ get ${FLANNEL_ETCD_PREFIX}/config
输出:
{"Network":"172.30.0.0/16", "SubnetLen": 21, "Backend": {"Type": "vxlan"}}
查看已分配的 Pod 子网段列表(/24):
source /opt/k8s/bin/environment.sh etcdctl \ --endpoints=${ETCD_ENDPOINTS} \ --ca-file=/etc/kubernetes/cert/ca.pem \ --cert-file=/etc/flanneld/cert/flanneld.pem \ --key-file=/etc/flanneld/cert/flanneld-key.pem \ ls ${FLANNEL_ETCD_PREFIX}/subnets
输出:
/kubernetes/network/subnets/172.30.96.0-21 /kubernetes/network/subnets/172.30.200.0-21 /kubernetes/network/subnets/172.30.112.0-21
查看某一 Pod 网段对应的节点 IP 和 flannel 接口地址:
source /opt/k8s/bin/environment.sh etcdctl \ --endpoints=${ETCD_ENDPOINTS} \ --ca-file=/etc/kubernetes/cert/ca.pem \ --cert-file=/etc/flanneld/cert/flanneld.pem \ --key-file=/etc/flanneld/cert/flanneld-key.pem \ get ${FLANNEL_ETCD_PREFIX}/subnets/172.30.96.0-21
输出:
{"PublicIP":"172.27.128.149","BackendType":"vxlan","BackendData":{"VtepMAC":"1e:38:19:e2:7e:3a"}}
验证各节点能通过 Pod 网段互通
在各节点上部署 flannel 后,检查是否创建了 flannel 接口(名称可能为 flannel0、flannel.0、flannel.1 等):
source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" ssh ${node_ip} "/usr/sbin/ip addr show flannel.1|grep -w inet" done
输出:
>>> 172.27.128.150 inet 172.30.112.0/32 scope global flannel.1 >>> 172.27.128.149 inet 172.30.96.0/32 scope global flannel.1 >>> 172.27.128.148 inet 172.30.200.0/32 scope global flannel.1
在各节点上 ping 所有 flannel 接口 IP,确保能通:
source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" ssh ${node_ip} "ping -c 1 172.30.112.0" ssh ${node_ip} "ping -c 1 172.30.96.0" ssh ${node_ip} "ping -c 1 172.30.200.0" done