前言
Kubernetes利用iptables达成以下两个目的:
1)对外暴露POD和服务
2)简单的负载均衡
在kubernetes worker node的iptables的NAT表的prerouting和output链表会出现如下规则作为kubernetes相关的数据包操作的入口:
-A PREROUTING -m comment --comment "kube hostport portals" -m addrtype --dst-type LOCAL -j KUBE-HOSTPORTS
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m comment --comment "handle ClusterIPs; NOTE: this must be before the NodePort rules" -j KUBE-PORTALS-CONTAINER
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A PREROUTING -m addrtype --dst-type LOCAL -m comment --comment "handle service NodePorts; NOTE: this must be the last rule in the chain" -j KUBE-NODEPORT-CONTAINER
-A OUTPUT -m comment --comment "kube hostport portals" -m addrtype --dst-type LOCAL -j KUBE-HOSTPORTS
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m comment --comment "handle ClusterIPs; NOTE: this must be before the NodePort rules" -j KUBE-PORTALS-HOST
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m addrtype --dst-type LOCAL -m comment --comment "handle service NodePorts; NOTE: this must be the last rule in the chain" -j KUBE-NODEPORT-HOST
PREROUTING链表用来处理外部进来的数据包
规则1 用来向使用hostport的POD转发数据包(用于IP Tables模式)
规则2 用来向kubernetes服务转发数据包(用于IPTables模式)
规则3 用来处理容器内向cluster service虚IP发出的请求(用于kube-proxy模式)
规则4 用来处理容器向nodeport发出的请求(用于kube-proxy模式)
OUTPUT链表用来处理发向外部的数据包
规则1 用来处理主机向host port发出的请求(用于IP Tables模式)
规则2 用来处理主机向kubernetes service发出的请求(用于IP Tables模式)
规则3 用来处理主机向cluster service 虚IP发出的请求(用于kube-proxy模式)
规则4 用来处理主机向nodeport发出的请求(用于kube-proxy模式)
kubernetes各种类型的服务对外暴露的顺序依次是hostport、cluster service、node external、 loadbalancer service 和nodeport service
-A KUBE-SERVICES -d cluster_IP/32 -p tcp -m comment --comment "namespace/pod_name:port_name cluster IP" -m tcp --dport 80 -j KUBE-SVC-SSSSSS
-A KUBE-SERVICES -d node_external_IP/32 -p tcp -m comment --comment "namespace/pod_name:port_name external IP" -m tcp --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d node_external_IP/32 -p tcp -m comment --comment "namespace/pod_name:port_name external IP" -m tcp --dport 80 -m physdev ! --physdev-is-in -m addrtype ! --src-type LOCAL -j KUBE-SVC-SSSSSS
-A KUBE-SERVICES -d node_external_IP/32 -p tcp -m comment --comment "namespace/pod_name:port_name external IP" -m tcp --dport 80 -m addrtype --dst-type LOCAL -j KUBE-SVC-SSSSSS
-A KUBE-SERVICES -d loadbalancer_IP/32 -p tcp -m comment --comment "namespace/pod_name:port_name loadbalancer IP" -m tcp --dport 80 -j KUBE-FW-SSSSSS
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
工作原理
1) cluster service 的入口是这个规则“-A[PREROUTING|OUTPUT]-m comment --comment 'kubernetes service portals' -j KUBE_SERVICES”, 跳到KUBE-SERVICE chain
2)这个KUBE-SERVICE chain 由一系列满足如下规则的rule组成:
满足访问某个cluster_VIP和port的请求将会被倒入到rule KUBE-SVC-XXXXX
3)KUBE-SVC-XXXXXX的组成如下:
-A KUBE-SVC-XXXXXX -m comment --comment "namespace/pod_name:port_name" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-AAAAAA
-A KUBE-SVC-XXXXXX -m comment --comment "namespace/pod_name:port_name" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-BBBBBB
-A KUBE-SVC-XXXXXX -m comment --comment "namespace/pod_name:port_name" -j KUBE-SEP-CCCCCC
会按照概率对所有后端的POD进行选择转发
4)KUBE-SEP-CCCCC的组成如下:
-A KUBE-SEP-CCCCCC -s POD_IP/32 -m comment --comment "namespace/pod_name:port_name" -j KUBE-MARK-MASQ
-A KUBE-SEP-CCCCCC -p tcp -m comment --comment "namespace/pod_name:port_name" -m tcp -j DNAT --to-destination POD_IP:PORT
需要进行一次DNAT, 把数据包定位到选定的POD, 然后经由路由进入远端或者本地的POD
External Load balancer service如何使用iptables对外暴露POD服务
1. 经由external load balancer转发的外部请求会带有外部IP地址,这个地址会匹配如下规则:
-A KUBE-SERVICES -d loadbalancer_IP/32 -p tcp -m comment --comment "namespace/pod_name:port_name loadbalancer IP" -m tcp --dport 80 -j KUBE-FW-SSSSSS
2. KUBE-FW-SSSSSS组成如下:
-A KUBE-FW-SSSSSS -m comment --comment "namespace/pod_name:port_name loadbalancer IP" -j KUBE-XLB-KKKKKK
-A KUBE-FW-SSSSSS -m comment --comment "namespace/pod_name:port_name loadbalancer IP" -j KUBE-MARK-DROP
3.KUBE-XLB-KKKKKK由一系列概率选择规则组成:
-A KUBE-XLB-KKKKKK -m comment --comment "Balancing rule 0 for namespace/pod_name:port_name" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-AAAAAA
-A KUBE-XLB-KKKKKK -m comment --comment "Balancing rule 1 for namespace/pod_name:port_name" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-BBBBBB
-A KUBE-XLB-KKKKKK -m comment --comment "Balancing rule 2 for namespace/pod_name:port_name" -j KUBE-SEP-CCCCCC
将数据包导到具体的POD相关iptables规则
4. KUBE-SEP-CCCCC的组成如下:
-A KUBE-SEP-CCCCCC -s POD_IP/32 -m comment --comment "namespace/pod_name:port_name" -j KUBE-MARK-MASQ
-A KUBE-SEP-CCCCCC -p tcp -m comment --comment "namespace/pod_name:port_name" -m tcp -j DNAT --to-destination POD_IP:PORT
需要进行一次DNAT,把数据包定位到选定POD,然后经由路由进入远端或者本地的POD
nodePort Service如何使用iptables对外暴露POD服务
1. KUBE-SERVICE chain的最末端将跳转到nodeport service对应的chain KUBE-NODEPORTS
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
2. KUBE-NODEPORTS里面包含一系列不同的nodeport service对应的规则
例如:
-A KUBE-NODEPORTS -p tcp -m comment --comment "namespace/pod_name:port_name" -m tcp --dport 80 -j KUBE-XLB-KKKKKK
最后跳转到为外部访问生成的load balance规则。
3.KUBE-XLB-KKKKKK由一系列概率选择规则组成:
-A KUBE-XLB-KKKKKK -m comment --comment "Balancing rule 0 for namespace/pod_name:port_name" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-AAAAAA
-A KUBE-XLB-KKKKKK -m comment --comment "Balancing rule 1 for namespace/pod_name:port_name" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-BBBBBB
-A KUBE-XLB-KKKKKK -m comment --comment "Balancing rule 2 for namespace/pod_name:port_name" -j KUBE-SEP-CCCCCC
将数据包导到具体的POD相关iptables规则
4. KUBE-SEP-CCCCC的组成如下:
-A KUBE-SEP-CCCCCC -s POD_IP/32 -m comment --comment "namespace/pod_name:port_name" -j KUBE-MARK-MASQ
-A KUBE-SEP-CCCCCC -p tcp -m comment --comment "namespace/pod_name:port_name" -m tcp -j DNAT --to-destination POD_IP:PORT
需要进行一次DNAT,把数据包定位到选定POD,然后经由路由进入远端或者本地的POD