一、K8s使用ingress发布服务
1、K8s发布服务方式
当我们将kubernetes的应用部署完之后,就需要对外发布服务的访问地址。
kubernetes 将服务发布到外部访问的方式主要有:
- LoadBlancer Service
- NodePort Service
- Ingress
LoadBlancer:Service LoadBlancer Service 是 kubernetes 深度结合云平台的一个组件;当使用 LoadBlancer Service 暴露服务时,实际上是通过向底层云平台申请创建一个负载均衡器来向外暴露服务;目前 GCE、AWS、阿里云等公有云都有针对LoadBlancer Service完整的解决方案。
NodePort Service:因为K8s 的rc具有副本控制能力, Pod如果出现意外情况会自动销毁并重建;因此Pod 的ip地址也会跟着变化。我们可以在service中定义nodeport,在每个节点上开起一个端口,然后转发到内部 Pod IP 上。这就是所谓的NodePort Service,实质上就是在每个 node 上暴露一个端口,然后将这个端口映射到某个具体的 service 来实现的。
虽然每个 node 的端口有很多(0~65535),但是由于安全性和易用性,实际上可用的端口范围为:30000-32767。如果在service指定的nodeport超过了这个范围,则会报错如下:
The Service "nginx-test" is invalid: spec.ports[0].nodePort: Invalid value: 38888: provided port is not in the valid range. The range of valid ports is 30000-32767
采用这种方式存在着诸多不遍:
- 首先端口数受限,使用此方式每一个node上都要开启相同的port
- 如果最终的应用访问要使用域名访问,还得在应用前端放一个nginx做转发。
又或者直接在k8s集群中使用deamonset方式部署nginx,将请求转发到集群内的pod上,这是个好想法,但每次新增或者修改服务,都要去修改nginx配置文件,然后平滑的重启nginx,无法做到服务的自动发现,也不是最终的解决方案。
Ingress Ingress:是在kubernetes 1.2版本才出现的,通过 Ingress 用户可以实现使用 nginx 等开源的反向代理负载均衡器实现对外暴露服务。使用 Ingress 时一般会有三个组件:
反向代理负载均衡器 反向代理负载均衡器通常使用nginx,部署方式可以选择 Replication Controller、Deployment、DaemonSet 等。
Ingress Controller Ingress Controller 通过连接api server,获取 service以及pod的信息,当得到这些变化信息后,Ingress Controller 再结合Ingress 生成配置,更新反向代理负载均衡器,达到服务发现的目的。
Ingress Ingress 简单理解就是个规则定义;其本质是根据rules规则来将对应的host访问请求转发到k8s后端的services。从而实现整体的服务发现和负载均衡。
定义ingress前,必须先部署ingress controller ,以实现为所有后端的service 提供一个统一的入口。在ingress-controller的rc文件中定义了一个默认后端。所以在部署ingress controller前要先启动默认后端的pod,否则启动ingress-controller会不成功。
2、通过ingress发布服务
下面我们来介绍在k8s 1.5.2集群环境中,通过配置ingress,发布前面配置好的dashboard和nginx服务。
1. 默认后端的yaml文件
# cat default-backend.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: default-http-backend
labels:
k8s-app: default-http-backend
namespace: default
spec:
replicas: 1
template:
metadata:
labels:
k8s-app: default-http-backend
spec:
terminationGracePeriodSeconds: 60
containers:
- name: default-http-backend
# Any image is permissable as long as:
# 1. It serves a 404 page at /
# 2. It serves 200 on a /healthz endpoint
image: docker.io/cdchen/defaultbackend:1.0
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
ports:
- containerPort: 8080
resources:
limits:
cpu: 10m
memory: 20Mi
requests:
cpu: 10m
memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
name: default-http-backend
namespace: default
labels:
k8s-app: default-http-backend
spec:
ports:
- port: 80
targetPort: 8080
selector:
k8s-app: default-http-backend
2. ingress ReplicationController的yaml文件
# cat nginx-ingress-controller.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: nginx-ingress-lb
labels:
name: nginx-ingress-lb
namespace: default
spec:
replicas: 1
template:
metadata:
labels:
name: nginx-ingress-lb
annotations:
prometheus.io/port: '10254'
prometheus.io/scrape: 'true'
spec:
terminationGracePeriodSeconds: 60
hostNetwork: true
containers:
- image: docker.io/cdchen/nginx-ingress-controller:0.9.0-beta.12
name: nginx-ingress-lb
readinessProbe:
httpGet:
path: /healthz
port: 10254
scheme: HTTP
livenessProbe:
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
timeoutSeconds: 1
ports:
- containerPort: 80
hostPort: 80
- containerPort: 443
hostPort: 443
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: KUBERNETES_MASTER
value: http://192.168.115.5:8080
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
- --apiserver-host=http://192.168.115.5:8080
3. Dashboard 和 nginx的ingress yaml文件
# cat k8s-dashboard.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: k8s-dashboard-ingress
namespace: default
spec:
rules:
- host: k8s.webui
http:
paths:
- path: /
backend:
serviceName: kubernetes-dashboard
servicePort: 80# cat k8s-nginx-test.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: k8s-nginx-test
namespace: default
spec:
rules:
- host: test.fjhb.cn
http:
paths:
- path: /
backend:
serviceName: nginx-test
servicePort: 80
4. 通过上述yaml文件创建对应的pod、rc及ingress
# kubectl get rc
# kubectl get pod
# cd ingress/
# kubectl create -f .
# kubectl get pod
# kubectl get ingress
# kubectl get rc
5. 访问测试
也可以通过查看nginx-ingress-lb pod的日志来验证。
二、Traefik简介
我们已经介绍了k8s ingress的实现原理和配置,本文将介绍替代方案traefik。在开始配置之前我们先回顾一下ingress发布k8s服务的实现方案,ingress方案需要使用下列的组件:
- 反向代理负载均衡器
- ingress control
- ingress
其中ingress control负责同apiserver进行通信,监测pod和service的变化,并更新反向代理负载均衡器的配置,同时让反向代理负载均衡器重载配置。
接下来我们来看看traefik,官网地址:Traefik Labs: Makes Networking Boring
Traefik (pronounced like traffic) is a modern HTTP reverse proxy and load balancer made to deploy microservices with ease. It supports several backends (Docker,Swarm mode,Kubernetes,Marathon, Consul,Etcd,Rancher,Amazon ECS, and a lot more) to manage its configuration automatically and dynamically.
Traefik本质上是一个http的方向代理和负载均衡,可以支持Docker, Swarm mode, Kubernetes, Marathon, Consul, Etcd, Rancher, Amazon ECS等后端服务,动态的管理这些服务的配置文件(我们可以理解为自动发现这些后端服务的配置变更,并重新加载服务的配置)。
下图是traefik的工作原理示意图:
Traefik的主要特点:
- Go语音编写、无需安装其他依赖包、速度快
- 支持Rest API、多后端类型支持
- 支持配置文件热加载,不需要重启app进程
- 支持Round Robin, rebalancer load-balancers等负载均衡策略
- 自带AngularJS Web UI图形化界面
- 支持https、自动更新https证书
- 支持websocket、HTTP/2, GRPC、高可用集群等
- 支持网络错误重试、后端自动熔断(当后端应用错误数过多的时候,可以自动熔断)
总而言之,在k8s集群中服务发布的方案选择,Traefik可用作为ingress的替代解决方案,traefik的特点足够说服我们将ingress替换成Traefik。
三、Kubernetes-Traefik2.0-Dnsmasq 实现外网映射
1、环境准备
- K8s:作为Traefik的搭建环境
- Docker:作为Dnsmasq的搭建环境,因为是dns服务器,所以单独拿出来做了下,也可以尝试本机直接搭建,或者也装在k8s里面
- Traefik:作为反向代理服务
- Dnsmasq:作为本地域名解析的dns服务器,这里也有很多其他的解决方案,也尝试过powerDns,但是由于本身这个东西就是可有可无,其实本地配个host就解决问题,但是咱们要高大上一些,就顺便搭建一个dns服务器,powerDns比较重搭建较复杂,考虑到产出比所以放弃
2、Dnsmasq搭建
准备3个配置文件分别为 dnsmasq.conf、dnsmasq.hosts、dnsmasq.resolv.conf 分别进行编辑:
# dnsmasq.conf文件用于配置DNS应用的配置, 指定dnsmasq.resolv.conf文件及dnsmasq.hosts文件的路径, dnsmasq.conf内容如下
cache-size=50000
dns-forward-max=1000
resolv-file=/etc/dnsmasq.resolv.conf
addn-hosts=/etc/dnsmasq.hosts
# 上面两个路径不要疑惑,这个写的是容器内部的目录,和咱们本地不一样没关系,我们是要映射这两个文件的
# dnsmasq.hosts文件, 就是我们经常修改的host了, 自定义指定ip和域名的映射关系, dnsmasq.hosts文件内容如下:
192.168.99.107 traefik.zcbb.dashboard
# 定义自己的域名啊...我这里定义的是自己的
# dnsmasq.resolv.conf文件, 由于DNS服务器是不知道baidu.com这些域名所对应的ip是什么, 所以要指定外网的DNS, 如果在dnsmasq.hosts文件找不到域名的映射关系, 则请求外网的DNS获取, dnsmasq.resolv.conf文件的内容如下:
nameserver 8.8.8.8
nameserver 114.114.114.114
接下来我们就可以启动dnsmasq了,开放53和18080端口 18080为web管理界面:
docker run \
--name dnsmasq \
-d \
-p 53:53/udp \
-p 18080:8080 \
-v /User/zcbb/dnsmasq.conf:/etc/dnsmasq.conf \
-v /User/zcbb/dnsmasq.hosts:/etc/dnsmasq.hosts \
-v /User/zcbb/dnsmasq.resolv.conf:/etc/dnsmasq.resolv.conf \
-v /etc/localtime:/etc/localtime \
--log-opt "max-size=100m" \
-e "HTTP_USER=admin" \
-e "HTTP_PASS=1" \
--restart always \
jpillora/dnsmasq
如图显示,则表示dns服务器启动成功,我们来编辑自己的dns服务器配置,把服务器ip放入/etc/resolv.conf,dns这块就配置完成了。
3、Traefik2.0 安装
trafik咱们选择的版本是2.0,与1.0出入较大,因为我入坑较晚,所以选了新的,目前各家公司可能还是选择1.x版本的较多,毕竟是稳定版本。
开始搭建:
# 首先我创建了一个namespace专门放traefik
kubectl create namespace traefik
#然后创建CRD资源
## IngressRoute
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ingressroutes.traefik.containo.us
spec:
scope: Namespaced
group: traefik.containo.us
version: v1alpha1
names:
kind: IngressRoute
plural: ingressroutes
singular: ingressroute
---
## IngressRouteTCP
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ingressroutetcps.traefik.containo.us
spec:
scope: Namespaced
group: traefik.containo.us
version: v1alpha1
names:
kind: IngressRouteTCP
plural: ingressroutetcps
singular: ingressroutetcp
---
## Middleware
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: middlewares.traefik.containo.us
spec:
scope: Namespaced
group: traefik.containo.us
version: v1alpha1
names:
kind: Middleware
plural: middlewares
singular: middleware
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: tlsoptions.traefik.containo.us
spec:
scope: Namespaced
group: traefik.containo.us
version: v1alpha1
names:
kind: TLSOption
plural: tlsoptions
singular: tlsoption
# 创建 RBAC 权限
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: traefik
name: traefik-ingress-controller
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
rules:
- apiGroups: [""]
resources: ["services","endpoints","secrets"]
verbs: ["get","list","watch"]
- apiGroups: ["extensions"]
resources: ["ingresses"]
verbs: ["get","list","watch"]
- apiGroups: ["extensions"]
resources: ["ingresses/status"]
verbs: ["update"]
- apiGroups: ["traefik.containo.us"]
resources: ["middlewares"]
verbs: ["get","list","watch"]
- apiGroups: ["traefik.containo.us"]
resources: ["ingressroutes"]
verbs: ["get","list","watch"]
- apiGroups: ["traefik.containo.us"]
resources: ["ingressroutetcps"]
verbs: ["get","list","watch"]
- apiGroups: ["traefik.containo.us"]
resources: ["tlsoptions"]
verbs: ["get","list","watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
namespace: traefik
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
name: traefik-ingress-controller
namespace: traefik
# 创建 Traefik 配置文件
kind: ConfigMap
apiVersion: v1
metadata:
name: traefik-config
data:
traefik.yaml: |-
serversTransport:
insecureSkipVerify: true
api:
insecure: true
dashboard: true
debug: true
metrics:
prometheus: ""
entryPoints:
web:
address: ":80"
websecure:
address: ":443"
providers:
kubernetesCRD: ""
log:
filePath: ""
level: error
format: json
accessLog:
filePath: ""
format: json
bufferingSize: 0
filters:
retryAttempts: true
minDuration: 20
fields:
defaultMode: keep
names:
ClientUsername: drop
headers:
defaultMode: keep
names:
User-Agent: redact
Authorization: drop
Content-Type: keep
# 为节点设置 Label 标签
kubectl label nodes minikube IngressProxy=true
# 部署 Traefik
apiVersion: v1
kind: Service
metadata:
name: traefik
spec:
ports:
- name: web
port: 80
- name: websecure
port: 443
- name: admin
port: 8080
selector:
app: traefik
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: traefik-ingress-controller
labels:
app: traefik
spec:
selector:
matchLabels:
app: traefik
template:
metadata:
name: traefik
labels:
app: traefik
spec:
serviceAccountName: traefik-ingress-controller
terminationGracePeriodSeconds: 1
containers:
- image: traefik:v2.0.5
name: traefik-ingress-lb
ports:
- name: web
containerPort: 80
hostPort: 80 #hostPort方式,将端口暴露到集群节点
- name: websecure
containerPort: 443
hostPort: 443 #hostPort方式,将端口暴露到集群节点
- name: admin
containerPort: 8080
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 1000m
memory: 1024Mi
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
args:
- --configfile=/config/traefik.yaml
volumeMounts:
- mountPath: "/config"
name: "config"
volumes:
- name: config
configMap:
name: traefik-config
tolerations: #设置容忍所有污点,防止节点被设置污点
- operator: "Exists"
nodeSelector: #设置node筛选器,在特定label的节点上启动
IngressProxy: "true"
# 配置第一个 ingressRoute
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: traefik-dashboard-route
namespace: traefik
spec:
entryPoints:
- web
routes:
- match: Host(`traefik.zcbb.dashboard`)
kind: Rule
services:
- name: traefik
port: 8080 # name也行
至此搭建traefik2.0完成,我们来看下dashboard,此时我们的dns已经可以解析咱们的域名了,访问 http://traefik.zcbb.dashboard,我们会看到如下界面:
我们可以顺便看一下,我们刚刚创建的ingressRoute,就是这条路由给我们映射了域名,提供我们从外部访问K8s内部的服务。
四、使用Traefik发布K8s内部服务
1、通过yaml文件用daemonset方式运行traefik
# mkdir traefik
# cd traefik
# docker pullk docker.io/traefik# cat traefik.ds.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: traefik-ingress-lb
namespace: default
labels:
k8s-app: traefik-ingress-lb
spec:
template:
metadata:
labels:
k8s-app: traefik-ingress-lb
name: traefik-ingress-lb
spec:
terminationGracePeriodSeconds: 60
hostNetwork: true
restartPolicy: Always
containers:
- image: docker.io/traefik
name: traefik-ingress-lb
resources:
limits:
cpu: 200m
memory: 30Mi
requests:
cpu: 100m
memory: 20Mi
ports:
- name: http
containerPort: 80
hostPort: 80
- name: admin
containerPort: 8081
args:
- --web
- --web.address=:8081
- --kubernetes
- --kubernetes.endpoint=http://192.168.115.5:8080
2、通过yaml文件创建webui的service、ingress
# cat ui.yaml
apiVersion: v1
kind: Service
metadata:
name: traefik-web-ui
namespace: default
spec:
selector:
k8s-app: traefik-ingress-lb
ports:
- name: web
port: 80
targetPort: 8081
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: traefik-web-ui
namespace: default
spec:
rules:
- host: traefik-ui.local
http:
paths:
- path: /
backend:
serviceName: traefik-web-ui
servicePort: web
3、通过yaml文件创建 kubernetes-dashboard和frontend的ingress
# cat traefik.ing.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: traefik-ingress
spec:
rules:
- host: k8s.webui
http:
paths:
- path: /
backend:
serviceName: kubernetes-dashboard
servicePort: 80
- host: k8s.frontend
http:
paths:
- path: /
backend:
serviceName: frontend
servicePort: 80
4、通过上述yaml配置文件创建pod和ingress
# cd kubernetes/traefik/
# kubectl get pod
# kubectl get svc
# kubectl create -f .
# kubectl get pod
# kubectl get svc
# kubectl get ingress
5、访问应用测试
修改测试机hosts文件,因为traefik采用demonset的方式运行,所以hosts记录指向k8s集群中任意的节点均可。
可以通过访问集群内任意节点的8081端口查看taefik的图形化界面。
五、Traefik-ingress配置HTTPS
实际生产环境中不仅仅只是http的转发访问,还有https的转发访问。
最简单的http访问traefik,访问过程参考见下:
client --- (via http) ---> traefik ---- (via http) ----> services
现在要实践的是更安全也更复杂的https访问traefik,有两种访问过程,参考见下:
1)后端service是普通http的 即client与traefik间采用https加密通信,但traefik与svc间则是明文的http通信:client --- (via https) ---> traefik ---- (via http) ----> services
2)后端service是https的 即client与traefik间采用https加密通信,但traefik与svc也是采用https通信:client --- (via https) ---> traefik ---- (via https) ----> services
下面我们来看看如何实现(伪)https,也就是上面说的第二种访问流程。
首先创建证书,想开启https,证书是少不了的。可以自己手动建一个证书,或者利用已经有的证书,这里我自己创建了一个ssl证书。
[root@k8smaster ~]# cd /opt/k8s/ssl
[root@k8smaster ssl]# ls
ssl.crt ssl.csr ssl.key
上面这个/opt/k8s/ssl目录是我创建的,路径可以随便只要和config文件里面的路径一致就行下面会说到。
下面开始配置证书:
[root@k8smaster ssl]# kubectl create secret generic traefik-cert --from-file=ssl.crt --from-file=ssl.key -n kube-system
secret "traefik-cert" created
创建一个configmap,保存traefix的配置。这里的traefix中配置了把所有http请求全部rewrite为https的规则,并配置相应的证书位置,同时我这里也创建了一个目录/opt/k8s/conf/。
[root@k8smaster conf]# cat traefik.toml
defaultEntryPoints = ["http","https"]
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[[entryPoints.https.tls.certificates]]
certFile = "/opt/k8s/ssl/ssl.crt"
keyFile = "/opt/k8s/ssl/ssl.key"
[root@k8smaster config]# kubectl create configmap traefik-conf --from-file=traefik.toml -n kube-system
configmap "traefik-conf" created
由于之前配置的是http现在要换成https所以需要更新下Traefik,这里主要是更新下关联创建的secret和configMap,并挂载相对应的主机目录。
安全起见操作之前先备份下(职场好习惯):
[root@k8smaster k8s]# cp traefik-deployment.yaml traefik-deployment.yaml.bk
[root@k8smaster k8s]# cat traefik-deployment.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: traefik-ingress-controller
namespace: kube-system
---
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
name: traefik-ingress-controller
namespace: kube-system
labels:
k8s-app: traefik-ingress-lb
spec:
selector:
matchLabels:
k8s-app: traefik-ingress-lb
template:
metadata:
labels:
k8s-app: traefik-ingress-lb
name: traefik-ingress-lb
spec:
serviceAccountName: traefik-ingress-controller
terminationGracePeriodSeconds: 60
hostNetwork: true
volumes:
- name: ssl
secret:
secretName: traefik-cert
- name: config
configMap:
name: traefik-conf
containers:
- image: traefik
name: traefik-ingress-lb
volumeMounts:
- mountPath: "/opt/k8s/ssl/"
name: "ssl"
- mountPath: "/opt/k8s/conf/"
name: "config"
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
- name: admin
containerPort: 8080
args:
- --configFile=/opt/k8s/conf/traefik.toml
- --api
- --kubernetes
- --logLevel=INFO
---
kind: Service
apiVersion: v1
metadata:
name: traefik-ingress-service
namespace: kube-system
spec:
selector:
k8s-app: traefik-ingress-lb
ports:
- protocol: TCP
port: 80
name: web
- protocol: TCP
port: 443
name: https
- protocol: TCP
port: 8080
name: admin
type: NodePort
[root@k8smaster k8s]#
[root@k8smaster k8s]# kubectl apply -f traefik-deployment.yaml
serviceaccount "traefik-ingress-controller" created
daemonset.extensions "traefik-ingress-controller" created
service "traefik-ingress-service" created
主要变化呢是更新了几个方面:
- kind: DaemonSet 官方默认是使用Deployment
- hostNetwork: true 开启Node Port端口转发
- volumeMounts 新增volumes挂载点
- ports 新增https443
- args 新增configfile
- Service层的443 ports
最后我们来测试下是否成功,这里我们可以登陆traefik-ui界面,可以看到原本http的访问,traefik会直接给我们重定向至https。