Kubernete-- 利用kubeadm 搭建一个kubernate集群
目标
- 利用 kubeadm 搭建一个四节点的k8s测试集群
- 利用harbor搭建一个单节点的私有镜像仓库
- k8s集群与私有镜像仓库整合
- 部署dashboard
- 部署Heapster 监控与统计
前期准备
准备以下4个节点,一个为k8s的master节点,2个为node节点,最后一个作为私有仓库镜像,系统为centos7.2:
hostname | ip | 说明 |
---|---|---|
k8s.master | 192.168.2.130 | k8s master节点 |
k8s.node1 | 192.168.2.131 | k8s node节点 |
k8s.node2 | 192.168.2.132 | k8s node节点 |
k8s.harbor | 192.168.2.139 | k8s 仓库镜像节点 |
K8S 安装
1. 所有节点的操作
更新yum
sudo yum update -y
执行以下命令关闭防火墙
systemctl stop firewalld && systemctl disable firewalld
setenforce 0
将SELINUX的值改成disabled
vim /etc/selinux/config
SELINUX=disabled
为了使上面的改动生效,需要重启CentOS;
安装docker 官方文档:https://docs.docker.com/engine/installation/linux/docker-ce/centos/
#1.配置仓库
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
#2.可以选择是否开启edge和test仓库
sudo yum-config-manager --enable docker-ce-edge
sudo yum-config-manager --enable docker-ce-test
sudo yum-config-manager --disable docker-ce-edge
sudo yum-config-manager --disable docker-ce-test
#3.安装docker-ce
sudo yum install docker-ce #由于repo中默认只开启stable仓库,故这里安装的是最新稳定版17.09
#4.可以查看所有仓库中所有docker版本,并选择特定版本安装
yum list docker-ce --showduplicates | sort -r
docker-ce.x86_64 17.09.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.09.0.ce-1.el7.centos @docker-ce-stable
docker-ce.x86_64 17.06.2.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.06.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.06.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.2.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.0.ce-1.el7.centos docker-ce-stable
sudo yum install <FQPN> 例如:sudo yum install docker-ce-17.09.0.ce
#5.启动并加入开机启动
sudo systemctl start docker
sudo systemctl enable docker
#6.关闭docker-daemon
sudo systemctl stop docker
sudo systemctl disable docker
设置系统参数
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
关闭swap
swapoff -a
临时关闭,重启后失效
修改 /etc/fstab 文件,注释掉 SWAP 的自动挂载,使用free -m确认swap已经关闭。 > swappiness参数调整,修改/etc/sysctl.d/k8s.conf添加下面一行:
vm.swappiness=0 执行sysctl -p /etc/sysctl.d/k8s.conf使修改生效。
如果服务器可以翻墙,可以直接通过yum命令安装,内容如下: vim /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
安装kubernetes相关的应用
yum update
yum install -y kubelet kubeadm kubectl
如果不能翻墙,只能先下载下来,然后安装,需要安装的rpm包url地址可以在这个网页中找到:
curl https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/primary.xml
1.9.2百度网盘下载地址
https://pan.baidu.com/s/1i6Qbo4d
我们这里只需要安装四个rpm包,kubectl,kubeadm, kubelet以及kubernetes-cni,可以直接搜索上面的网页然后找到合适版本的rpm包。我们这里安装最新版本1.9.3,对应的地址如下:
kubeadm-1.9.2-0.x86_64.rpm
kubelet-1.9.3-0.x86_64.rpm
kubernetes-cni-0.6.0-0.x86_64.rpm
kubectl-1.9.2-0.x86_64.rpm
socat-1.7.3.2-2.el7.x86_64.rpm
yum install -y *.rpm
systemctl enable kubelet && systemctl start kubelet
journalctl -xeu kubelet查看错误信息
修改kubernetes的网络配置: 修改文件/etc/systemd/system/kubelet.service.d/10-kubeadm.conf,找到下面这一行:
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
把systemd改成cgroupfs;
然后执行以下命令重启kubelet:
systemctl restart kubelet
systemctl status kubelet
接下来开始基于Kubeadm 创建k8s集群,不过在开始之前,我们先准备下需要用到的镜像,因为kubeadm创建的k8s集群中的kub-api, kube-scheduler, kube-proxy, kube-controller-manager,etcd等服务都是直接拉取镜像跑在k8s集群中,为了避免安装过程中下载镜像浪费太多时间,这里先把镜像下载好。 我们直接用1.9.2,如果服务器可以翻墙,直接拉取镜像:
#必须镜像
gcr.io/google_containers/kube-apiserver-amd64:v1.9.2
gcr.io/google_containers/kube-proxy-amd64:v1.9.2
gcr.io/google_containers/kube-controller-manager-amd64:v1.9.2
gcr.io/google_containers/kube-scheduler-amd64:v1.9.2
gcr.io/google_containers/etcd-amd64:3.1.11
gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.7
gcr.io/google_containers/k8s-dns-kube-dns-amd641.14.7
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7
gcr.io/google_containers/pause-amd64:3.0
#calico
quay.io/calico/node:v2.6.7
quay.io/calico/kube-controllers:v1.0.3
quay.io/coreos/etcd:v3.1.10
#flannel
quay.io/coreos/flannel:v0.9.1-amd64
#Dashboard
k8s.gcr.io/kubernetes-dashboard-amd64
导入镜像
### 导入镜像 ###
[root@k8s-master images]# docker load -i cni.tar
.
.
.
[root@k8s-master images]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/calico/node v2.6.7 7c694b9cac81 8 days ago 281.6 MB
gcr.io/google_containers/kube-controller-manager-amd64 v1.9.2 769d889083b6 3 weeks ago 137.8 MB
gcr.io/google_containers/kube-proxy-amd64 v1.9.2 e6754bb0a529 3 weeks ago 109.1 MB
gcr.io/google_containers/kube-apiserver-amd64 v1.9.2 7109112be2c7 3 weeks ago 210.4 MB
gcr.io/google_containers/kube-scheduler-amd64 v1.9.2 2bf081517538 3 weeks ago 62.71 MB
quay.io/calico/kube-controllers v1.0.3 34aebe64326d 3 weeks ago 52.25 MB
k8s.gcr.io/kubernetes-dashboard-amd64 v1.8.2 c87ea0497294 3 weeks ago 102.3 MB
quay.io/calico/cni v1.11.2 6f0a76fc7dd2 7 weeks ago 70.78 MB
gcr.io/google_containers/etcd-amd64 3.1.11 59d36f27cceb 9 weeks ago 193.9 MB
quay.io/coreos/flannel v0.9.1-amd64 2b736d06ca4c 12 weeks ago 51.31 MB
gcr.io/google_containers/k8s-dns-sidecar-amd64 1.14.7 db76ee297b85 3 months ago 42.03 MB
gcr.io/google_containers/k8s-dns-kube-dns-amd64 1.14.7 5d049a8c4eec 3 months ago 50.27 MB
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64 1.14.7 5feec37454f4 3 months ago 40.95 MB
quay.io/coreos/etcd v3.1.10 47bb9dd99916 6 months ago 34.56 MB
gcr.io/google_containers/pause-amd64 3.0 99e59f495ffa 21 months ago 746.9 kB
在各个节点上运行ip_vs 因为可能遇到
[root@k8s-master /etc/modprobe.d]$ docker logs 71a413c7b015
I0307 03:32:57.394549 1 feature_gate.go:184] feature gates: map[]
time="2018-03-07T03:32:57Z" level=warning msg="Running modprobe ip_vs failed with message: `modprobe: ERROR: could not insert 'ip_vs': Exec format error\ninsmod /lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/net/netfilter/ipvs/ip_vs.ko.xz`, error: exit status 1"
time="2018-03-07T03:32:57Z" level=error msg="Could not get ipvs family information from the kernel. It is possible that ipvs is not enabled in your kernel. Native loadbalancing will not work until this is fixed."
在kube-proxy 运行之前先运行ip_vs
[root@k8s-master ~]$ modprobe -r ip_vs
[root@k8s-master ~]$ lsmod|grep ip_vs
[root@k8s-master ~]$ modprobe ip_vs
[root@k8s-master ~]$ lsmod|grep ip_vs
ip_vs 141092 0
nf_conntrack 133387 9 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6
libcrc32c 12644 3 ip_vs,nf_nat,nf_conntrack
[root@k8s-master ~]$ modprobe -r ip_vs
[root@k8s-master ~]$ insmod /lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/net/netfilter/ipvs/ip_vs.ko.xz
[root@k8s-master ~]$ lsmod|grep ip_vs
ip_vs 141092 0
nf_conntrack 133387 9 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6
libcrc32c 12644 3 ip_vs,nf_nat,nf_conntrack
[root@k8s-master ~]$
2. master节点安装
使用kubeadm初始化master节点
kubeadm init --kubernetes-version=v1.9.2 --pod-network-cidr=172.16.0.0/16 --apiserver-advertise-address=192.168.2.130
注意命令执行后的最后一句kubeadm join --token d8b040.0aa73933666c2865 192.168.2.130:6443 --discovery-token-ca-cert-hash sha256:c24a3a8404036a9e96e30dcada132eceb35e5db3e036c9e4ea8809cb9c623531,在node加入的时候我们就是用这一句来将node加入到集群的
kubeadm会自动检查当前环境是否有上次命令执行的"残留"。如果有,必须清理后再行执行init。我们可以通过"kubeadm reset"来清理环境,以备重来。
创建kube目录,添加kubectl配置
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
检查k8s组件状态
kubectl get cs
添加Calico(网络组件)
注意修改CALICO_IPV4POOL_CIDR参数
wget https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml
kubectl apply -f calico.yaml
2. node节点安装
kubeadm join --token d8b040.0aa73933666c2865 192.168.2.130:6443 --discovery-token-ca-cert-hash sha256:c24a3a8404036a9e96e30dcada132eceb35e5db3e036c9e4ea8809cb9c623531
在master上查看
kubectl get nodes
kubectl get pods --all-namespaces
kubectl describe node ******
节点重置/移除节点
### 驱离k8s-node-1节点上的pod ###
[root@k8s-master ~]# kubectl drain k8s.node1 --delete-local-data --force --ignore-daemonsets
### 删除节点 ###
[root@k8s-master ~]# kubectl delete node k8s.node1
### 重置节点 ###
[root@k8s-node-1 ~]# kubeadm reset
token过期后重新生成token
#重新生成新的token
[root@k8s-master ~]# kubeadm token create
[root@k8s-master ~]# kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
aa78f6.8b4cafc8ed26c34f 23h 2018-03-26T16:36:29+08:00 authentication,signing <none> system:bootstrappers:kubeadm:default-node-token
#获取ca证书sha256编码hash值
[root@k8s-master ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
0fd95a9bc67a7bf0ef42da968a0d55d92e52898ec37c971bd77ee501d845b538
#节点加入集群
kubeadm join --token aa78f6.8b4cafc8ed26c34f 192.168.2.130:6443 --discovery-token-ca-cert-hash sha256:0fd95a9bc67a7bf0ef42da968a0d55d92e52898ec37c971bd77ee501d845b538 --skip-preflight-checks
--skip-preflight-checks,可以防止每次初始化都去检查配置文件,否则可能在多次init后报错[etcd在使用、kubelet在使
3. 部署Dashboard 在master节点上执行:
wget https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
kubectl apply -f kubernetes-dashboard.yaml
默认配置部署的dashboard账号kubernetes-dashboard绑定的角色为kubernetes-dashboard-minimal,我们需要为其配置一个权限更高的用户 Dashboard的服务方式已经被修改为nodeport
vi kubernetes-dashboard-rbac-admin.yml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-admin
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard-admin
labels:
k8s-app: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard-admin
namespace: kube-system
[root@k8s-master yml]# kubectl apply -f kubernetes-dashboard-rbac-admin.yml
我们使用自己创建的kubernetes-dashboard-admin用户登录dashboard,并且使用token方式登录
### 找到kubernetes-dashboard-admin的secret记录
[root@k8s-master yml]# kubectl get secret --all-namespaces | grep kubernetes-dashboard-admin
kube-system kubernetes-dashboard-admin-token-2w8mg kubernetes.io/service-account-token 3 22h
## 查看secret的token值
[root@master images]# kubectl get secret --all-namespaces | grep kubernetes-dashboard-admin
kube-system kubernetes-dashboard-admin-token-2w8mg kubernetes.io/service-account-token 3 18h
[root@master images]# kubectl describe secret kubernetes-dashboard-admin-token-2w8mg -n kube-system
Name: kubernetes-dashboard-admin-token-2w8mg
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name=kubernetes-dashboard-admin
kubernetes.io/service-account.uid=b89b863f-1d24-11e8-9bec-080027fdd465
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1025 bytes
namespace: 11 bytes
token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbi10b2tlbi0ydzhtZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImI4OWI4NjNmLTFkMjQtMTFlOC05YmVjLTA4MDAyN2ZkZDQ2NSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiJ9.VNxhEYW_QXUMJC-M4H4Mbe_ziDz8g-SbmYtDzCGAN-n6Cpbu0xsle9ZN-uAzMaZIE35aPGhs3AncnnuhBNzdXp3PDYtZCuepaskLQXc73hcXYoRTmNjHM2y23oQQliatBl_ICsujiWI7cbmkvZKU0YVmicn3TYgHKVODFxexvxN5kS9Pirv-vT6g03SnvFCktAteV2nO6pQele0Qyt6UGPZX1lnJDrmAGr3_jzo9YGSDdcZePchiJnBOVh4P0ufRqPylNij5-Upxi3uCU10sOuySeaUuPMerHffW0rafrRPeD_6cyB0cAqawj3CTwFl1IbR4-UOsTZlId1LkCoaCCg
查看dashboard的端口,并使用上面的token登录
kubectl get svc --all-namespaces
kubectl describe svc kubernetes-dashboard-admin
4. 部署Heapster 监控与统计
wget https://github.com/kubernetes/heapster/archive/v1.3.0.tar.gz
tar -zxvf v1.3.0.tar.gz
cd heapster-1.3.0/deploy/kube-config/influxdb
kubectl create -f ./*
5. k8s 添加私有镜像 利用kubectl创建docker-registry的secret
kubectl create secret docker-registry myregistrykey --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL
kubectl get secret --all-namespaces #查看创建的secret