k8s笔记9--升级kubeadm部署的集群
1 介绍
近年来 k8s 社区非常活跃,版本迭代很快,因此根据业务需要升级k8s集群是一个非常重要的技能。本文介绍如何将k8s从1.19.x 升级 到1.20.x 版本。
k8s的升级流程主要分为如下3个部分:
- 升级主控制面节点
核心命令 kubeadm upgrade apply
1)检查集群处于可升级状态;
2)强制执行版本倾斜(version skew )策略;
3)确保控制面镜像可用且能pull到该机器;
4)若组件配置需要升级,则生成对应的替换文件;
5)升级控制面板组件(若失败则回滚);
6)应用新的 kube-dns 和 kube-proxy manifests ,确保创建所有必需的RBAC规则;
7)为API server生成新证书和key文件, 如果他们在180天内过期则还要备份旧文件; - 升级其它控制面节点(本文只有1个主节点,因此省略第二步);
核心命令 kubeadm upgrade node
1)从集群拉取 kubeadm ClusterConfiguration;
2)选择性地备份 kube-apiserver certificate;
3)为控制面板组件升级静态 Pod manifests;
4)为当前节点升级 kubelet configuration; - 升级工作节点
核心命令 kubeadm upgrade node
1)从集群拉取 kubeadm ClusterConfiguration;
2) 为当前节点升级 kubelet configuration;
2 集群升级
2.1 前期准备
- 备份 etcd
备份etcd数据库 # kubectl -n kube-system exec -it etcd-kmaster -- sh -c "ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key etcdctl --endpoints=https://127.0.0.1:2379 snapshot save /var/lib/etcd/snapshot.db" 查看数据文件状态 # kubectl -n kube-system exec -it etcd-kmaster -- sh -c "ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key etcdctl --endpoints=https://127.0.0.1:2379 snapshot status -w table /var/lib/etcd/snapshot.db" +----------+----------+------------+------------+ | HASH | REVISION | TOTAL KEYS | TOTAL SIZE | +----------+----------+------------+------------+ | 2eb468ae | 531875 | 1661 | 5.9 MB | +----------+----------+------------+------------+ 查看备份文件 # ls /var/lib/etcd/ member snapshot.db # mkdir $HOME/backup # cp /var/lib/etcd/snapshot.db ~/backup/
- 备份k8s etcd核心数据文件
# cp -r /etc/kubernetes/pki/etcd $HOME/backup/
- 确定升级版本
# apt update # apt-cache madison kubeadm 笔者已经安装了1.19.4,现打算升级到 1.20.0-00
2.2 升级master节点
- 升级kubeadm
# apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.20.0-00 && apt-mark hold kubeadm
- 检查kubeadm
# kubeadm version kubeadm version: &version.Info{ Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:57:36Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
- 核实升级plan
# kubeadm upgrade plan ...... [upgrade/versions] Latest version in the v1.19 series: v1.19.7 Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT AVAILABLE kubelet 4 x v1.19.4 v1.20.2 Upgrade to the latest stable version: COMPONENT CURRENT AVAILABLE kube-apiserver v1.19.7 v1.20.2 kube-controller-manager v1.19.7 v1.20.2 kube-scheduler v1.19.7 v1.20.2 kube-proxy v1.19.7 v1.20.2 CoreDNS 1.7.0 1.7.0 etcd 3.4.13-0 3.4.13-0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.20.2 Note: Before you can perform this upgrade, you have to update kubeadm to v1.20.2. ......
- 按照提示升级
此处升级为1.20.0,所以没有使用推荐的v1.20.2 # kubeadm upgrade apply v1.20.0 输出: ...... [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.20.0". Enjoy! [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
- drain 控制面板节点
# kubectl drain kmaster --ignore-daemonsets node/kmaster cordoned
- 升级 kubelet 和 kubectl
# apt-mark unhold kubelet kubectl && apt-get update && apt-get install -y kubelet=1.20.0-00 kubectl=1.20.0-00 && apt-mark hold kubelet kubectl 升级后重启kubelet 节点 # systemctl daemon-reload # systemctl restart kubelet
- Uncordon 控制面板节点
# kubectl uncordon kmaster
- 查看节点状态
# kubectl get nodes 发现节点为Ready状态
2.3 升级worker节点
- 升级kubeadm
# apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.20.0-00 && apt-mark hold kubeadm
- 检查kubeadm
# kubeadm version
- 更新本地kubelet 配置
# kubeadm upgrade node ...... [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [upgrade] The configuration for this node was successfully updated! ......
- drain knode01
# kubectl drain knode01 --ignore-daemonsets [--delete-emptydir-data 如果存储了本地数据,则需要delete emptydir]
- 升级 kubelet 和 kubectl
# apt-mark unhold kubelet kubectl && apt-get update && apt-get install -y kubelet=1.20.0-00 kubectl=1.20.0-00 && apt-mark hold kubelet kubectl 升级后重启kubelet 节点 # systemctl daemon-reload # systemctl restart kubelet
- Uncordon worker节点
# kubectl uncordon knode01
- 查看节点状态
# kubectl get nodes 发现knode01节点为Ready状态
至此master节点和一个worker节点升级完毕,其它节点可以按照worker节点的方法逐个升级。
3 注意事项
- 如果升级出错,可以从以下位置的文件进行恢复,直接执行 kubeadm upgrade apply --force 即可
# ls /etc/kubernetes/tmp/ kubeadm-backup-kubeadm-backup-etcd-2021-01-30-04-29-58/ kubeadm-backup-manifests-2021-01-30-04-29-58/
- 如果集群有多个master节点,则第2,3 个节点升级使用 kubeadm upgrade node 而非apply。
4 说明
- 软件环境
升级前 k8s 集群版本为:v1.19.4,升级后为1.20.0;
测试系统为ubuntu 16.04 server版本; - 参考文档
1 administer-cluster/kubeadm/kubeadm-upgrade/