部署etcd集群 集群规划
主机名 角色 ip
HDss-1-12.host.com ectc lead 192.168.1.12
HDss-1-21.host.com ectc follow 192.168.1.21
HDss-1-22.host.com ectc follow 192.168.1.22
部署master节点服务
1. 在主配置服务器上生成证书 编辑 ca-config.json文件
{
"signing": {
"default": {
"expiry": "175200h"
},
"profiles": {
"server": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth"
]
},
"client": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"client auth"
]
},
"peer": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
2.在主配置服务器上生成证书 编辑 etcd-peer-csr.json文件
{
"CN": "k8s-etcd",
"hosts": [
"192.168.1.11",
"192.168.1.12",
"192.168.1.21",
"192.168.1.22"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
}
3.将编辑好的json文件上传到主控服务端
[root@hdss-1-200 certs]# ll
total 28
-rw-r--r-- 1 root root 836 Feb 9 16:52 ca-config.json
-rw-r--r-- 1 root root 993 Feb 9 15:30 ca.csr
-rw-r--r-- 1 root root 346 Feb 9 15:29 ca-csr.json
-rw------- 1 root root 1675 Feb 9 15:30 ca-key.pem
-rw-r--r-- 1 root root 1346 Feb 9 15:30 ca.pem
-rw-r--r-- 1 root root 375 Feb 9 16:52 etcd-peer-csr.json
4.生成承载式证书如下:
[root@hdss-1-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json |cfssl-json -bare etcd-peer
2021/02/09 16:53:42 [INFO] generate received request
2021/02/09 16:53:42 [INFO] received CSR
2021/02/09 16:53:42 [INFO] generating key: rsa-2048
2021/02/09 16:53:43 [INFO] encoded CSR
2021/02/09 16:53:43 [INFO] signed certificate with serial number 376584182207976051180015643037273875708040992548
2021/02/09 16:53:43 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@hdss-1-200 certs]# ll
total 40
-rw-r--r-- 1 root root 836 Feb 9 16:52 ca-config.json
-rw-r--r-- 1 root root 993 Feb 9 15:30 ca.csr
-rw-r--r-- 1 root root 346 Feb 9 15:29 ca-csr.json
-rw------- 1 root root 1675 Feb 9 15:30 ca-key.pem
-rw-r--r-- 1 root root 1346 Feb 9 15:30 ca.pem
-rw-r--r-- 1 root root 1062 Feb 9 16:53 etcd-peer.csr
-rw-r--r-- 1 root root 375 Feb 9 16:53 etcd-peer-csr.json
-rw------- 1 root root 1675 Feb 9 16:53 etcd-peer-key.pem
-rw-r--r-- 1 root root 1428 Feb 9 16:53 etcd-peer.pem
[root@hdss-1-200 certs]#
5. 开始部署etcd集群在12 21 22三台服务器上,这里部署文档以hdss-1-12.host.com主机为例,另外两台安装部署方法类似
创建用户
[root@hdss-1-12 ~]# useradd -s /sbin/nologin -M etcd
[root@hdss-1-12 ~]# id etcd
uid=1001(etcd) gid=1001(etcd) groups=1001(etcd)
6.下载软件并解压(下载etcd软件,建议用不超3.3的版本)
[root@hdss-1-12 ~]# mkdir /opt/src
[root@hdss-1-12 ~]# cd /opt/src/
[root@hdss-1-12 src]# ls
etcd-v3.1.20-linux-amd64.tar.gz
[root@hdss-1-12 src]# tar xf etcd-v3.1.20-linux-amd64.tar.gz -C /opt/
7.建立软连接,方便后续维护
[root@hdss-1-12 src]# cd /opt
[root@hdss-1-12 opt]# mv etcd-v3.1.20-linux-amd64 etcd-v3.1.20
[root@hdss-1-12 opt]# ln -s /opt/etcd-v3.1.20 /opt/etcd
[root@hdss-1-12 opt]# ll
total 0
lrwxrwxrwx 1 root root 17 Feb 9 17:11 etcd -> /opt/etcd-v3.1.20
drwxr-xr-x 3 478493 89939 123 Oct 11 2018 etcd-v3.1.20
drwxr-xr-x. 2 root root 6 Oct 31 2018 rh
drwxr-xr-x 2 root root 45 Feb 9 17:09 src
[root@hdss-1-12 opt]#
8. 创建目录,拷贝证书、私钥
[root@hdss-1-12 ~]# mkdir -p /opt/etcd/certs /data/etcd /data/logs/etcd-server
9.将运维主机上生成的ca.pem etc-peer-key.pem etc-peer.pem 拷贝到/opt/etcd/certs目录中,私钥文件权限为600
[root@hdss-1-12 etcd]# cd certs/
[root@hdss-1-12 certs]# pwd
/opt/etcd/certs
[root@hdss-1-12 certs]# ls
[root@hdss-1-12 certs]# scp hdss-1-200:/opt/certs/ca.pem .
[root@hdss-1-12 certs]# scp hdss-1-200:/opt/certs/etcd-peer.pem .
[root@hdss-1-12 certs]# scp hdss-1-200:/opt/certs/etcd-peer-key.pem .
[root@hdss-1-12 certs]# ll
total 12
-rw-r--r-- 1 root root 1346 Feb 9 17:19 ca.pem
-rw------- 1 root root 1675 Feb 9 17:22 etcd-peer-key.pem
-rw-r--r-- 1 root root 1428 Feb 9 17:20 etcd-peer.pem
10.更改属主属组
[root@hdss-1-12 certs]# chown -R etcd.etcd /opt/etcd/certs /data/etcd /data/logs/etcd-server
[root@hdss-1-12 certs]# ll
total 12
-rw-r--r-- 1 etcd etcd 1346 Feb 9 17:19 ca.pem
-rw------- 1 etcd etcd 1675 Feb 9 17:22 etcd-peer-key.pem
-rw-r--r-- 1 etcd etcd 1428 Feb 9 17:20 etcd-peer.pem
[root@hdss-1-12 certs]#
11.创建etcd服务启动脚本IP地址改成本机IP etcd-server-startup.sh 并上传
#!/bin/sh
./etcd --name etcd-server-1-12 \
--data-dir /data/etcd/etcd-server \
--listen-peer-urls https://192.168.1.12:2380 \
--listen-client-urls https://192.168.1.12:2379,http://127.0.0.1:2379 \
--quota-backend-bytes 8000000000 \
--initial-advertise-peer-urls https://192.168.1.12:2380 \
--advertise-client-urls https://192.168.1.12:2379,http://127.0.0.1:2379 \
--initial-cluster etcd-server-1-12=https://192.168.1.12:2380,etcd-server-1-21=https://192.168.1.21:2380,etcd-server-1-22=https://192.168.1.22:2380 \
--ca-file ./certs/ca.pem \
--cert-file ./certs/etcd-peer.pem \
--key-file ./certs/etcd-peer-key.pem \
--client-cert-auth \
--trusted-ca-file ./certs/ca.pem \
--peer-ca-file ./certs/ca.pem \
--peer-cert-file ./certs/etcd-peer.pem \
--peer-key-file ./certs/etcd-peer-key.pem \
--peer-client-cert-auth \
--peer-trusted-ca-file ./certs/ca.pem \
--log-output stdout
12 赋权并修改属主和属组
[root@hdss-1-12 etcd]# chmod +x /opt/etcd/etcd-server-startup.sh
[root@hdss-1-12 etcd]# chown -R etcd.etcd /opt/etcd-v3.1.20/ /data/etcd /data/logs/etcd-server
[root@hdss-1-12 etcd]#
13.为了使使etcd后端运行,需要安装supervisor (管理后台程序的软件)
[root@hdss-1-12 etcd]# yum install supervisor -y
14.启动supervisor
[root@hdss-1-12 etcd]# systemctl start supervisord
[root@hdss-1-12 etcd]# systemctl enable supervisord
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
15.更改supervisord的配置文件 etcd-server.ini
root@hdss-1-12 supervisord.d]# vi /etc/supervisord.d/etcd-server.ini
[program:etcd-server-1-12]
command=/opt/etcd/etcd-server-startup.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/etcd ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
16.创建后端启动etcd
[root@hdss-1-12 supervisord.d]# supervisorctl update
etcd-server-1-12: added process group
[root@hdss-1-12 supervisord.d]# supervisorctl status
etcd-server-1-12 STARTING
[root@hdss-1-12 supervisord.d]# netstat -luntp|grep etcd
tcp 0 0 192.168.1.12:2379 0.0.0.0:* LISTEN 60692/./etcd
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 60692/./etcd
tcp 0 0 192.168.1.12:2380 0.0.0.0:* LISTEN 60692/./etcd
17.另外两台部署流程和上面一样,需要修改俩配置文件
(1) 21上的配置文件etcd-server-startup.sh
[root@hdss-1-21 etcd]# vim etcd-server-startup.sh
#!/bin/sh
./etcd --name etcd-server-1-21 \
--data-dir /data/etcd/etcd-server \
--listen-peer-urls https://192.168.1.21:2380 \
--listen-client-urls https://192.168.1.21:2379,http://127.0.0.1:2379 \
--quota-backend-bytes 8000000000 \
--initial-advertise-peer-urls https://192.168.1.21:2380 \
--advertise-client-urls https://192.168.1.21:2379,http://127.0.0.1:2379 \
--initial-cluster etcd-server-1-12=https://192.168.1.12:2380,etcd-server-1-21=https://192.168.1.21:2380,etcd-server-1-22=https://192.168.1.22:2380 \
--ca-file ./certs/ca.pem \
--cert-file ./certs/etcd-peer.pem \
--key-file ./certs/etcd-peer-key.pem \
--client-cert-auth \
--trusted-ca-file ./certs/ca.pem \
--peer-ca-file ./certs/ca.pem \
--peer-cert-file ./certs/etcd-peer.pem \
--peer-key-file ./certs/etcd-peer-key.pem \
--peer-client-cert-auth \
--peer-trusted-ca-file ./certs/ca.pem \
--log-output stdout
(2)21 etcd-server.ini 配置文件
[root@hdss-1-21 etcd]# vi /etc/supervisord.d/etcd-server.ini
[program:etcd-server-1-21]
command=/opt/etcd/etcd-server-startup.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/etcd ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
(3) 22上的配置文件etcd-server-startup.sh
[root@hdss-1-22 etcd]# vim etcd-server-startup.sh
#!/bin/sh
./etcd --name etcd-server-1-22 \
--data-dir /data/etcd/etcd-server \
--listen-peer-urls https://192.168.1.22:2380 \
--listen-client-urls https://192.168.1.22:2379,http://127.0.0.1:2379 \
--quota-backend-bytes 8000000000 \
--initial-advertise-peer-urls https://192.168.1.22:2380 \
--advertise-client-urls https://192.168.1.22:2379,http://127.0.0.1:2379 \
--initial-cluster etcd-server-1-12=https://192.168.1.12:2380,etcd-server-1-21=https://192.168.1.21:2380,etcd-server-1-22=https://192.168.1.22:2380 \
--ca-file ./certs/ca.pem \
--cert-file ./certs/etcd-peer.pem \
--key-file ./certs/etcd-peer-key.pem \
--client-cert-auth \
--trusted-ca-file ./certs/ca.pem \
--peer-ca-file ./certs/ca.pem \
--peer-cert-file ./certs/etcd-peer.pem \
--peer-key-file ./certs/etcd-peer-key.pem \
--peer-client-cert-auth \
--peer-trusted-ca-file ./certs/ca.pem \
--log-output stdout
(4)22 etcd-server.ini 配置文件
[root@hdss-1-22 etcd]# vi /etc/supervisord.d/etcd-server.ini
[program:etcd-server-1-22]
command=/opt/etcd/etcd-server-startup.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/etcd ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
18.如果有报错可以查看日志
[root@hdss-1-12 supervisord.d]# tail -fn 200 /data/logs/etcd-server/etcd.stdout.log
查看集群状态是否正常
[root@hdss-1-22 etcd]# ./etcdctl cluster-health
[root@hdss-1-22 etcd]# ./etcdctl member list