Cluster概念
- Cluster:集群,为解决某个特定问题将多台计算机组合起来形成的单个系统
- Linux Cluster类型:
- LB:Load Balancing,负载均衡
-HA:High Availiablity,高可用,SPOF(single Point Of failure)- MTBF:Mean Time Between Failure 平均无故障时间
- MTTR:Mean Time To Restoration( repair)平均恢复前时间
- A=MTBF/(MTBF+MTTR) (0,1):99%, 99.5%, 99.9%, 99.99%, 99.999%
- HPC:High-performance computing,高性能 www.top500.org
- LB:Load Balancing,负载均衡
- 分布式系统:
- 分布式存储:Ceph,GlusterFS,FastDFS,MogileFS
- 分布式计算:hadoop,Spark
集群和分布式
- 集群:同一个业务系统,部署在多台服务器上。集群中,每一台服务器实现的功能没有差别,数据和代码都是一样的
- 分布式:一个业务被拆成多个子业务,或者本身就是不同的业务,部署在多台服务器上。分布式中,每一台服务器实现的功能是有差别的,数据和代码也是不一样的,分布式每台服务器功能加起来,才是完整的业务
- 分布式是以缩短单个任务的执行时间来提升效率的,而集群则是通过提高单位时间内执行的任务数来提升效率
Cluster分类
- lvs:Linux Virtual Server,阿里四层SLB (Server Load Balance)使用
- nginx:支持七层调度,阿里七层SLB使用Tengine
- haproxy:支持七层调度
- ats:Apache Traffic Server,yahoo捐助给apache
- perlbal:Perl 编写
- pound
分类
-
基于工作的协议层次划分:
- 传输层(通用):DPORT
- LVS:
- nginx:stream
- haproxy:mode tcp
- 应用层(专用):针对特定协议,自定义的请求模型分类
- proxy server:
- http:nginx, httpd, haproxy(mode http), …
- fastcgi:nginx, httpd, …
- mysql:mysql-proxy, …
- proxy server:
- 传输层(通用):DPORT
-
会话保持:负载均衡
- (1) session sticky:同一用户调度固定服务器
- Source IP:LVS sh算法(对某一特定服务而言)
- Cookie
- (2) session replication:每台服务器拥有全部session
- session multicast cluster
- (3) session server:专门的session服务器
- Memcached,Redis
- (1) session sticky:同一用户调度固定服务器
-
keepalived:vrrp协议
LVS
工作原理:
VS根据请求报文的目标IP和目标协议及端口将其调度转发至某RS,根据调度算法来挑选RS
kernel支持lvs
[root@firewalld ~]# grep -i -A 10 ipvs /boot/config-3.10.0-1062.el7.x86_64
CONFIG_NETFILTER_XT_MATCH_IPVS=m
CONFIG_NETFILTER_XT_MATCH_LENGTH=m
CONFIG_NETFILTER_XT_MATCH_LIMIT=m
CONFIG_NETFILTER_XT_MATCH_MAC=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_NFACCT=m
CONFIG_NETFILTER_XT_MATCH_OSF=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
--
# IPVS transport protocol load balancing support
#
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_AH_ESP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y
CONFIG_IP_VS_PROTO_SCTP=y
#
# IPVS scheduler 默认支持的算法
#
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
-
lvs集群类型中的术语:
- VS:Virtual Server,Director Server(DS)
Dispatcher(调度器),Load Balancer - RS:Real Server(lvs), upstream server(nginx)
backend server(haproxy) - CIP:Client IP
- VIP: Virtual serve IP VS外网的IP
- DIP: Director IP VS内网的IP
- RIP: Real server IP
- 访问流程:CIP <–> VIP == DIP <–> RIP
- VS:Virtual Server,Director Server(DS)
-
lvs: ipvsadm/ipvs
- ipvsadm:用户空间的命令行工具,规则管理器
用于管理集群服务及RealServer - ipvs:工作于内核空间netfilter的INPUT钩子上的框架
- ipvsadm:用户空间的命令行工具,规则管理器
-
lvs集群的类型:
- lvs-nat:修改请求报文的目标IP,多目标IP的DNAT
- lvs-dr:操纵封装新的MAC地址
- lvs-tun:在原请求IP报文之外新加一个IP首部
- lvs-fullnat:修改请求报文的源和目标IP
lvs-nat模式
本质是多目标IP的DNAT,通过将请求报文中的目标地址和目标端口修改为某挑出的RS的RIP和PORT实现转发
- (1)RIP和DIP应在同一个IP网络,且应使用私网地址;RS的网关要指向DIP
- (2)请求报文和响应报文都必须经由Director转发,Director易于成为系统瓶颈
- (3)支持端口映射,可修改请求报文的目标PORT
- (4)VS必须是Linux系统,RS可以是任意OS系统
LVS-DR模式
LVS-DR:Direct Routing,直接路由,LVS默认模式,应用最广泛,通过为请求报文重新封装一个MAC首部进行转发,源MAC是DIP所在的接口的MAC,目标MAC是某挑选出的RS的RIP所在接口的MAC地址;源IP/PORT,以及目标IP/PORT均保持不变
- (1) Director和各RS都配置有VIP
- (2) 确保前端路由器将目标IP为VIP的请求报文发往Director
- 第一种:在前端网关做静态绑定VIP和Director的MAC地址(不用)
LVS存在单点失败
- 第二种:在RS上使用arptables工具(不用)
arptables -A IN -d $VIP -j DROP
arptables -A OUT -s $VIP -j mangle --mangle-ip-s $RIP
路由转发给RS,RS收到后转发给VS,VS调度转发给RS
- 第三种:在RS上修改内核参数以限制arp通告及应答级别(默认)
/proc/sys/net/ipv4/conf/all/arp_ignore
/proc/sys/net/ipv4/conf/all/arp_announce
lo网卡配置VIP,VIP不应答ARP
[root@firewalld ~]# find /proc -name "arp_ignore"
/proc/sys/net/ipv4/conf/all/arp_ignore
/proc/sys/net/ipv4/conf/default/arp_ignore
/proc/sys/net/ipv4/conf/ens33/arp_ignore
/proc/sys/net/ipv4/conf/ens37/arp_ignore
/proc/sys/net/ipv4/conf/lo/arp_ignore
有针对所有和单个网卡的配置/etc/sysctl.conf
- (3)RS的RIP可以使用私网地址,也可以是公网地址;RIP与DIP在同一IP网络;RIP的网关不能指向DIP,以确保响应报文不会经由Director
- (4)RS和Director要在同一个物理网络
- (5)请求报文要经由Director,但响应报文不经由Director,而由RS直接发往Client
- (6)不支持端口映射(端口不能修败)
- (7)RS可使用大多数OS系统
lvs-tun
转发方式:不修改请求报文的IP首部(源IP为CIP,目标IP为VIP),而在原IP报文之外再封装一个IP首部(源IP是DIP,目标IP是RIP),将报文发往挑选出的目标RS;RS直接响应给客户端(源IP是VIP,目标IP是CIP)
(1) DIP, VIP, RIP都应该是公网地址
(2) RS的网关一般不能指向DIP
(3) 请求报文要经由Director,但响应不经由Director
(4) 不支持端口映射
(5) RS的OS须支持隧道功能
实验:实现基于NAT模型的LVS
1、RS环境搭建
为了方便观察,RS1和RS2网页不一样
[root@RS2 ~]# yum install httpd -y
[root@RS2 ~]# systemctl start httpd
[root@RS2 ~]# echo "welcome to RS2" >/var/www/html/index.html
[root@RS2 ~]# vieth0
IPADDR=192.168.8.27
GATEWAY=192.168.8.7
[root@RS2 ~]# systemctl restart network
[root@RS1 ~]# yum install httpd -y
[root@RS1 ~]# systemctl start httpd
[root@RS1 ~]# echo "welcome to RS2" >/var/www/html/index.html
[root@RS1 ~]# vieth0
IPADDR=192.168.8.17
GATEWAY=192.168.8.7
[root@RS1 ~]# systemctl restart network
2、LVS配置转发功能及安装ipvsadm
[root@LVS ~]# vim /etc/sysctl.conf
net.ipv4.ip_forward = 1
[root@LVS ~]# sysctl -p
net.ipv4.ip_forward = 1
[root@LVS ~]# yum install ipvsadm
[root@LVS ~]# ipvsadm -L 查看
[root@LVS ~]# ipvsadm -A -t 172.16.8.7:80 -s rr #rr轮询
[root@LVS ~]# ipvsadm -a -t 172.16.8.7:80 -r 192.168.8.17 -m #-m(NAT模型,默认为DR模型)
[root@LVS ~]# ipvsadm -a -t 172.16.8.7:80 -r 192.168.8.27 -m
[root@LVS ~]# ipvsadm -Ln #查看状态
[root@LVS ~]# ipvsadm -Ln --stats 统计信息
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Conns InPkts OutPkts InBytes OutBytes
-> RemoteAddress:Port
TCP 172.16.8.7:80 0 0 0 0 0
-> 192.168.8.17:80 0 0 0 0 0
-> 192.168.8.27:80 0 0 0 0 0
[root@LVS ~]# ipvsadm -Ln --rate 输出速率信息
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port CPS InPPS OutPPS InBPS OutBPS
-> RemoteAddress:Port
TCP 172.16.8.7:80 0 0 0 0 0
-> 192.168.8.17:80 0 0 0 0 0
-> 192.168.8.27:80 0 0 0 0 0
[root@CentOS6 ~]# while true ;do curl 172.16.8.7; sleep 0.5 ;done
welcome to RS1
welcome to RS2
welcome to RS1
welcome to RS2
welcome to RS1
welcome to RS2
welcome to RS1
#[root@LVS ~]# ipvsadm -e -t 172.16.8.7:80 -r 192.168.8.27 -m 修改
[root@LVS ~]# ipvsadm -E -t 172.16.8.7:80 -s wrr 修改算法wrr
[root@LVS ~]# ipvsadm -e -t 172.16.8.7:80 -r 192.168.8.27 -w 2 -m 增加权重
[root@LVS ~]# ipvsadm -e -t 172.16.8.7:80 -r 192.168.8.17 -w 6 -m
[root@LVS ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.8.7:80 wrr
-> 192.168.8.17:80 Masq 6 0 0
-> 192.168.8.27:80 Masq 2 0 0
[root@CentOS6 ~]# while true ;do curl 172.16.8.7; sleep 0.5 ;done
welcome to RS2
welcome to RS1
welcome to RS1
welcome to RS1
welcome to RS2
welcome to RS1
welcome to RS1
welcome to RS1
修改端口
[root@LVS ~]# ipvsadm -d -t 172.16.8.7:80 -r 192.168.8.17 删除
[root@LVS ~]# ipvsadm -d -t 172.16.8.7:80 -r 192.168.8.27
[root@LVS ~]# ipvsadm -a -t 172.16.8.7:80 -r 192.168.8.17:8080 -w 6 -m
[root@LVS ~]# ipvsadm -a -t 172.16.8.7:80 -r 192.168.8.27:8080 -w 2 -m
[root@LVS ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.8.7:80 wrr
-> 192.168.8.17:8080 Masq 6 0 0
-> 192.168.8.27:8080 Masq 2 0 0
[root@RS2 ~]# sed -i "s/Listen 80/Listen 8080/" /etc/httpd/conf/httpd.conf
[root@RS2 ~]# systemctl restart httpd
[root@RS1 ~]# sed -i "s/Listen 80/Listen 8080/" /etc/httpd/conf/httpd.conf
[root@RS1 ~]# systemctl restart httpd
[root@CentOS6 ~]# while true ;do curl 172.16.8.7; sleep 0.5 ;done
welcome to RS2
welcome to RS1
welcome to RS1
welcome to RS1
welcome to RS2
welcome to RS1
welcome to RS1
welcome to RS1
[root@LVS ~]# ipvsadm -E -t 172.16.8.7:80 -s sh 指向一个RS
[root@CentOS6 ~]# while true ;do curl 172.16.8.7; sleep 0.5 ;done
welcome to RS1
welcome to RS1
welcome to RS1
[root@LVS ~]# ipvsadm -E -t 172.16.8.7:80 -s dh
[root@CentOS6 ~]# while true ;do curl 172.16.8.7; sleep 0.5 ;done
welcome to RS1
welcome to RS1
welcome to RS1
指向一个RS,主要用于服务商提供缓存业务(长城宽带提供缓存,用户访问优酷时直接指向该缓存服务器,不占用带宽)
缺点:不能知道RS状态
实验:实现基于NAT模型的LVS调度Apache
1、基于fcig端口安装WordPress及mariadb配置
[root@RS1 ~]# yum install httpd php php-fpm php-mysql -y
[root@RS1 ~]# vim /etc/httpd/conf.d/fcgi.conf
DirectoryIndex index.php
ProxyRequests Off
ProxyPassMatch ^/(.*\.php)$ fcgi://127.0.0.1:9000/var/www/html/$1
[root@RS1 ~]# tar xf wordpress-4.8.3-zh_CN.tar.gz -C /var/www/html/
drwxr-xr-x 5 nobody nfsnobody 4096 Nov 16 2017 wordpress
[root@RS1 /var/www/html]# setfacl -Rm apache:rwx wordpress
[root@MariaDB ~]# yum install mariadb-server -y
[root@MariaDB ~]# systemctl start mariadb.service
[root@MariaDB ~]# mysql -e "create database wpdb"
[root@MariaDB ~]# mysql -e "grant all on wpdb.* to wpuser@'192.168.8.%' identified by 'centos'"
win登录http://192.168.8.17/wordpress/安装软件
[root@RS1 ~]# yum install httpd php php-fpm php-mysql -y
[root@RS1 ~]# vim /etc/httpd/conf.d/fcgi.conf
DirectoryIndex index.php
ProxyRequests Off
ProxyPassMatch ^/(.*\.php)$ fcgi://127.0.0.1:9000/var/www/html/$1
[root@RS1 /var/www/html]# scp -r wordpress 192.168.8.27:/var/www/html/
2、LVS配置
[root@LVS ~]# yum install ipvsadm -y
[root@LVS ~]# vim /etc/sysctl.conf
[root@LVS ~]# sysctl -p
net.ipv4.ip_forward = 1
[root@LVS ~]# ipvsadm -A -t 172.16.8.7:80 -s rr
[root@LVS ~]# ipvsadm -a -t 172.16.8.7:80 -r 192.16.8.17 -m
[root@LVS ~]# ipvsadm -a -t 172.16.8.7:80 -r 192.16.8.27 -m
[root@LVS ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.8.7:80 rr
-> 192.16.8.17:80 Masq 1 0 0
-> 192.16.8.27:80 Masq 1 0 0
实验:实现基于DR模型(同网段)的LVS
1、路由器设置
[root@ROUTING ~]# vim /etc/sysctl.conf
net.ipv4.ip_forward = 1
[root@ROUTING ~]# sysctl -p
net.ipv4.ip_forward = 1
2、配置RS1、RS2 的VIP及配置相关
[root@RS2 ~]# vim ipvs_RS.sh
#!/bin/bash
vip=192.168.8.100
mask='255.255.255.255'
dev=lo:1
case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig $dev $vip netmask $mask #broadcast $vip up
#route add -host $vip dev $dev
;;
stop)
ifconfig $dev down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
;;
*)
echo "Usage: $(basename $0) start|stop"
exit 1
;;
esac
[root@RS2 ~]# bash ipvs_RS.sh start
[root@RS2 ~]# scp ipvs_RS.sh 192.168.8.17:
[root@RS1 ~]# bash ipvs_RS.sh start
##注意:两个网卡上的网段不能一样,否则找不到出口
3、LVS配置
[root@LVS ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens37
IPADDR=192.168.8.100
NETMASK=255.255.255.0
GATEWAY=192.168.8.2 VIP必须有网关,但可以随意配,只要有就行
[root@LVS ~]# ipvsadm -A -t 192.168.8.100:80 -s rr
[root@LVS ~]# ipvsadm -a -t 192.168.8.100:80 -r 192.168.8.17
[root@LVS ~]# ipvsadm -a -t 192.168.8.100:80 -r 192.168.8.27
[root@LVS ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.8.100:80 rr
-> 192.168.8.17:80 Route 1 0 0
-> 192.168.8.27:80 Route 1 0 0
[root@LVS ~]# vim /etc/sysctl.conf
net.ipv4.ip_forward = 1
[root@LVS ~]# sysctl -p
net.ipv4.ip_forward = 1
lvs配置脚本
#!/bin/bash
vip='192.168.8.100'
iface='lo:1'
mask='255.255.255.255'
port='80'
rs1='192.168.8.17'
rs2='192.168.8.27'
scheduler='wrr'
type='-g'
case $1 in
start)
ifconfig $iface $vip netmask $mask #broadcast $vip up
iptables -F
ipvsadm -A -t ${vip}:${port} -s $scheduler
ipvsadm -a -t ${vip}:${port} -r ${rs1} $type -w 1
ipvsadm -a -t ${vip}:${port} -r ${rs2} $type -w 1
;;
stop)
ipvsadm -C
ifconfig $iface down
;;
*)
echo "Usage $(basename $0) start|stop"
exit 1
esac
实验:基于DR模型(不同网段)的LVS
1、配置router
[root@router ~]# ip a a 10.0.0.10/8 dev ens33:1
[root@router ~]# vim /etc/sysctl.conf
net.ipv4.ip_forward = 1
[root@router ~]# sysctl -p
net.ipv4.ip_forward = 1
[root@RS2 ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
[root@RS2 ~]# echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
[root@RS2 ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
[root@RS2 ~]# echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
[root@RS1 ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
[root@RS1 ~]# echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
[root@RS1 ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
[root@RS1 ~]# echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
2、RS安装配置httpd
[root@RS1 ~]# ip a a 10.0.0.100/32 dev lo:1
[root@RS2 ~]# ip a a 10.0.0.100/32 dev lo:1
[root@RS1 ~]# yum install httpd -y
[root@RS1 ~]# echo RS1 >/var/www/html/index.html
其余步骤省略
3、配置LVS
[root@LVS ~]# yum install ipvsadm
[root@LVS ~]# ip a a 10.0.0.100/32 dev lo:1
[root@LVS ~]# ipvsadm -A -t 10.0.0.100:80 -s rr
[root@LVS ~]# ipvsadm -a -t 10.0.0.100:80 -r 192.168.8.17
[root@LVS ~]# ipvsadm -a -t 10.0.0.100:80 -r 192.168.8.27
4、测试
[root@client ~]# while : ;do curl 10.0.0.100 ;done
welcome to RS2
welcome to RS1
welcome to RS2
welcome to RS1
welcome to RS2
welcome to RS1
实验:基于DR模型(不同网段)的LVS(二)
在这里插入代码片
长久连接
[root@LVS ~]# ipvsadm -E -t 10.0.0.100:80 -s rr -p
-p(默认360秒,一直发给一个主机)
[root@LVS ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.0.0.100:80 rr persistent 360(默认360)
-> 192.168.8.17:80 Route 1 0 2
-> 192.168.8.27:80 Route 1 0 0
[root@LVS ~]# ipvsadm -A -t 10.0.0.100:443 -s rr
[root@LVS ~]# ipvsadm -a -t 10.0.0.100:443 -r 192.168.8.17
[root@LVS ~]# ipvsadm -a -t 10.0.0.100:443 -r 192.168.8.27
[root@LVS ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.0.0.100:80 rr persistent 360
-> 192.168.8.17:80 Route 1 0 2
-> 192.168.8.27:80 Route 1 0 0
TCP 10.0.0.100:443 rr
-> 192.168.8.17:443 Route 1 0 0
-> 192.168.8.27:443 Route 1 0 0
两个集群,但是一个服务
[root@LVS ~]# iptables -t mangle -A PREROUTING -d 10.0.0.100 -p tcp -m multiport --dport 80,443 -j MARK --set-mark 10
[root@LVS ~]# ipvsadm -A -f 10 -s rr
[root@LVS ~]# ipvsadm -a -f 10 -r 192.168.8.17
[root@LVS ~]# ipvsadm -a -f 10 -r 192.168.8.27
[root@LVS ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 10 rr
-> 192.168.8.17:0 Route 1 0 1
-> 192.168.8.27:0 Route 1 0 0
一个集群
实验:ldirectord实现LVS高可用性
LVS配置
[root@LVS ~]# cp /usr/share/doc/ldirectord-3.9.6/ldirectord.cf /etc/ha.d/
[root@LVS ~]# vim /etc/ha.d/ldirectord.cf
checktimeout=3
checkinterval=1
autoreload=yes
logfile="/var/log/ldirectord.log"
quiescent=no
# Sample for an http virtual service
virtual=192.168.8.100:80
real=192.168.8.17:80 gate
real=192.168.8.27:80 gate
fallback=127.0.0.1:80 gate
service=http
scheduler=rr
protocol=tcp
checktype=negotiate
checkport=80
request="index.html"
receive="Test Page"
[root@LVS ~]# echo sorry server >/var/www/html/index.html
[root@LVS ~]# ip a a 192.168.8.100/32 dev lo:1
[root@CentOS7 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.8.7 0.0.0.0 UG 100 0 0 ens33
[root@LVS ~]# systemctl start httpd
[root@LVS ~]# systemctl start ldirectord.service
[root@LVS ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.8.100:80 rr
-> 192.168.8.17:80 Route 1 0 26
-> 192.168.8.27:80 Route 1 0 77
2、router配置
[root@router ~]# cat /etc/sysctl.conf
net.ipv4.ip_forward=1
[root@CentOS7 ~]# sysctl -p
net.ipv4.ip_forward = 1
3、RS配置
[root@RS2 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.8.7 0.0.0.0 UG 100 0 0 ens33
192.168.8.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
[root@RS2 ~]# systemctl restart httpd
[root@RS1 ~]# systemctl restart network
[root@RS1 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.8.7 0.0.0.0 UG 100 0 0 ens33
192.168.8.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
[root@RS1 ~]# systemctl start httpd
[root@RS1 ~]# curl 192.168.8.17
welcome to RS1
[root@RS1 ~]# ip a a 192.168.8.100/32 dev lo:1
[root@RS1 ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
[root@RS1 ~]# echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
[root@RS1 ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
[root@RS1 ~]# echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
[root@RS2 ~]# ip a a 192.168.8.100/32 dev lo:1
[root@RS2 ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
[root@RS2 ~]# echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
[root@RS2 ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
[root@RS2 ~]# echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
实验:实现Keepalived浮动IP
1、LVS1、2环境配置
[root@LVS2 ~]# ssh-keygen -t rsa -P "" -f /root/.ssh/id_rsa
[root@LVS2 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub 127.0.0.1
[root@LVS2 ~]# scp -r /root/.ssh 192.168.8.37:/root/
####LVS相互访问
[root@LVS2 ~]# yum install keepalived
[root@LVS2 ~]# rpm -ql keepalived
[root@LVS2 ~]# cp /etc/keepalived/keepalived.conf{,.bak}
2、LVS配置
[root@LVS2 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost #邮件发送
}
notification_email_from keepalived@localhost #谁发送邮件
smtp_server 127.0.0.1 #邮件发送地址
smtp_connect_timeout 30
router_id ka1 #主机名(lvs需不一样)
vrrp_mcast_group4 224.100.100.100 #组播(前面是224,后面都可以)
}
vrrp_instance VI_1 { #第一个服务
state MASTER #主服务(另一个为backup备用)
interface ens33 #接口
virtual_router_id 51 #节点号,必须一致
priority 100 #优先级(数字越大越优先)
advert_int 1 #多久广播一次
authentication {
auth_type PASS #预共享密钥认证
auth_pass 123456 #共享密钥(8位)
}
virtual_ipaddress {
192.168.8.100/24 dev ens33 label ens33:1 #VIP地址
}
[root@LVS2 ~]# scp /etc/keepalived/keepalived.conf 192.168.8.37:/etc/keepalived/
[root@LVS1 ~]# vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id ka2 #修改
vrrp_mcast_group4 224.100.100.100
}
vrrp_instance VI_1 {
state backup #修改
interface ens33
virtual_router_id 51
priority 80 #修改
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
192.168.8.100/24 dev ens33 label ens33:1
3、抓看查看VIP浮动
[root@RS2 ~]# tcpdump -i tcpdump -i ens33 -nn host 224.100.100.100
[root@LVS2 ~]# systemctl start keepalived.service
查看RS2抓包及LVS2 IP可以知道其拥有192.168.8.100
[root@LVS1 ~]# systemctl start keepalived.service
查看RS2抓包LVS1宣称拥有192.168.8.100优先级高查看LVS1及LVS2 IP可以知道LVS1拥有192.168.8.100
[root@LVS1 ~]# systemctl stop keepalived.service
停止后IP变动
上一实验基础上实现脚本通知
[root@LVS2 ~]# vim /etc/keepalived/notify.sh
contact='root@localhost'
notify() {
mailsubject="$(hostname) to be $1, vip floating"
mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
echo "$mailbody" | mail -s "$mailsubject" $contact
}
case $1 in
master)
notify master
;;
backup)
notify backup
;;
fault)
notify fault
;;
*)
echo "Usage: $(basename $0) {master|backup|fault}"
exit 1
;;
esac
[root@LVS2 ~]# chmod +x /etc/keepalived/notify.sh
[root@LVS2 ~]# vim /etc/keepalived/keepalived.conf
virtual_ipaddress {
192.168.8.100/24 dev ens33 label ens33:1
}
notify_master "/etc/keepalived/notify.sh master" 增加脚本
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
###LVS1修改照上述例子(略)
自定义日志
[root@LVS2 ~]# vim /etc/sysconfig/keepalived
KEEPALIVED_OPTIONS="-D -S 6"
[root@LVS2 ~]# vim /etc/rsyslog.conf
local6.* /var/log/keepalived.log
[root@LVS2 ~]# systemctl restart rsyslog
[root@LVS2 ~]# systemctl restart keepalived.service
[root@LVS2 ~]# cat /var/log/keepalived.log
Jul 30 22:50:31 CentOS7 Keepalived[52056]: Stopping
Jul 30 22:50:31 CentOS7 Keepalived_healthcheckers[52057]: Stopped
QQ邮件通知
[root@LVS2 ~]# cat .mailrc
set from=[email protected]
set smtp=smtp.qq.com
set smtp-auth-user=[email protected]
set smtp-auth-password=lszbrwwphsszbdae
set smtp-auth=login
set ssl-verify=ignore
测试:echo test mail | mail –s test [email protected]
实验:多服务多主高可用
LVS1配置
[root@LVS1 ~]# yum install keepalived
[root@LVS1 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id ka2
vrrp_mcast_group4 224.100.100.100
}
vrrp_instance VI_1 {
state backup
interface ens33
virtual_router_id 51
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.8.100/24 dev ens33 label ens33:1
}
}
virtual_server 192.168.8.100 80 {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
sorry_server 127.0.0.1 80
real_server 192.168.8.17 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.8.27 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
vrrp_instance VI_2 {
state MASTER
interface ens33
virtual_router_id 60
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 2222
}
virtual_ipaddress {
192.168.8.200/24 dev ens33 label ens33:2
}
}
virtual_server 192.168.8.200 443 {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
sorry_server 127.0.0.1 80
real_server 192.168.8.57 443 {
weight 1
SSL_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.8.67 443 {
weight 1
SSL_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
2、LVS2配置
[root@LVS2 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id ka1
vrrp_mcast_group4 224.100.100.100
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.8.100 dev ens33 label ens33:1
}
}
vrrp_instance VI_2 {
state backup
interface ens33
virtual_router_id 60
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 2222
}
virtual_ipaddress {
192.168.8.200 dev ens33 label ens33:2
}
}
virtual_server 192.168.8.100 80 {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
sorry_server 127.0.0.1 80
real_server 192.168.8.17 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.8.27 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
virtual_server 192.168.8.200 80 {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
sorry_server 127.0.0.1 80
real_server 192.168.8.57 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.8.67 80 {
weight 1
SSL_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
其他配置省略(RS配置)路由指向