- 一、分析LVS—DR群集工作原理
- 二、LVS-DR中的ARP问题及办法
-
- 问题1:
-
- 产生原因:在LVS-DR负载均衡集群中,负载均衡器与节点服务器都要配置相同的VIP地址![在这里插入图片描述](https://img-blog.csdnimg.cn/20200922165522336.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0NOX0xpVGlhbnBlbmc=,size_16,color_FFFFFF,t_70#pic_center)
- 造成结果:在局域网中具有相同的IP地址,势必会造成各服务器ARP通信的紊乱![在这里插入图片描述](https://img-blog.csdnimg.cn/20200922170626329.png#pic_center)
- 改进办法
- 问题2:
- 解决ARP的两个问题的设置方法
- 应用场景
- 实验
- 测试DR负载均衡
- 其他设置
一、分析LVS—DR群集工作原理
将Client与群集机器放在同一网络中数据包流经的路线为 1 -> 2 -> 3 -> 4 .
1. Client向目标VIP发出请求,Director(负载均衡器)接收
2.Director根据负载均衡算法选择RealServer_1,不修改也不封装IP报文,而是将数据帧的MAC地址改为RealServer_1的MAC地址,然后在局域网上发送
3. RealServer_1收到这个帧,解封装后发现目标IP与本机匹配(RealServer事先绑定了VIP),于是处理这个报文。随后重新封装报文,发送到局域网
4.Client将收到回复报文。Client认为得到正常的服务,而不会知道是哪一台服务器处理的
在LVS-DR负载均衡集群中,负载均衡器与节点服务器都要配置相同的VIP地址internet返回给客户
二、LVS-DR中的ARP问题及办法
问题1:
产生原因:在LVS-DR负载均衡集群中,负载均衡器与节点服务器都要配置相同的VIP地址
- 在局域网中具有相同的IP地址,势必会造成各服务器ARP通信的紊乱当ARP广播发送到LVS-DR集群时,因为负载均衡器和节点服务器都是连接到相同的网络上,它们都会接收到ARP广播
- 只有前端的负载均衡器进行响应,其他节点服务器不应该响应ARP广播VIP地址
造成结果:在局域网中具有相同的IP地址,势必会造成各服务器ARP通信的紊乱
改进办法
- 对节点服务器进行处理,使其不响应针对VIP的ARP请求
- 使用虚接口lo:0承载VIP地址设置内核参数arp _ignore=1:系统只响应目的IP为本地IP的ARP请求**
问题2:
产生原因:ARP请求使用IP包的源IP地址(即VIP)作为ARP请求包中的源IP地址
- RealServer返回报文(源IP是VIP)经路由器转发,重新封装报文时,需要先获取路由器的MAC地址
- 发送ARP请求时,Linux默认使用IP包的源IP地址(即VIP)作为ARP请求包中的源IP地址,而不使用发送接口的IP地址
造成结果:MAC地址表中调度器VIP地址对应的MAC地址被RealServer的MAC地址覆盖,是调度器失效
- 路由器收到ARP请求后,将更新ARP表项
- 原有的VIP对应Director的MAC地址会被更新为VIP对应RealServer的MAC地址
- 路由器根据ARP表项,会将新来的请求报文转发给RealServer,导致Director的VIP失效
解决办法
- 对节点服务器进行处理,设置内核参数arp_announce=2:系统不使用IP包的源地址来设置ARP请求的源地址,而选择发送接口的IP地址
解决ARP的两个问题的设置方法
修改letc/sysctl.conf文件
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_annougce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
应用场景
实验
实验设计拓扑
实验步骤
1.搭建调度器
1.配置调度器
【负载均衡调度器】
###############配置VIP地址#########################
[root@localhost ~]# cd /etc/sysconfig/network-scripts/
[root@localhost network-scripts]# cp ifcfg-ens33 ifcfg-ens33:0
[root@localhost network-scripts]# vi ifcfg-ens33:0
[root@localhost network-scripts]# systemctl restart network
[root@localhost network-scripts]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.30.11 netmask 255.255.255.0 broadcast 192.168.30.255
……省略部分
ens33:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.30.100 netmask 255.255.255.255 broadcast 192.168.30.100
ether 00:0c:29:2e:3b:31 txqueuelen 1000 (Ethernet)
###################安装ipvsadm程序###############################
[root@localhost ~]# yum -y install ipvsadm
[root@localhost ~]# modprobe ip_vs
[root@localhost ~]# cat /proc/net/ip_vs
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn
[root@localhost ~]# ipvsadm -v
ipvsadm v1.27 2008/5/15 (compiled with popt and IPVS v1.2.1)
[root@localhost ~]# systemctl status ipvsadm
##################清空负载分配规则########################
[root@localhost ~]# ipvsadm -C
###############创建虚拟服务器规则################################
[root@localhost ~]# ipvsadm -A -t 192.168.30.100:80 -s rr
##############添加Web服务器节点负载分配规则##############################
[root@localhost ~]# ipvsadm -a -t 192.168.30.100:80 -r 192.168.30.22:80 -g -w 1
[root@localhost ~]# ipvsadm -a -t 192.168.30.100:80 -r 192.168.30.33:80 -g -w 1
##############保存规则##############################
[root@localhost ~]# ipvsadm --save > /etc/sysconfig/ipvsadm
[root@localhost ~]# cat /etc/sysconfig/ipvsadm
-A -t 192.168.30.100:http -s rr
-a -t 192.168.30.100:http -r 192.168.30.22:http -g -w 1
-a -t 192.168.30.100:http -r 192.168.30.33:http -g -w 1
[root@localhost network-scripts]# systemctl start ipvsadm.service
[root@localhost network-scripts]# systemctl status ipvsadm.service
● ipvsadm.service - Initialise the Linux Virtual Server
Loaded: loaded (/usr/lib/systemd/system/ipvsadm.service; disabled; vendor preset: disabled)
Active: active (exited) since Tue 2020-09-22 18:52:53 CST; 3s ago
Process: 22042 ExecStart=/bin/bash -c exec /sbin/ipvsadm-restore < /etc/sysconfig/ipvsadm
……省略部分
######################调整/proc响应参数##########################
[root@localhost network-scripts]# vi /etc/sysctl.conf
####插入下列配置使主机不充当路由器,即关闭ARP功能
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0
[root@localhost network-scripts]# sysctl -p
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0
2.搭建服务器池
2.1配置WEB1
[root@localhost ~]# cd /etc/sysconfig/network-scripts/
[root@localhost network-scripts]# cp ifcfg-lo ifcfg-lo:0
[root@localhost network-scripts]# vi ifcfg-enslo:0
DEVICE=lo:0
IPADDR=192.168.30.100
NETMASK=255.255.255.255
ONBOOT=yes
[root@localhost network-scripts]# systemctl restart network
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.30.33 netmask 255.255.255.0 broadcast 192.168.30.255
……省略部分
lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 192.168.30.100 netmask 255.255.255.255
[root@localhost network-scripts]# vi /etc/sysctl.conf
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
[root@mysql2 network-scripts]# sysctl -p
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
[root@localhost network-scripts]# vi /etc/rc.local
/sbin/route add -host 192.168.30.100 dev lo:0
[root@localhost network-scripts]# route add -host 192.168.30.100 dev lo:0
[root@mysql2 network-scripts]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.30.11 0.0.0.0 UG 100 0 0 ens33
192.168.30.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
192.168.30.100 0.0.0.0 255.255.255.255 UH 0 0 0 lo
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
[root@localhost ~]# yum -y install nfs-utils //使用showmount需要安装这个工具
[root@localhost ~]# showmount -e 192.168.30.44 //查看共享状况
Export list for 192.168.30.44:
/opt/web2 192.168.30.0/24
/opt/web1 192.168.30.0/24
[root@mysql2 ~]# yum -y install httpd //安装httpd
[root@mysql2 ~]# systemctl start httpd
[root@mysql2 ~]# systemctl enable httpd
#######挂载共享文件############
[root@localhost html]# vi /etc/fstab
192.168.30.44:/opt/web1 /var/www/html nfs defaults,_netdev 0 0
[root@localhost html]# mount 192.168.30.44:/opt/web1 /var/www/html/
2.2配置WEB2
[root@localhost html]# cd /etc/sysconfig/network-scripts/
[root@localhost network-scripts]#cp ifcfg-lo ifcfg-lo:0
[root@localhost network-scripts]# vi ifcfg-enslo:0
DEVICE=lo:0
IPADDR=192.168.30.100
NETMASK=255.255.255.255
ONBOOT=yes
[root@localhost network-scripts]# systemctl restart network
[root@localhost network-scripts]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.30.22 netmask 255.255.255.0 broadcast 192.168.30.255
……省略部分
lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 192.168.30.100 netmask 255.255.255.255
[root@localhost network-scripts]# vi /etc/sysctl.conf
###################插入下面配置,解决ARP映射问题参数#################
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
[root@mysql2 network-scripts]# sysctl -p
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
[root@localhost network-scripts]# vi /etc/rc.local
/sbin/route add -host 192.168.30.100 dev lo:0 //添加VIP本地访问路由
[root@localhost network-scripts]# route add -host 192.168.30.100 dev lo:0
[root@mysql2 network-scripts]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.30.11 0.0.0.0 UG 100 0 0 ens33
192.168.30.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
192.168.30.100 0.0.0.0 255.255.255.255 UH 0 0 0 lo
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
[root@localhost ~]# yum -y install nfs-utils
[root@localhost ~]# showmount -e 192.168.30.44 //若查看不到,可能是nfs服务器发布失败,去nfs服务器再次发布一下:exportsfs
Export list for 192.168.30.44:
/opt/web2 192.168.30.0/24
/opt/web1 192.168.30.0/24
[root@mysql2 ~]# yum -y install httpd
[root@mysql2 ~]# systemctl start httpd
[root@mysql2 ~]# systemctl enable httpd
#######挂载共享文件############
[root@localhost html]# vi /etc/fstab
192.168.30.44:/opt/web2 /var/www/html nfs defaults,_netdev 0 0
[root@localhost html]# mount 192.168.30.44:/opt/web2 /var/www/html/
3.搭建共享储存
[root@localhost ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.30.44 netmask 255.255.255.0 broadcast 192.168.30.255
inet6 fe80::a52a:406e:6512:1c66 prefixlen 64 scopeid 0x20<link>
[root@localhost ~]# route -n //查看路由表,看网关
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.30.11 0.0.0.0 UG 100 0 0 ens33
192.168.30.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
[root@localhost ~]# rpm -q nfs-utils //查看nfs是否安装
nfs-utils-1.3.0-0.61.el7.x86_64
[root@localhost ~]# rpm -q rpcbind //查看rpcbind是否安装
rpcbind-0.2.0-47.el7.x86_64
[root@localhost ~]# yum -y install nfs-utils //确实安装了
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
Package 1:nfs-utils-1.3.0-0.61.el7.x86_64 already installed and latest version
Nothing to do
[root@localhost ~]# yum -y install rpcbind
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
Package rpcbind-0.2.0-47.el7.x86_64 already installed and latest version
Nothing to do
[root@localhost ~]# systemctl start nfs
[root@localhost ~]# systemctl enable nfs
Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.
[root@localhost ~]# systemctl start rpcbind
[root@localhost ~]# systemctl enable rpcbind
[root@localhost ~]# vi /etc/exports
/opt/web1 192.168.30.0/24(rw,sync)
/opt/web2 192.168.30.0/24(rw,sync)
[root@localhost ~]# systemctl restart nfs
[root@localhost ~]# systemctl restart rpcbind
[root@localhost ~]# showmount -e
Export list for localhost.localdomain:
/opt/web2 192.168.30.0/24
/opt/web1 192.168.30.0/24
[root@localhost web2]# exportfs -vr
exporting 192.168.30.0/24:/opt/web2
exporting 192.168.30.0/24:/opt/web1
[root@localhost ~]# mkdir /opt/web1/ /opt/web1/
[root@localhost ~]# vi /opt/web1/index.html
<html>
<title>I'm Web1</title>
<body><h1>I'm Web1</h1></body>
<img src="web1.jpg" />
</html>
[root@localhost ~]# vi /opt/web2/index.html
<html>
<title>I'm Web2</title>
<body><h1>I'm Web2</h1></body>
<img src="web2.png" />
</html>
测试DR负载均衡
其他设置
———————————————————————————————————
-----常用命令的解释如下:------
LVS的负载调度算法有四种最常用的:轮询算法(rr)、加权轮询(wrr)、最少轮询(lc)、加权最少轮询(wlc)
1)创建虚拟服务器(注意:NAT模式要两块网卡,调度器的地址是外网口地址)
群集的VIP地址为192.168.80.33,针对TCP 80端口提供负载分流服务,使用的轮询调度算法。对于负载均衡调度器来说,VIP必须是本机实际已启用的IP地址
ipvsadm -A -t 192.168.20.11:80 -s rr
//选项 “-A"表示添加虚拟服务器,”-t"用来指定VIP地址及TCP端口,"-s"用来指定负载调度算法——rr、wrr、lc、wlc
2)添加服务器节点
ipvsadm -a -t 192.168.20.11:80 -r 192.168.80.33:80 -m
ipvsadm -a -t 192.168.20.11:80 -r 192.168.80.44:80 -m
//选项 “-a"表示添加真实服务器,”-t"用来指定VIP地址及TCP端口,"-r"用来指定RIP地址及TCP端口,"-m"表示使用NAT群集模式("-g"是DR模式,"-i"是TUN模式)
{ -m参数后面还可以跟-w的参数,这里没有做的"-w"用来设置权重(权重为0时表示暂停节点)}
4)删除服务器节点
ipvsadm -d -r 192.168.90.22:80 -t 192.168.80.88:80
//需要从服务器池中删除某一个节点时,使用选项"-d"。执行删除操作必须指定目标对象,包括节点地址、虚拟IP地址。如上所示的操作将会删除LVS群集192.168.80.88中的节点192.168.90.22
若需要删除整个虚拟服务器时,使用选项-D并指定虚拟IP地址即可,无需要指定节点。例如:“ipvsadm -D -t 192.168.80.11:80",则删除此虚拟服务器。
ipvsadm -L //查看节点状态,加个"-n"将以数字形式显示地址、端口信息
ipvsadm-save > /etc/sysconfig/ipvsadm //保存策略
使用导出/导入工具ipvsadm-save/ipvsadm-restore可以保存、恢复LVS策略,方法类似于iptables的规则的导出、导入