CIP 客户端主机IP地址
VIP 虚拟IP地址(负载均衡网卡)
DIP Director的IP地址
RIP 真实IP地址
LVS调度算法:决定了如何在集群节点之间分布工作负荷
Lvs调度器 IP:192.168.1.14 VIP:192.168.1.29
RS-1 IP:192.168.1.15
RS-2 IP:192.168.1.16
一.Lvs调度器: M/S
- 准备:
yum install -y httpd
/etc/init.d/httpd start
netstat -lnt|grep 80 ps -ef|grep httpd
yum install -y lrzsz
yum install kernel-devel -y
ln -s /usr/src/kernels/2.6.32-358.el6.i686 /usr/src/linux
ll /usr/src/ |grep linux
- 安装ipvsadm:
tar -zxf ipvsadm-1.24.tar.gz -C /home/gaigai.xu
cd ipvsadm-1.24/
yum install gcc gcc-c++ -y 编译器
make
make install
Ipvsadm 执行
lsmod |grep ip_vs 模块加载
- 配置LVS虚拟IP(VIP)
vim /etc/sysctl.conf
net.ipv4.ip_forward = 1
sysctl -p
ifconfig eth1:0 192.168.1.29/24 netmask 255.255.255.255 up 要已有eth1才能绑好虚拟IP
Ifconfig eth1:0
route add -host 192.168.1.29 dev eth1 添加路由
route -n
ipvsadm -C 清空列表
ipvsadm --set 30 5 60 设定超时三个参数
ipvsadm -A -t 192.168.1.29:80 -s wrr -p 20 制定Ip端口,指定调度算法,绘画保持
ipvsadm -L -n 查看
ipvsadm -a -t 192.168.1.29:80 -r 192.168.1.14 -g -w 1 添加rslserver IP,DR模式,指定权重
ipvsadm -d -t 192.168.1.29:80 -r 192.168.1.15 删除某个rslserver
二.RS-1/RS-2
ifconfig lo:17 192.168.1.29 netmask 255.255.255.255 up 绑定VIP
route add -host 192.168.1.29 dev lo 添加路由
echo "1">/proc/sys/net/ipv4/conf/lo/arp_ignore 拟制ARP
echo "2">/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1">/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2">/proc/sys/net/ipv4/conf/all/arp_announce
LVS-master中:
watch --interval=1 ipvsadm -Ln 1秒钟跳一下
LVS-slave:装好LVS(同master)
LVS切换:LVS-slave上执行着 sh check_lb.sh & 监听VIP是否正常
当主机当即:会自动执行ipvs_dr_ctl 启动接管RS 实现自动切换
LVS三种模式:
- NAT(数据包来回都要经过调度器,需要开启内核转发net.ipv.ip_forward=1)
- DR(更改目的MAC)
- TUNNEL (IP隧道)
master自启动脚本见附件(分三种)
LVS RS节点健康检查功能(见脚本)
LVS调度器裂脑问题:
导致裂脑的原因:
- 高可用服务器之间心跳链路故障,导致无法相互检查心跳
- 高可用服务器之间开启了防火墙,阻挡了心跳线
- 高可用服务器网卡地址信息配置故障,导致发送心跳失败
- 其他服务配置不当,如心跳方式不同/心跳广播冲突/软件BUG等
防止裂脑方案:
- 加冗余线路
- 检查到裂脑时,强行关闭心跳检查(远程关闭主节点,控制电源的电路)
- 做好脑裂的监控报警
- 报警后,备节点在接管时设置比较长的时间去接管
- 启动磁盘锁,正在服务的一方锁住磁盘,裂脑发生时,让对方完全抢不走“共享资源”
生产环境常见的LVS集群
- 通过开发上面的脚本来进行解决
- Heartbeart+lvs+ldirectord脚本配置方案
- Redhat提供的工具piranha来配置LVS
- Keepalive+lvs
LVS集群分发请求RS不均衡生产环境实战解决
- LVS自身的会话保持参数设定(-p 300,persistent 300) 优化:用cookies/sesson
- LVS调度算法设置:rr/wrr/wlc
- 后端RS节点的会话保持参数(apache的keepalive参数)
- 访问量较少的情况,不均衡的现象更加明显
- 用户发送的请求资源大小,时间长短
LVS故障排查思路
- 调度器上LVS调度规则及IP的正确性
- RS节点上VIP绑定和拟制的检查
对绑定的VIP做实时监控,出问题报警及自动处理
把绑定的vip做成配置文件/etc/sysconfig/network-scripts/lo:
- RS节点上自身提供服务检查
- 辅助排除工具有tcpdump/ping
- 负载均衡和反向代理集群的三角盘查理论
Haproxy--用户--web server
Keepalive高可用集群
用来监控LVS集群系统中各个服务节点的状态
原理:VRRP(虚拟路由器冗余协议) 主副负载均衡器切换满足VRRP协议 目的:为了解决静态路由出现的单节点故障问题,能保证网络的不间断稳定运行
安装配置keepalived
- rz keepalived-1.1.19.tar.gz
- ln -s /usr/src/kernels/2.6.32-358.el6.x86_64/ /usr/src/linux
- tar zxf keepalived-1.1.19.tar.gz
- cd keepalived-1.1.19
- ./configure
- make
- make install
- /bin/cp /usr/local/etc/rc.d/init.d/keepalived /etc/init.d/
- /bin/cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/ 配置启动脚本参数
- mkdir /etc/keepalived -p 创建默认配置文件目录
- /bin/cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/
- /bin/cp /usr/local/sbin/keepalived /usr/sbin/
- /etc/init.d/keepalived start
- ps -ef|grep keep
- man keepalived
- cat -n /etc/keepalived/keepalived.conf
单实例: M/S:
- cp keepalived.conf.dan /etc/keepalived.conf
- /etc/init.d/keepalived start
- ps -ef|grep keep
- ip add 两台都有VIP:192.168.1.29
多实例: M/S:
- cp keepalived.conf.multi /etc/keepalived.conf
- /etc/init.d/keepalived start
- ip add|grep -E "1.29|1.28" 两台都有VIP:192.168.1.29 192.168.1.28
准备把192.168.1.29-VIP进行域名解析www.etiantian.org
默认情况下keepalived只在对方机器宕机的时候接管业务,
check_myself.sh 是监控master本机,服务出现异常让slave可以接管
配置Keepalived日志
默认路劲:cat /var/log/messages
- vim /etc/sysconfig/keepalived
改写:KEEPALIVED_OPTIONS="-D -d -S 0"
- vim /etc/rsyslog.conf 加入
#keepalived
local0.* /var/log/keepalived.log
- /etc/init.d/rsyslog restart 生成keepalived.log
- /etc/init.d/keepalived restart
- cat /var/log/keepalived.log
LVS DR+Keepalived主从模式声场环境实战
Master eth0 192.168.1.21 用于wan数据转发
eth1 192.168.2.21 用于LAN数据转发
eth2 192.168.3.21 用于LB间心跳连接
VIP 192.168.1.29 解析到www.etiantian.org
Backup eth0 192.168.1.26 用于wan数据转发
eth1 192.168.2.26
eth2 192.168.3.26
RS-01 eth0 192.168.1.19
eth1 192.168.2.19
RS-02 eth0 192.168.1.27
eth1 192.168.2.27
M/B 安装好LVS+keepalived
M: 见脚本keepalived.conf.lvs-keep-m
B: 见脚本keepalived.conf.lvs-keep-s
M: 1. mv keepalived.conf.lvs-keep-m /etc/keepalived.conf
- /etc/init.d/keepalived start
- ipvsadm -Ln
B:1. mv keepalived.conf.lvs-keep-s /etc/keepalived.conf
- /etc/init.d/keepalived start
当M宕机,B自动接管
RS:1.chmod 700 rs_ctl
- /etc/init.d/rs_ctl start
- /etc/init.d/httpd start
IE登录192.168.1.29(VIP)
ipvsadm -Ln 查看后端服务分配
LVS排错
- 配置服务器间心跳连接路由 (两台LVS上)
主节点:192.168.1.21
route add -host 192.168.1.26 dev eth0 到对端
被节点:192.168.1.26
route add -host 192.168.1.21 dev eth0 到对端
写入rc.local
- 工作中如果流量巨大,一般负载均衡不能开iptables
- Keepalived用ip add查看
- Linux内核转发(DR模式不需要)
- 生产环境下的命令组合:ipvsadm -Ln --stats ipvsadm -Lnc
ipvsadm -Ln --thresholds ipvsadm -Ln --timeout