1 Executing secondary network check script
2 Executing SSH check script: save_binary_logs --command=test -
3Monitoring server xxx is reachable, Master is not reachable from xxx. OK
4探测3次 Master is not reachable from health checker!
5 读取配置文件,找server信息
6 检查存活的slave,readonly relay_log_purge filter等信息
7 Terminating monitoring script.
8 开始failover(failover之后会执行脚本,mha server会关闭,每次关闭后都需要在workdir下删除之前failvoer文件etc/mha/workdir/3334/3334.failover.error,正常切换后,其他从库会自动指向新主库)
Phase 1: Configuration Check Phase completed.
Phase 2: Dead Master Shutdown Phase..
Forcing shutdown so that applications never connect to the current master..
Tue Dec 17 20:24:14 2019 - [info] Executing master IP deactivation script:
Tue Dec 17 20:24:14 2019 - [info] /etc/mha/scripts/3334/master_ip_failover.sh --orig_master_host=10.xxx --orig_master_ip=10.xxxx--orig_master_port=3334 --command=stopssh --ssh_user=root
2019-12-17 20:24:14 master_ip_failover.sh [info]: -------start stop or stopssh
Executing SHUTDOWN script:
Tue Dec 17 20:24:14 2019 - [info] /etc/mha/scripts/3334/power_manager.sh --command=stopssh --ssh_user=root --host=10.xx --ip=10.2xx --port=3334 --pid_file=/data00/mysql_3334/mysqld.pid
-------------power off script do not power off machine-------------
Tue Dec 17 20:24:14 2019 - [info] Power off done.
Tue Dec 17 20:24:15 2019 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Tue Dec 17 20:24:15 2019 - [info]
Tue Dec 17 20:24:15 2019 - [info] * Phase 3: Master Recovery Phase..
Tue Dec 17 20:24:15 2019 - [info]
Tue Dec 17 20:24:15 2019 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Tue Dec 17 20:24:15 2019 - [info]
Tue Dec 17 20:24:15 2019 - [info] The latest binary log file/position on all slaves is mysql-bin.000003:154
Tue Dec 17 20:24:15 2019 - [info] Latest slaves (Slaves that received relay log files to the latest):
Tue Dec 17 20:24:15 2019 - [info] 10.2xx(10.xxx:3334) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled
Tue Dec 17 20:24:15 2019 - [info] Replicating from 10.xx(1xx:3334)
Tue Dec 17 20:24:15 2019 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Dec 17 20:24:15 2019 - [info] 10.2xxx(10.xxx:3334) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled
Tue Dec 17 20:24:15 2019 - [info] Replicating from 10.23xx(10.xx:3334)
Tue Dec 17 20:24:15 2019 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Dec 17 20:24:15 2019 - [info] The oldest binary log file/position on all slaves is mysql-bin.000003:154
Tue Dec 17 20:24:15 2019 - [info] Oldest slaves:
Tue Dec 17 20:24:15 2019 - [info] xxx(xx:3334) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled
Tue Dec 17 20:24:15 2019 - [info] Replicating from 10.xx(10.xx:3334)
Tue Dec 17 20:24:15 2019 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Dec 17 20:24:15 2019 - [info] 1xxx(1xxx:3334) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled
Tue Dec 17 20:24:15 2019 - [info] Replicating from 10.2xxx(10.2xxx:3334)
Tue Dec 17 20:24:15 2019 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Dec 17 20:24:15 2019 - [info]
Tue Dec 17 20:24:15 2019 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..
Tue Dec 17 20:24:15 2019 - [info]
Tue Dec 17 20:24:15 2019 - [warning] Dead Master is not SSH reachable. Could not save it's binlogs. Transactions that were not sent to the latest slave (Read_Master_Log_Pos to the tail of the dead master's binlog) were lost.
Tue Dec 17 20:24:15 2019 - [info]
Tue Dec 17 20:24:15 2019 - [info] * Phase 3.3: Determining New Master Phase..
Tue Dec 17 20:24:15 2019 - [info]
Tue Dec 17 20:24:15 2019 - [info] Finding the latest slave that has all relay logs for recovering other slaves..
Tue Dec 17 20:24:15 2019 - [info] All slaves received relay logs to the same position. No need to resync each other.
Tue Dec 17 20:24:15 2019 - [info] Searching new master from slaves..
Tue Dec 17 20:24:15 2019 - [info] Candidate masters from the configuration file:
Tue Dec 17 20:24:15 2019 - [info] xx(xxx:3334) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled
Tue Dec 17 20:24:15 2019 - [info] Replicating from 10.xxx(10.xx:3334)
Tue Dec 17 20:24:15 2019 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Dec 17 20:24:15 2019 - [info] 1xxx(1xxx:3334) Version=5.7.23-log (oldest major version between slaves) log-bin:enabled
Tue Dec 17 20:24:15 2019 - [info] Replicating from 10.xxx(10.xxx:3334)
Tue Dec 17 20:24:15 2019 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Dec 17 20:24:15 2019 - [info] Non-candidate masters:
Tue Dec 17 20:24:15 2019 - [info] Searching from candidate_master slaves which have received the latest relay log events..
Tue Dec 17 20:24:15 2019 - [info] New master is 10xxx2(10.xx:3334)
Tue Dec 17 20:24:15 2019 - [info] Starting master failover..
Tue Dec 17 20:24:15 2019 - [info] * Phase 3.4: New Master Diff Log Generation Phase..
Tue Dec 17 20:24:15 2019 - [info]
Tue Dec 17 20:24:15 2019 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Tue Dec 17 20:24:15 2019 - [info]
Tue Dec 17 20:24:15 2019 - [info] * Phase 3.5: Master Log Apply Phase..
Tue Dec 17 20:24:15 2019 - [info]
Tue Dec 17 20:24:15 2019 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Tue Dec 17 20:24:15 2019 - [info] Starting recovery on 10.xxx(xxx:3334)..
Tue Dec 17 20:24:15 2019 - [info] This server has all relay logs. Waiting all logs to be applied..
Tue Dec 17 20:24:15 2019 - [info] done.
Tue Dec 17 20:24:15 2019 - [info] All relay logs were successfully applied.
Tue Dec 17 20:24:15 2019 - [info] Getting new master's binlog name and position..
Tue Dec 17 20:24:15 2019 - [info] mysql-bin.000002:1564
Tue Dec 17 20:24:15 2019 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.xxx', MASTER_PORT=3334, MASTER_LOG_FILE='mysql-bin.000002', MASTER_LOG_POS=1564, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Tue Dec 17 20:24:15 2019 - [info] Executing master IP activate script:
Tue Dec 17 20:24:15 2019 - [info] /etc/mha/scripts/3334/master_ip_failover.sh --command=start --ssh_user=root --orig_master_host=10.xx --orig_master_ip=10.2xxx --orig_master_port=3334 --new_master_host=10.xxx --new_master_ip=10.xxx --new_master_port=3334 --new_master_user='mha' --new_master_password=xxx
2019-12-17 20:24:15 master_ip_failover.sh [info]: new master ip----->10xxx
2019-12-17 20:24:15 master_ip_failover.sh [iofo]: new master port---->3334
mysql_cmd---->/usr/local/mysql-5.7.10/bin/mysql -umha -ppassword -h10.xxx -P3334
mysql_cmd_meta--->/usr/local/mysql-5.7.10/bin/mysql -u mha -pxxx -h 10.xxx
2019-12-17 20:24:15 master_ip_failover.sh [info]: set new master read_only=0
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1193 (HY000) at line 1: Unknown system variable 'rpl_semi_sync_slave_enabled'
2019-12-17 20:24:15 master_ip_failover.sh [error]: mysql set readonly=0 on new master fail
Tue Dec 17 20:24:15 2019 - [error][/root/perl5/lib/perl5/MHA/MasterFailover.pm, ln1612] Failed to activate master IP address for xx(1xx:3334) with return code 1:0
Tue Dec 17 20:24:15 2019 - [error][/root/perl5/lib/perl5/MHA/ManagerUtil.pm, ln177] Got ERROR: at /root/perl5/bin/masterha_manager line 65.
Tue Dec 17 20:24:15 2019 - [info]
----- Failover Report -----
3334: MySQL Master failover xxx(xx:3334)
Master 1xx(xxx:3334) is down!
Check MHA Manager logs at n230-036-009.novalocal:/etc/mha/workdir/3334/3334.log for details.
Started automated(non-interactive) failover.
Invalidated master IP address on 1xx(10.xx:3334)
Power off 10.xxx.
The latest slave 10.2xx(10.xx:3334) has all relay logs for recovery.
Selected 10xxx(10.xxx:3334) as a new master.
10.xxx(10.xx:3334): OK: Applying all logs succeeded.
Failed to activate master IP address for 10.2xx(10.xxx:3334) with return code 1:0
Got Error so couldn't continue failover from here.
Tue Dec 17 20:24:15 2019 - [info] Sending mail..
orig_master_host====1xx
new_mster_host=====
----body--->Master 1xx(10.xx:3334) is down! Check MHA Manager logs at n230-036-009.novalocal:/etc/mha/workdir/3334/3334.log for details. Started automated(non-interactive) failover. Invalidated master IP address on 10.xx(10.xx:3334) Power off 1xx. The latest slave 10.23xx(10.xx:3334) has all relay logs for recovery. Selected 10.2xxx(10.2xx:3334) as a new master. 10.2xx(10.xx:3334): OK: Applying all logs succeeded. Failed to activate master IP address for 10.2xx(10.xx:3334) with return code 1:0 Got Error so couldn't continue failover from here.
2019-12-17 20:24:15 send_master_failover_mail.sh [info]: ------failover success-----
2019-12-17 20:24:15 send_master_failover_mail.sh [info]: Master 10.23xx(10.xx:3334) is down!
Check MHA Manager logs at n230-036-009.novalocal:/etc/mha/workdir/3334/3334.log for details.
Started automated(non-interactive) failover.
Invalidated master IP address on 10.2xx(10.2xx:3334)
Power off 10.xx.
The latest slave 10.xx(10.xx:3334) has all relay logs for recovery.
Selected 10.xx(10xx:3334) as a new master.
10.2xx(10.2xx:3334): OK: Applying all logs succeeded.
Failed to activate master IP address for 10.2xx(10.2xx:3334) with return code 1:0
Got Error so couldn't continue failover from here.
cat: /mha/workdir/3334/3334.log: No such file or directory
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 93 100 17 100 76 72 324 --:--:-- --:--:-- --:--:-- 326
mha failover过程
猜你喜欢
转载自blog.csdn.net/aoerqileng/article/details/103587114
今日推荐
周排行