手把手教你配置哨兵以及故障转移（redis一主两从）
·
一、redis 集群介绍
主从架构无法实现master和slave角色的自动切换，即当master出现redis服务异常、主机断电、磁盘损坏等问题导致master无法使用，而redis主从复制无法实现自动的故障转移(将slave 自动提升为新master)，需要手动修改环境配置,才能切换到slave redis服务器，另外当单台Redis服务器性能无法满足业务写入需求的时候,也无法横向扩展Redis服务的并行写入性能。
需要解决以上的两个核心问题：
-
master和slave角色的无缝切换，让业务无感知从而不影响业务使用；
-
可横向动态扩展Redis服务器，从而实现多台服务器并行写入以实现更高并发的目的。
Redis 集群实现方式：
-
客户端分片: 由应用决定将不同的KEY发送到不同的Redis服务器；
-
代理分片: 由代理决定将不同的KEY发送到不同的Redis服务器,代理程序如:codis,twemproxy等；
-
Redis Cluster。
二、哨兵 (Sentinel) 工作原理
2.1 sentinel 架构和故障转移
故障转移流程：
-
多个sentinel发现并确认master有问题；
-
选举出一个sentinel作为领导；
-
选就出一个slave作为master；
-
通知其余slave成为新的master的slave；
-
通知客户端主从变化；
-
等待老的master复活成为新master的slave；
Sentinel 进程是用于监控redis集群中Master主服务器工作的状，在Master主服务器发生故障的时候，可以实现Master和Slave服务器的切换，保证系统的高可用。
哨兵(Sentinel) 是一个分布式系统，可以在一个架构中运行多个哨兵(sentinel) 进程，这些进程使用流言协议(gossip protocols)来接收关于Master主服务器是否下线的信息，并使用投票协议(Agreement Protocols)来决定是否执行自动故障迁移,以及选择哪个Slave作为新的Master。
Redis Sentinel中的Sentinel节点个数应该为大于等于3且最好为奇数。
三、实现哨兵
3.1 哨兵的准备实现主从复制架构
哨兵的前提是已经实现了一个redis的主从复制的运行环境，从而实现一个一主两从基于哨兵的高可用redis架构
注意: master 的配置文件中masterauth 和slave 都必须相同
例：准备主从环境配置
主从安装配置
#在所有主从节点执行(centos7版本相对老旧，故用脚本)
#!/bin/bash
VERSION=redis-7.0.4
PASSWORD=123456
INSTALL_DIR=/etc/redis
FILE_DIR=/root
REDIS_FILE=${VERSION}.tar.gz
CPUS=`lscpu |awk '/^CPU\(s\)/{print $2}'`
color () {
RES_COL=60
MOVE_TO_COL="echo -en \\033[${RES_COL}G"
SETCOLOR_SUCCESS="echo -en \\033[1;32m"
SETCOLOR_FAILURE="echo -en \\033[1;31m"
SETCOLOR_WARNING="echo -en \\033[1;33m"
SETCOLOR_NORMAL="echo -en \E[0m"
echo -n "$1" && $MOVE_TO_COL
echo -n "["
if [ $2 = "success" -o $2 = "0" ] ;then
${SETCOLOR_SUCCESS}
echo -n $" OK "
elif [ $2 = "failure" -o $2 = "1" ] ;then
${SETCOLOR_FAILURE}
echo -n $"FAILED"
else
${SETCOLOR_WARNING}
echo -n $"WARNING"
fi
${SETCOLOR_NORMAL}
echo -n "]"
echo
}
install() {
cd ${FILE_DIR}
if [ -e ${REDIS_FILE} ];then
color "Redis 源码已存在，开始安装" 0
else
wget http://download.redis.io/releases/${VERSION}.tar.gz &> /dev/null && color "Redis 源码下载成功，开始安装" 0 || { color "Redis 源码下载失败" 1 ; exit; }
fi
yum -y install gcc jemalloc-devel systemd-devel make pcre-devel openssl-devel zlib-devel perl-ExtUtils-Embed &> /dev/null && color "依赖包安装成功" 0 || { color "依赖包安装失败" 1 ;exit; }
tar xf ${VERSION}.tar.gz
cd ${VERSION}
make -j $CPUS PREFIX=${INSTALL_DIR} install &> /dev/null && color "Redis 编译安装完成" 0 || { color "Redis 编译安装失败" 1 ;exit ; }
ln -s ${INSTALL_DIR}/bin/redis-* /usr/bin/
mkdir -p ${INSTALL_DIR}/{etc,log,data,run}
cp redis.conf ${INSTALL_DIR}/etc/
sed -i -e 's/bind 127.0.0.1/bind 0.0.0.0/' -e "/# requirepass/a requirepass $PASSWORD" -e "/^dir .*/c dir ${INSTALL_DIR}/data/" -e "/logfile .*/c logfile ${INSTALL_DIR}/log/redis-6379.log" -e "/^pidfile .*/c pidfile ${INSTALL_DIR}/run/redis_6379.pid" -e 's/^# masterauth .*/masterauth 123456/' ${INSTALL_DIR}/etc/redis.conf
if id redis &> /dev/null ;then
color "Redis 用户已存在" 1
else
useradd -r -s /sbin/nologin redis
color "Redis 用户创建成功" 0
fi
chown -R redis.redis ${INSTALL_DIR}
cat >> /etc/sysctl.conf <<EOF
net.core.somaxconn = 1024
vm.overcommit_memory = 1
EOF
sysctl -p
echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.d/rc.local
chmod +x /etc/rc.d/rc.local
cat > /usr/lib/systemd/system/redis.service <<EOF
[Unit]
Description=Redis persistent key-value database
After=network.target
[Service]
ExecStart=${INSTALL_DIR}/bin/redis-server ${INSTALL_DIR}/etc/redis.conf --supervised systemd
ExecStop=/bin/kill -s QUIT \$MAINPID
#Type=notify
User=redis
Group=redis
RuntimeDirectory=redis
RuntimeDirectoryMode=0755
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now redis &> /dev/null && color "Redis 服务启动成功,Redis信息如下:" 0 || { color "Redis 启动失败" 1 ;exit; }
sleep 2
redis-cli -a $PASSWORD INFO Server 2> /dev/null
}
install
#在所有主从节点执行 （centos8）
[root@centos8 ~]# dnf -y install redis
[root@centos8 ~]# vim /etc/redis.conf
bind 0.0.0.0
masterauth "123456"
requirepass "123456"
#或者非交互执行
[root@centos8 ~]# sed -i -e 's/bind 127.0.0.1/bind 0.0.0.0/' -e 's/^# masterauth .*/masterauth 123456/' -e 's/^# requirepass .*/requirepass 123456/' /etc/redis.conf
[root@centos8 ~]#systemctl enable --now redis
从单独配置
#在所有从节点执行
[root@centos7 ~]# echo "replicaof 10.0.0.8 6379" >> /etc/redis/etc/redis.conf
[root@centos7 ~]# systemctl restart redis
master服务器状态
[root@redis-master ~]# redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> INFO replication
# Replication
role:master
connected_slaves:2
slave0:ip=192.168.169.157,port=6379,state=online,offset=42,lag=1
slave1:ip=192.168.169.158,port=6379,state=online,offset=42,lag=1
master_failover_state:no-failover
master_replid:2909e062f15a57a7500788b351d5eb16f372b65b
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:42
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:42
注：如果配置完成slave，且重启后服务正常，查看是否防火墙拦截
配置slave1
[root@redis-slave1 ~]# redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> replicaof 192.168.169.156 6379
OK Already connected to specified master
127.0.0.1:6379> CONFIG SET masterauth "123456"
OK
127.0.0.1:6379> INFO replication
# Replication
role:slave
master_host:192.168.169.156
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_read_repl_offset:126
slave_repl_offset:126
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:ce916fe9dce4664ca3bf5f4eea1cccb6a558de57
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:126
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:126
配置slave2
[root@redis-slave2 ~]# redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> replicaof 192.168.169.156 6379
OK Already connected to specified master
127.0.0.1:6379> CONFIG SET masterauth "123456"
OK
127.0.0.1:6379> INFO replication
# Replication
role:slave
master_host:192.168.169.156
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_read_repl_offset:238
slave_repl_offset:238
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:ce916fe9dce4664ca3bf5f4eea1cccb6a558de57
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:238
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:238
3.2 编辑哨兵的配置文件
sentinel配置
Sentinel实际上是一个特殊的redis服务器,有些redis指令支持,但很多指令并不支持.默认监听在26379/tcp端口。
哨兵可以不和Redis服务器部署在一起，但一般部署在一起以节约成本。
所有redis节点使用相同的以下示例的配置文件。
#如果是编译安装，在源码目录有sentinel.conf，复制到安装目录即可，如:/etc/redis/etc/sentinel.conf
[root@centos8 ~]#vim /etc/redis/etc/redis-sentinel.conf
bind 0.0.0.0
port 26379
daemonize yes
pidfile "redis-sentinel.pid"
logfile "sentinel_26379.log"
dir "/tmp" #工作目录
sentinel monitor mymaster 10.0.0.8 6379 2
#mymaster是集群的名称，此行指定当前mymaster集群中master服务器的地址和端口
#2为法定人数限制(quorum)，即有几个sentinel认为master down了就进行故障转移，一般此值是所有sentinel节点(一般总数是>=3的 奇数,如:3,5,7等)的一半以上的整数值，比如，总数是3，即3/2=1.5，取整为2,是master的ODOWN客观下线的依据
#mymaster集群中master的密码，注意此行要在上面行的下面
sentinel auth-pass mymaster 123456
#(SDOWN)判断mymaster集群中所有节点的主观下线的时间，单位：毫秒，建议3000
sentinel down-after-milliseconds mymaster 30000
#发生故障转移后，可以同时向新master同步数据的slave的数量，数字越小总同步时间越长，但可以减轻新master的负载压力
sentinel parallel-syncs mymaster 1
#所有slaves指向新的master所需的超时时间，单位：毫秒
sentinel failover-timeout mymaster 180000
sentinel deny-scripts-reconfig yes #禁止修改脚本
logfile /var/log/redis/sentinel.log
三个哨兵配置如下
[root@redis-master ~]# cp /root/redis-7.0.4/sentinel.conf /etc/redis/etc/
# 默认二进制自带配置
[root@redis-master redis]# grep '^[a-Z]' /etc/redis/etc/redis-sentinel.conf
protected-mode no
port 26379
daemonize no
pidfile /var/run/redis-sentinel.pid
logfile ""
dir /tmp
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 30000
acllog-max-len 128
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000
sentinel deny-scripts-reconfig yes
SENTINEL resolve-hostnames no
SENTINEL announce-hostnames no
SENTINEL master-reboot-down-after-period mymaster 0
# 修改完配合
[root@redis-master ~]# grep -vE "^#|^$" /etc/redis/etc/redis-sentinel.conf
bind 0.0.0.0
protected-mode no
port 26379
daemonize no
pidfile "/var/run/redis-sentinel.pid"
logfile "/var/log/redis/redis-sentinel.log"
dir "/tmp"
sentinel monitor mymaster 10.0.0.8 6379 2
sentinel auth-pass mymaster 123456
sentinel down-after-milliseconds mymaster 3000
acllog-max-len 128
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000
sentinel deny-scripts-reconfig yes
SENTINEL resolve-hostnames no
SENTINEL announce-hostnames no
SENTINEL master-reboot-down-after-period mymaster 0
#以下内容自动生成，不需要修改
#sentinel myid此行自动生成必须唯一,修改此值需重启redis和sentinel服务
.....
# Generated by CONFIG REWRITE
latency-tracking-info-percentiles 50 99 99.9
maxclients 4064
supervised systemd
user default on nopass sanitize-payload ~* &* +@all
sentinel myid 26d748641e992d67e255f174bf75f1646bebff63
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
sentinel current-epoch 0
sentinel known-replica mymaster 10.0.0.28 6379
sentinel known-replica mymaster 10.0.0.18 6379
sentinel known-sentinel mymaster 10.0.0.18 26379 26d748641e992d67e255f174bf75f1646bebff63
sentinel known-sentinel mymaster 10.0.0.18 26379 fd911e80e4a4fdcb3bae50ac3874b80666d060b6
[root@redis-master ~]# scp /etc/redis/etc/redis-sentinel.conf redis-slave1:/etc/redis/etc/
[root@redis-master ~]# scp /etc/redis/etc/redis-sentinel.conf redis-slave2:/etc/redis/etc/
3.3 启动哨兵
#确保每个哨兵主机myid不同(编译安装可能没有)
[root@redis-master ~]# vim /etc/redis//etc/redis-sentinel.conf | grep 'sentinel myid'
#如果是编译安装在所有节点新增service文件
[root@redis-master ~]# cat /lib/systemd/system/redis-sentinel.service
[Unit]
Description=Redis Sentinel
After=network.target
[Service]
ExecStart=/etc/redis/bin/redis-sentinel /etc/redis/etc/redis-sentinel.conf --supervised systemd
ExecStop=/bin/kill -s QUIT $MAINPID
Type=notify
User=redis
Group=redis
RuntimeDirectory=redis
RuntimeDirectoryMode=0755
[Install]
WantedBy=multi-user.targetd
[root@redis-mvcaster ~]# systemctl daemon-reload
#注意所有节点的目录权限,否则无法启动服务
[root@redis-master ~]# chown -R redis.redis /etc/redis/
[root@redis-master ~]# mkdir /var/log/redis
[root@redis-master ~]# chown -R redis.redis /var/log/redis/
[root@redis-master ~]# systemctl enable --now redis-sentinel.service
3.4 验证哨兵端口
[root@redis-master ~]# ss -nlt
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 511 *:26379 *:*
LISTEN 0 511 *:6379 *:*
LISTEN 0 128 *:22 *:*
LISTEN 0 100 [::1]:25 [::]:*
LISTEN 0 511 [::1]:6379 [::]:*
LISTEN 0 128 [::]:22 [::]:*
[root@redis-slave1 ~]# ss -nlt
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 511 *:26379 *:*
LISTEN 0 511 *:6379 *:*
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 511 [::]:26379 [::]:*
LISTEN 0 511 [::1]:6379 [::]:*
LISTEN 0 128 [::]:22 [::]:*
LISTEN 0 100 [::1]:25 [::]:*\
3.5 查看哨兵日志
master哨兵日志
[root@redis-master log]# tail -f /var/log/redis/redis-sentinel.log
20542:X 10 Aug 2022 14:03:03.023 * Increased maximum number of open files to 4096 (it was originally set to 1024).
20542:X 10 Aug 2022 14:03:03.023 * monotonic clock: POSIX clock_gettime
20542:X 10 Aug 2022 14:03:03.023 * Running mode=sentinel, port=26379.
20542:X 10 Aug 2022 14:03:03.024 # Sentinel ID is 26d748641e992d67e255f174bf75f1646bebff63
20542:X 10 Aug 2022 14:03:03.026 # +monitor master mymaster 192.168.169.156 6379 quorum 2
20542:X 10 Aug 2022 14:03:21.351 * +sentinel-invalid-addr sentinel f559db87a709f6f2c861907a807827ed00b28fc5 192.168.169.158 26379 @ mymaster 192.168.169.156 6379
20542:X 10 Aug 2022 14:03:21.352 * +sentinel sentinel 26d748641e992d67e255f174bf75f1646bebff63 192.168.169.158 26379 @ mymaster 192.168.169.156 6379
20542:X 10 Aug 2022 14:03:21.377 * Sentinel new configuration saved on disk
20542:X 10 Aug 2022 14:03:21.351 * +sentinel-invalid-addr sentinel f559db87a709f6f2c861907a807827ed00b28fc5 192.168.169.158 26379 @ mymaster 192.168.169.156 6379
20542:X 10 Aug 2022 14:03:21.352 * +sentinel sentinel 26d748641e992d67e255f174bf75f1646bebff63 192.168.169.158 26379 @ mymaster 192.168.169.156 6379
20542:X 10 Aug 2022 14:03:21.377 * Sentinel new configuration saved on disk
20542:X 10 Aug 2022 14:06:40.957 # +sdown sentinel 26d748641e992d67e255f174bf75f1646bebff63 192.168.169.158 26379 @ mymaster 192.168.169.156 6379
20542:X 10 Aug 2022 14:06:41.312 # -sdown sentinel 26d748641e992d67e255f174bf75f1646bebff63 192.168.169.158 26379 @ mymaster 192.168.169.156 6379
20542:X 10 Aug 2022 14:08:43.989 * +sentinel-address-switch master mymaster 192.168.169.156 6379 ip 192.168.169.157 port 26379 for 26d748641e992d67e255f174bf75f1646bebff63
20542:X 10 Aug 2022 14:08:44.039 * Sentinel new configuration saved on disk
slave的哨兵日志
[root@redis-slave1 ~]# tail -f /var/log/redis/redis-sentinel.log
20297:X 10 Aug 2022 14:03:03.047 # systemd supervision error: NOTIFY_SOCKET not found!
20297:X 10 Aug 2022 14:03:03.047 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
20297:X 10 Aug 2022 14:03:03.047 # Redis version=7.0.4, bits=64, commit=00000000, modified=0, pid=20297, just started
20297:X 10 Aug 2022 14:03:03.047 # Configuration loaded
20297:X 10 Aug 2022 14:03:03.051 * Increased maximum number of open files to 4096 (it was originally set to 1024).
20297:X 10 Aug 2022 14:03:03.051 * monotonic clock: POSIX clock_gettime
20297:X 10 Aug 2022 14:03:03.052 * Running mode=sentinel, port=26379.
20297:X 10 Aug 2022 14:03:03.052 # Sentinel ID is 26d748641e992d67e255f174bf75f1646bebff63
20297:X 10 Aug 2022 14:03:03.052 # +monitor master mymaster 192.168.169.156 6379 quorum 2
20297:X 10 Aug 2022 14:03:05.046 * +sentinel sentinel 26d748641e992d67e255f174bf75f1646bebff63 192.168.169.156 26379 @ mymaster 192.168.169.156 6379
20297:X 10 Aug 2022 14:03:05.106 * Sentinel new configuration saved on disk
20297:X 10 Aug 2022 14:03:05.106 * +sentinel-address-switch master mymaster 192.168.169.156 6379 ip 192.168.169.158 port 26379 for 26d748641e992d67e255f174bf75f1646bebff63
20297:X 10 Aug 2022 14:03:05.123 * Sentinel new configuration saved on disk
20297:X 10 Aug 2022 14:03:05.123 * +sentinel-address-switch master mymaster 192.168.169.156 6379 ip 192.168.169.157 port 26379 for 26d748641e992d67e255f174bf75f1646bebff63
20297:X 10 Aug 2022 14:03:05.127 * Sentinel new configuration saved on disk
20297:X 10 Aug 2022 14:03:07.147 * +sentinel-address-switch master mymaster 192.168.169.156 6379 ip 192.168.169.156 port 26379 for 26d748641e992d67e255f174bf75f1646bebff63
3.6 当前sentinel状态
在sentinel状态中尤其是最后一行，涉及到masterIP是多少，有几个slave，有几个sentinels，必须是符合全部服务器数量
[root@redis-master ~]# redis-cli -p 26379
127.0.0.1:26379> INFO sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_tilt_since_seconds:-1
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=10.0.0.8:6379,slaves=2,sentinels=3
#两个slave,三个sentinel服务器,如果sentinels值不符合,检查myid可能冲突
3.7 停止Redis Master 节点测试故障转移
[root@redis-master ~]# killall redis-server
查看各节点上哨兵信息：
[root@redis-master ~]# redis-cli -p 26379
127.0.0.1:26379> INFO sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_tilt_since_seconds:-1
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.169.156:6379,slaves=2,sentinels=3
故障转移时sentinel的信息：
20603:X 10 Aug 2022 14:27:02.006 # +sdown master mymaster 192.168.169.156 6379
20603:X 10 Aug 2022 14:27:03.114 # +odown master mymaster 192.168.169.156 6379 #quorum 3/2
20603:X 10 Aug 2022 14:27:03.114 # +new-epoch 1
20603:X 10 Aug 2022 14:27:03.114 # +try-failover master mymaster 192.168.169.156 6379
20603:X 10 Aug 2022 14:27:03.151 * Sentinel new configuration saved on disk
20603:X 10 Aug 2022 14:27:03.151 # +vote-for-leader 26d748641e992d67e255f174bf75f1646bebff63 1
20603:X 10 Aug 2022 14:27:03.152 # fd911e80e4a4fdcb3bae50ac3874b80666d060b6 voted for 26d748641e992d67e255f174bf75f1646bebff63 1
20603:X 10 Aug 2022 14:27:03.152 # 26d748641e992d67e255f174bf75f1646bebff63 voted for 26d748641e992d67e255f174bf75f1646bebff63 1
20603:X 10 Aug 2022 14:27:03.214 # +elected-leader master mymaster 192.168.169.156 6379
20603:X 10 Aug 2022 14:27:03.214 # +failover-state-select-slave master mymaster 192.168.169.156 6379
20603:X 10 Aug 2022 14:27:03.316 # +selected-slave slave 192.168.169.158:6379 192.168.169.158 6379 @ mymaster 192.168.169.156 6379
20603:X 10 Aug 2022 14:27:03.316 * +failover-state-send-slaveof-noone slave 192.168.169.158:6379 192.168.169.158 6379 @ mymaster 192.168.169.156 6379
20603:X 10 Aug 2022 14:27:03.375 * +failover-state-wait-promotion slave 192.168.169.158:6379 192.168.169.158 6379 @ mymaster 192.168.169.156 6379
20603:X 10 Aug 2022 14:27:04.268 # -odown master mymaster 192.168.169.156 6379
20603:X 10 Aug 2022 14:27:04.406 * Sentinel new configuration saved on disk
20603:X 10 Aug 2022 14:27:04.406 # +promoted-slave slave 192.168.169.158:6379 192.168.169.158 6379 @ mymaster 192.168.169.156 6379
20603:X 10 Aug 2022 14:27:04.406 # +failover-state-reconf-slaves master mymaster 192.168.169.156 6379
20603:X 10 Aug 2022 14:27:04.407 * +slave-reconf-sent slave 192.168.169.157:6379 192.168.169.157 6379 @ mymaster 192.168.169.156 6379
20603:X 10 Aug 2022 14:27:04.776 * +slave-reconf-inprog slave 192.168.169.157:6379 192.168.169.157 6379 @ mymaster 192.168.169.156 6379
20603:X 10 Aug 2022 14: 27:08.930 * +slave-reconf-done slave 192.168.169.157:6379 192.168.169.157 6379 @ mymaster 192.168.169.156 6379
20603:X 10 Aug 2022 14:27:08.959 # +failover-end master mymaster 192.168.169.156 6379
20603:X 10 Aug 2022 14:27:08.959 # +switch-master mymaster 192.168.169.156 6379 192.168.169.158 6379
20603:X 10 Aug 2022 14:27:08.959 * +slave slave 192.168.169.157:6379 192.168.169.157 6379 @ mymaster 192.168.169.158 6379
20603:X 10 Aug 2022 14:27:08.959 * +slave slave 192.168.169.156:6379 192.168.169.156 6379 @ mymaster 192.168.169.158 6379
20603:X 10 Aug 2022 14:27:08.981 * Sentinel new configuration saved on disk
20603:X 10 Aug 2022 14:27:11.969 # +sdown slave 192.168.169.156:6379 192.168.169.156 6379 @ mymaster 192.168.169.158 6379
3.8 故障转移后的redis配置文件会被自动修改
故障转移后redis.conf中的replicaof行的master IP会被修改
[root@redis-slave1 ~]# grep replicaof /etc/redis/etc/redis.conf
# Master-Replica replication. Use replicaof to make a Redis instance a copy of
# replicaof <masterip> <masterport>
replicaof 192.168.169.158 6379
哨兵配置文件的sentinel monitor IP 同样也会被修改
[root@redis-master etc]# grep "^[a-Z]" /etc/redis/etc/redis-sentinel.conf
bind 0.0.0.0
protected-mode no
port 26379
daemonize no
pidfile "/var/run/redis-sentinel.pid"
logfile "/var/log/redis/redis-sentinel.log"
dir "/tmp"
sentinel monitor mymaster 192.168.169.158 6379 2
sentinel auth-pass mymaster 123456
sentinel down-after-milliseconds mymaster 3000
acllog-max-len 128
sentinel deny-scripts-reconfig yes
sentinel resolve-hostnames no
sentinel announce-hostnames no
latency-tracking-info-percentiles 50 99 99.9
maxclients 4064
supervised systemd
user default on nopass sanitize-payload ~* &* +@all
sentinel myid 26d748641e992d67e255f174bf75f1646bebff63
sentinel config-epoch mymaster 1
sentinel leader-epoch mymaster 1
sentinel current-epoch 1
sentinel known-replica mymaster 192.168.169.156 6379
sentinel known-replica mymaster 192.168.169.157 6379
sentinel known-sentinel mymaster 192.168.169.157 26379 fd911e80e4a4fdcb3bae50ac3874b80666d060b6
sentinel known-sentinel mymaster 192.168.169.157 26379 26d748641e992d67e255f174bf75f1646bebff63
[root@redis-slave2 ~]# grep "^[a-Z]" /etc/redis/etc/redis-sentinel.conf
bind 0.0.0.0
protected-mode no
port 26379
daemonize no
pidfile "/var/run/redis-sentinel.pid"
logfile "/var/log/redis/redis-sentinel.log"
dir "/tmp"
sentinel monitor mymaster 192.168.169.158 6379 2
sentinel auth-pass mymaster 123456
sentinel down-after-milliseconds mymaster 3000
acllog-max-len 128
sentinel deny-scripts-reconfig yes
sentinel resolve-hostnames no
sentinel announce-hostnames no
latency-tracking-info-percentiles 50 99 99.9
maxclients 4064
supervised systemd
user default on nopass sanitize-payload ~* &* +@all
sentinel myid 26d748641e992d67e255f174bf75f1646bebff63
sentinel config-epoch mymaster 1
sentinel leader-epoch mymaster 1
sentinel current-epoch 1
sentinel known-replica mymaster 192.168.169.156 6379
sentinel known-replica mymaster 192.168.169.157 6379
sentinel known-sentinel mymaster 192.168.169.158 26379 26d748641e992d67e255f174bf75f1646bebff63
sentinel known-sentinel mymaster 192.168.169.157 26379 fd911e80e4a4fdcb3bae50ac3874b80666d060b6
sentinel known-sentinel mymaster 192.168.169.158 26379 f559db87a709f6f2c861907a807827ed00b28fc5
3.9 当前redis状态
新的master状态
[root@redis-slave2 ~]# redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> INFO replication
# Replication
role:master #提升为master
connected_slaves:1
slave0:ip=192.168.169.157,port=6379,state=online,offset=648778,lag=1
master_failover_state:no-failover
master_replid:078ac6ddb95a69113734e9c4bd21a1b69ffee7e1
master_replid2:ce916fe9dce4664ca3bf5f4eea1cccb6a558de57
master_repl_offset:648923
second_repl_offset:498127
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:648923
另一个slave指向新的master
[root@redis-slave1 ~]# redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> INFO replication
# Replication
role:slave
master_host:192.168.169.158
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_read_repl_offset:659274
slave_repl_offset:659274
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:078ac6ddb95a69113734e9c4bd21a1b69ffee7e1
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:659274
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:499469
repl_backlog_histlen:159806
3.10 恢复故障的原master重新加入redis集群
[root@redis-master ~]# systemctl start redis
#sentinel会自动修改下面行指向新的master
[root@redis-master ~]# grep replicaof /etc/redis/etc/redis.conf
# Master-Replica replication. Use replicaof to make a Redis instance a copy of
# replicaof <masterip> <masterport>
replicaof 192.168.169.158 6379
在原 master上观察状态
[root@redis-master ~]# redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> INFO replication
# Replication
role:slave
master_host:192.168.169.158
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_read_repl_offset:708339
slave_repl_offset:708339
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:078ac6ddb95a69113734e9c4bd21a1b69ffee7e1
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:708339
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:685253
repl_backlog_histlen:23087
[root@redis-master ~]# redis-cli -p 26379
127.0.0.1:26379> INFO sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_tilt_since_seconds:-1
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.169.158:6379,slaves=2,sentinels=3
观察新master上状态和日志
[root@redis-slave2 ~]# redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> INFO replication
# Replication
role:master
connected_slaves:2
slave0:ip=192.168.169.157,port=6379,state=online,offset=735926,lag=1
slave1:ip=192.168.169.156,port=6379,state=online,offset=735926,lag=0
master_failover_state:no-failover
master_replid:078ac6ddb95a69113734e9c4bd21a1b69ffee7e1
master_replid2:ce916fe9dce4664ca3bf5f4eea1cccb6a558de57
master_repl_offset:735926
second_repl_offset:498127
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:735926
[root@redis-slave2 ~]# tail -f /var/log/redis/redis-sentinel.log
20603:X 10 Aug 2022 14:45:13.032 # -sdown sentinel fd911e80e4a4fdcb3bae50ac3874b80666d060b6 192.168.169.157 26379 @ mymaster 192.168.169.158 6379
20603:X 10 Aug 2022 14:45:19.845 # +sdown sentinel fd911e80e4a4fdcb3bae50ac3874b80666d060b6 192.168.169.157 26379 @ mymaster 192.168.169.158 6379
3.11 sentinel运维
手动让主节点下线
sentinel failover <masterName>
范例: 手动故障转移
[root@centos8 ~]#vim /etc/redis.conf
replica-priority 10 #指定优先级,值越小sentinel会优先将之选为新的master,默为值为100
[root@centos8 ~]#redis-cli -p 26379
127.0.0.1:26379> sentinel failover mymaster
OK
3.12 应用程序如何连接 redis
Redis 官方客户端：https://redis.io/clients
3.12.1 客户端连接 sentinel 工作原理
客户端获取sentinel节点集合,选举出一个sentinel；
由这个sentinel 通过masterName 获取master节点信息,客户端通过sentinel get-master-addr-byname master-name这个api来获取对应主节点信息；
客户端发送role指令确认mater的信息,验证当前获取的“主节点”是真正的主节点，这样的目的是为了防止故障转移期间主节点的变化；
客户端保持和sentinel节点集合的联系，即订阅sentinel节点相关频道，时刻获取关于主节点的相关信息,获取新的master 信息变化,并自动连接新的master
3.12.2 java 连接Sentinel哨兵
java 客户端连接Redis：https://github.com/xetorthio/jedis/blob/master/pom.xml
#jedis/pom.xml 配置连接redis
<properties>
<redishosts>localhost:6379,localhost:6380,localhost:6381,localhost:6382,localhost:6383,localhost:6384,localhost:6385</redis-hosts>
<sentinelhosts>localhost:26379,localhost:26380,localhost:26381</sentinel-hosts>
<clusterhosts>localhost:7379,localhost:7380,localhost:7381,localhost:7382,localhost:7383,localhost:7384,localhost:7385</cluster-hosts>
<github.global.server>github</github.global.server>
</propertis>
java客户端连接单机的redis是通过Jedis来实现的，java代码用的时候只要创建Jedis对象就可以建多个Jedis连接池来连接redis，应用程序再直接调用连接池即可连接Redis。而Redis为了保障高可用,服务一般都是Sentinel部署方式，当Redis服务中的主服务挂掉之后,会仲裁出另外一台Slaves服务充当Master。这个时候,我们的应用即使使用了Jedis 连接池,如果Master服务挂了,应用将还是无法连接新的Master服务，为了解决这个问题, Jedis也提供了相应的Sentinel实现,能够在Redis Sentinel主从切换时候,通知应用,把应用连接到新的Master服务。
Redis Sentinel的使用也是十分简单的,只是在JedisPool中添加了Sentinel和MasterName参数，JRedisSentinel底层基于Redis订阅实现Redis主从服务的切换通知，当Reids发生主从切换时，Sentinel会发送通知主动通知Jedis进行连接的切换，JedisSentinelPool在每次从连接池中获取链接对象的时候,都要对连接对象进行检测,如果此链接和Sentinel的Master服务连接参数不一致,则会关闭此连接,重新获取新的Jedis连接对象
3.12.3 python 连接Sentinel哨兵
[root@centos8 ~]#yum -y install python3 python3-redis
[root@centos8 ~]#cat sentinel_test.py
#!/usr/bin/python3
import redis
from redis.sentinel import Sentinel
#连接哨兵服务器(主机名也可以用域名)
sentinel = Sentinel([('10.0.0.8', 26379),
('10.0.0.18', 26379),
('10.0.0.28', 26379)],
socket_timeout=0.5)
redis_auth_pass = '123456'
#mymaster 是配置哨兵模式的redis集群名称，此为默认值,实际名称按照个人部署案例来填写
#获取主服务器地址
master = sentinel.discover_master('mymaster')
print(master)
#获取从服务器地址
slave = sentinel.discover_slaves('mymaster')
print(slave)
#获取主服务器进行写入
master = sentinel.master_for('mymaster', socket_timeout=0.5,password=redis_auth_pass, db=0)
w_ret = master.set('name', 'wang')
#输出：True
#获取从服务器进行读取（默认是round-roubin）
slave = sentinel.slave_for('mymaster', socket_timeout=0.5,password=redis_auth_pass, db=0)
r_ret = slave.get('name')
print(r_ret)
#输出：wang
[root@centos8 ~]#chmod +x sentinel_test.py
[root@centos8 ~]#./sentinel_test.py
('10.0.0.8', 6379)
[('10.0.0.18', 6379), ('10.0.0.28', 6379)]
b'wang'
更多推荐
9
0- 0
已为社区贡献9条内容
所有评论(0)