Redis集群方案redis-twemproxy-keepalived
linux /
2013年12月31日 15时14分 /
15732人浏览
redis现在的版本还不支持分布式,这会在3.0版本加上。Redis目前只支持“主从同步”和“从从同步”,所以需要根据需求应用自己来实现。
tp ------ 192.168.10.15
tp1------ 192.168.10.18
tp tp1 虚拟vip: 192.168.10.243
v1 ------ 192.168.10.224
v2 ------ 192.168.10.14
v1 v2 虚拟vip : 192.168.10.241
v3 ------ 192.168.10.16
v4 ------ 192.168.10.17
v3 v4 虚拟vip: 192.168.10.242
- 1. 先做redis 热备节点 2组
R1 与 R2 为第一组节点
V1 (192.168.10.224) V2 (192.168.10.14) 通过keepalived
主(master): V1
备份(backup):V2
虚拟ip 192.168.10.241
搭建步骤:主备相同
1. 安装redis
wget http://redis.googlecode.com/files/redis-2.0.0-rc4.tar.gz
tar zxvf redis-2.0.0-rc4.tar.gz
cd redis-2.0.0-rc4
make
cp redis.conf /etc/ 这个文件是redis启动的配置文件
cp redis-benchmark redis-cli redis-server /usr/bin/
redis-server /etc/redis.conf
redis-cli
注意修改配置:daemonize yes
2. 安装高可用软件 keepalived
wget http://www.keepalived.org/software/keepalived-1.1.20.tar.gz
tar zxvf keepalived-1.1.20.tar.gz
cd keepalived-1.1.20
./configure --with-kernel-dir=/usr/src/kernels/2.6.18-164.el5-x86_64/
make && make install
cp /usr/local/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/
cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/
mkdir /etc/keepalived
cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/
cp /usr/local/sbin/keepalived /usr/sbin
/etc/init.d/keepalived start
报错:
!!! OpenSSL is not properly installed on your system. !!!
!!! Can not include OpenSSL headers files. !!!
解决
yum install openssl-devel
报错
configure: error: Popt libraries is required
解决
yum install popt-devel
配置文件修改注意所监控网卡硬件名称
V1配置文件
Keepalived:
state MASTER
virtual_ipaddress {
192.168.10.241 ###VIP
}
redis_backup.sh :
SLAVEOF 192.168.10.14 6379
redis_master.sh:
SLAVEOF 192.168.10.14 6379
V2 配置文件
Keepalived:
state BACKUP
virtual_ipaddress {
192.168.10.241 ###VIP
}
redis_backup.sh :
SLAVEOF 192.168.10.224 6379
redis_master.sh:
SLAVEOF 192.168.10.224 6379
测试此互备节点:
1.启动Master上的Redis
redis-server /etc/redis.conf
2.启动Slave上的Redis
redis-server /etc/redis.conf
3.启动Master上的Keepalived
/etc/init.d/keepalived start
4.启动Slave上的Keepalived
/etc/init.d/keepalived start
5.尝试通过VIP连接Redis:
redis-cli -h 192.168.10.241 INFO
连接成功,Slave也连接上来了。
role:master
6.尝试插入一些数据:
redis-cli -h 192.168.10.241 SET Hellor1 Redis
OK
从VIP读取数据
redis-cli -h 192.168.10.241 GET Hellor1
"Redis"
从Master读取数据
redis-cli -h 192.168.10.224 GET Hellor1
"Redis"
从Slave读取数据
redis-cli -h 192.168.10.14 GET Hellor1
"Redis"
下面,模拟故障产生:
将Master上的Redis进程杀死:
killall -9 redis-server
查看Master上的Keepalived日志
tailf /var/log/keepalived-redis-state.log
[fault]
Mon Dec 30 10:06:13 CST 2013
同时Slave上的日志显示:
tailf /var/log/keepalived-redis-state.log
[master]
Mon Dec 30 10:06:17 CST 2013
Being master....
Run SLAVEOF cmd ...
OK
Run SLAVEOF NO ONE cmd ...
OK
然后我们可以发现,Slave已经接管服务,并且担任Master的角色了。
redis-cli -h 192.168.10.241 INFO
redis-cli -h 192.168.10.14 INFO
role:master
然后我们恢复Master的Redis进程
redis-server /etc/redis.conf
查看Master上的Keepalived日志
tailf /var/log/keepalived-redis-state.log
[master]
Mon Dec 30 10:10:13 CST 2013
Being master....
Run SLAVEOF cmd ...
OK
Run SLAVEOF NO ONE cmd ...
OK
同时Slave上的日志显示:
tailf /var/log/keepalived-redis-state.log
[backup]
Mon Dec 30 10:10:12 CST 2013
Being slave....
Run SLAVEOF cmd ...
OK
可以发现目前的Master已经再次恢复了Master的角色,故障切换以及自动恢复都成功了。
R3 与 R4 为第二组节点
参考第一组即可
V3(192.168.10.16) V4 (192.168.10.17) 通过keepalived
主(master): V3
备份(backup):V4
虚拟ip 192.168.10.242
virtual_router_id 不同
- 2. 做twemproxy 代理互备
Tp (192.168.10.15)
Tp1(192.168.10.18)
搭建步骤:
1.安装 twemproxy
yum install automake
yum install libtool
git clone git://github.com/twitter/twemproxy.git
cd twemproxy
CFLAGS="-ggdb3 -O0" autoreconf -fvi && ./configure --prefix=/usr/local/twemproxy --enable-debug=log
编译的时候报错
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force -I m4
configure.ac:8: error: Autoconf version 2.64 or higher is required
configure.ac:8: the top level
autom4te: /usr/bin/m4 failed with exit status: 63
aclocal: autom4te failed with exit status: 63
autoreconf: aclocal failed with exit status: 63
原因是autoconf版本过低
升级
查看当前版本
#rpm -qf /usr/bin/autoconf
autoconf-2.63-5.1.el6.noarch
卸载当前版本
rpm -e --nodeps autoconf-2.63
下载新版本
wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.64.tar.gz
解压安装
tar zxvf autoconf-2.64.tar.gz
cd autoconf-2.64
./configure --prefix=/usr
make && make install
查看是否安装成功
/usr/bin/autoconf -V
CFLAGS="-ggdb3 -O0" autoreconf -fvi && ./configure --prefix=/usr/local/twemproxy --enable-debug=log
make && make install
cp conf/nutcracker.yml /usr/local/twemproxy/conf/
测试
/usr/local/twemproxy/sbin/nutcracker -t
nutcracker: configuration file 'conf/nutcracker.yml' syntax is ok
--------------------------------------------------------------------------
启动命令
调试启动
/usr/local/twemproxy/sbin/nutcracker -c
以守护进程启动
/usr/local/twemproxy/sbin/nutcracker -d -c
2.安装 keepalived
参考上面安装步骤即可
修改配置文件
Tp 配置文件
Keepalived:
vrrp_script chk_t {
script "/etc/keepalived/scripts/t_check.sh" ###监控脚本
interval 2 ###监控时间
}
vrrp_instance VI_1 {
state MASTER ###设置为MASTER
interface eth1 ###监控网卡
virtual_router_id 53
priority 101 ###权重值
authentication {
auth_type PASS ###加密
auth_pass redis ###密码
}
track_interface {
eth1
}
virtual_ipaddress {
192.168.10.243 ###VIP
}
notify_master /etc/keepalived/scripts/t_master.sh
notify_backup /etc/keepalived/scripts/t_backup.sh
notify_fault /etc/keepalived/scripts/t_fault.sh
notify_stop /etc/keepalived/scripts/t_stop.sh
}
/usr/local/twemproxy/conf/nutcracker.yml
redis:
listen: 192.168.10.243:6379
hash: fnv1a_64
distribution: ketama
auto_eject_hosts: true
redis: true
server_retry_timeout: 2000
server_failure_limit: 1
servers:
- 192.168.10.241:6379:1 node1
- 192.168.10.242:6379:1 node2
Tp1 配置文件
Keepalived:
vrrp_script chk_t {
script "/etc/keepalived/scripts/t_check.sh" ###监控脚本
interval 2 ###监控时间
}
vrrp_instance VI_1 {
state BACKUP ###设置为BACKUP
interface eth1 ###监控网卡
virtual_router_id 53
priority 99 ###权重值
authentication {
auth_type PASS ###加密
auth_pass redis ###密码
}
track_interface {
eth1
}
virtual_ipaddress {
192.168.10.243 ###VIP
}
notify_master /etc/keepalived/scripts/t_master.sh
notify_backup /etc/keepalived/scripts/t_backup.sh
notify_fault /etc/keepalived/scripts/t_fault.sh
notify_stop /etc/keepalived/scripts/t_stop.sh
}
/usr/local/twemproxy/conf/nutcracker.yml
redis:
listen: 192.168.10.243:6379
hash: fnv1a_64
distribution: ketama
auto_eject_hosts: true
redis: true
server_retry_timeout: 2000
server_failure_limit: 1
servers:
- 192.168.10.241:6379:1 node1
- 192.168.10.242:6379:1 node2
测试
redis-cli -h 192.168.10.243
redis> set pp pp
OK
redis> get pp
"pp"
关掉tp 查看tp1 日志
tailf /var/log/keepalived-t-state.log
[backup]
Tue Dec 31 14:37:20 CST 2013
Being slave....
[master]
Tue Dec 31 14:47:28 CST 2013
Being master....
Run twemproxy ...
Tp2 接管 vip 继续提供服务
redis-cli -h 192.168.10.243
redis> get pp
"pp"
重启tp 查看tp1 日志
tailf /var/log/keepalived-t-state.log
[backup]
Tue Dec 31 14:54:37 CST 2013
Being slave....
redis-cli -h 192.168.10.243
redis> get pp
"pp"
Tp接管主服务 tp2切换备份服务
3.可能遇到相关问题和处理办法:
1. 如果两机器不能连通,请注意关掉防火墙:
service iptables stop #临时关闭
chkconfig iptables off #永久关闭
2. windows上编写的shell脚本,要注意通过dos2unix 命令转换,否则shell脚本可能有不可见符号不能正常执行。