Redis集群方案redis-twemproxy-keepalived
        
            linux / 
2013年12月31日 15时14分 / 
16347人浏览
 
         
        
            redis现在的版本还不支持分布式,这会在3.0版本加上。Redis目前只支持“主从同步”和“从从同步”,所以需要根据需求应用自己来实现。
 tp ------ 192.168.10.15
tp1------ 192.168.10.18
tp tp1 虚拟vip: 192.168.10.243
 
v1 ------ 192.168.10.224
v2 ------ 192.168.10.14
v1 v2 虚拟vip : 192.168.10.241
 
v3 ------ 192.168.10.16
v4 ------ 192.168.10.17
v3 v4 虚拟vip: 192.168.10.242
 
tp ------ 192.168.10.15
tp1------ 192.168.10.18
tp tp1 虚拟vip: 192.168.10.243
 
v1 ------ 192.168.10.224
v2 ------ 192.168.10.14
v1 v2 虚拟vip : 192.168.10.241
 
v3 ------ 192.168.10.16
v4 ------ 192.168.10.17
v3 v4 虚拟vip: 192.168.10.242
 
 
	- 1.       先做redis 热备节点 2组
R1 与 R2 为第一组节点
V1 (192.168.10.224)  V2 (192.168.10.14) 通过keepalived 
 
主(master):  V1
备份(backup):V2
虚拟ip  192.168.10.241
 
 
搭建步骤:主备相同
      1. 安装redis
    
    wget http://redis.googlecode.com/files/redis-2.0.0-rc4.tar.gz
tar zxvf redis-2.0.0-rc4.tar.gz
cd redis-2.0.0-rc4
make
cp redis.conf /etc/ 这个文件是redis启动的配置文件
cp redis-benchmark redis-cli redis-server /usr/bin/
 
redis-server /etc/redis.conf
redis-cli
注意修改配置:daemonize yes  
 
      2. 安装高可用软件 keepalived 
 
wget http://www.keepalived.org/software/keepalived-1.1.20.tar.gz
tar zxvf keepalived-1.1.20.tar.gz
cd keepalived-1.1.20
./configure --with-kernel-dir=/usr/src/kernels/2.6.18-164.el5-x86_64/
make && make install
 
cp /usr/local/etc/rc.d/init.d/keepalived  /etc/rc.d/init.d/
cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/
mkdir /etc/keepalived
cp /usr/local/etc/keepalived/keepalived.conf  /etc/keepalived/
cp /usr/local/sbin/keepalived  /usr/sbin
/etc/init.d/keepalived start
 
 
报错:
!!! OpenSSL is not properly installed on your system. !!!
!!! Can not include OpenSSL headers files. !!!
解决
yum install openssl-devel
 
报错
configure: error: Popt libraries is required
解决
yum install popt-devel
 
 
 
配置文件修改注意所监控网卡硬件名称
 
V1配置文件
Keepalived:
 
state MASTER      
 
virtual_ipaddress {
             192.168.10.241  ###VIP
        }
 
 
redis_backup.sh :
 
SLAVEOF 192.168.10.14 6379
 
redis_master.sh:
 
SLAVEOF 192.168.10.14 6379
 
 
V2 配置文件
 
Keepalived:
 
 
state BACKUP      
 
virtual_ipaddress {
             192.168.10.241  ###VIP
        }
 
redis_backup.sh :
 
SLAVEOF 192.168.10.224  6379
 
redis_master.sh:
 
SLAVEOF 192.168.10.224 6379
 
 
测试此互备节点:
 
1.启动Master上的Redis
redis-server /etc/redis.conf
2.启动Slave上的Redis
redis-server /etc/redis.conf
3.启动Master上的Keepalived
/etc/init.d/keepalived start
4.启动Slave上的Keepalived
  /etc/init.d/keepalived start
5.尝试通过VIP连接Redis:
redis-cli -h 192.168.10.241 INFO
连接成功,Slave也连接上来了。
role:master
6.尝试插入一些数据:
redis-cli -h 192.168.10.241  SET Hellor1 Redis
OK
从VIP读取数据
redis-cli -h 192.168.10.241  GET Hellor1
"Redis"
从Master读取数据
redis-cli -h 192.168.10.224 GET Hellor1
"Redis"
从Slave读取数据
  redis-cli -h 192.168.10.14 GET Hellor1
"Redis"
 
 
下面,模拟故障产生:
 
将Master上的Redis进程杀死:
killall -9 redis-server
查看Master上的Keepalived日志
tailf /var/log/keepalived-redis-state.log
[fault]
Mon Dec 30 10:06:13 CST 2013
同时Slave上的日志显示:
tailf /var/log/keepalived-redis-state.log
[master]
Mon Dec 30 10:06:17 CST 2013
Being master....
Run SLAVEOF cmd ...
OK
Run SLAVEOF NO ONE cmd ...
OK 
然后我们可以发现,Slave已经接管服务,并且担任Master的角色了。
redis-cli -h 192.168.10.241 INFO
redis-cli -h 192.168.10.14 INFO
role:master
然后我们恢复Master的Redis进程
redis-server /etc/redis.conf
查看Master上的Keepalived日志
tailf /var/log/keepalived-redis-state.log
[master]
Mon Dec 30 10:10:13 CST 2013
Being master....
Run SLAVEOF cmd ...
OK
Run SLAVEOF NO ONE cmd ...
OK
同时Slave上的日志显示:
tailf /var/log/keepalived-redis-state.log
[backup]
Mon Dec 30 10:10:12 CST 2013
Being slave....
Run SLAVEOF cmd ...
OK
可以发现目前的Master已经再次恢复了Master的角色,故障切换以及自动恢复都成功了。
 
 
R3 与 R4 为第二组节点
参考第一组即可
 
V3(192.168.10.16)  V4 (192.168.10.17) 通过keepalived 
 
主(master):  V3
备份(backup):V4
虚拟ip  192.168.10.242
virtual_router_id 不同
	- 2.       做twemproxy 代理互备
Tp (192.168.10.15) 
Tp1(192.168.10.18)
 
搭建步骤:
 
1.安装 twemproxy
 
 
yum install automake
yum install libtool
git clone git://github.com/twitter/twemproxy.git
cd twemproxy
 
CFLAGS="-ggdb3 -O0" autoreconf -fvi && ./configure --prefix=/usr/local/twemproxy  --enable-debug=log 
 
编译的时候报错
 
autoreconf: Entering directory `.'  
autoreconf: configure.ac: not using Gettext  
autoreconf: running: aclocal --force -I m4  
configure.ac:8: error: Autoconf version 2.64 or higher is required  
configure.ac:8: the top level  
autom4te: /usr/bin/m4 failed with exit status: 63  
aclocal: autom4te failed with exit status: 63  
autoreconf: aclocal failed with exit status: 63  
 
原因是autoconf版本过低
 
升级
查看当前版本
#rpm -qf /usr/bin/autoconf   
autoconf-2.63-5.1.el6.noarch
 
卸载当前版本
rpm -e --nodeps autoconf-2.63    
 
下载新版本
wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.64.tar.gz 
 
 
解压安装
tar zxvf autoconf-2.64.tar.gz  
cd autoconf-2.64  
./configure --prefix=/usr  
make && make install  
 
查看是否安装成功
/usr/bin/autoconf -V
 
 
CFLAGS="-ggdb3 -O0" autoreconf -fvi && ./configure --prefix=/usr/local/twemproxy  --enable-debug=log 
make && make install  
 
cp conf/nutcracker.yml /usr/local/twemproxy/conf/  
 
测试
/usr/local/twemproxy/sbin/nutcracker -t 
nutcracker: configuration file 'conf/nutcracker.yml' syntax is ok
 
--------------------------------------------------------------------------
启动命令
调试启动
 
/usr/local/twemproxy/sbin/nutcracker -c 
 
以守护进程启动
/usr/local/twemproxy/sbin/nutcracker -d -c 
 
 
2.安装 keepalived
参考上面安装步骤即可
 
 
修改配置文件
 
Tp 配置文件
 
Keepalived:
 
vrrp_script chk_t {
                script "/etc/keepalived/scripts/t_check.sh"   ###监控脚本
                interval 2                                        ###监控时间
}
vrrp_instance VI_1 {
        state MASTER                            ###设置为MASTER
        interface eth1                          ###监控网卡
        virtual_router_id 53
        priority 101                            ###权重值
        authentication {
                     auth_type PASS             ###加密
                     auth_pass redis            ###密码
        }
        track_interface {
            eth1
        }
        virtual_ipaddress {
             192.168.10.243                         ###VIP
        }
        notify_master /etc/keepalived/scripts/t_master.sh
        notify_backup /etc/keepalived/scripts/t_backup.sh
        notify_fault  /etc/keepalived/scripts/t_fault.sh
        notify_stop   /etc/keepalived/scripts/t_stop.sh
}
 
/usr/local/twemproxy/conf/nutcracker.yml 
redis:
  listen: 192.168.10.243:6379
  hash: fnv1a_64
  distribution: ketama
  auto_eject_hosts: true
  redis: true
  server_retry_timeout: 2000
  server_failure_limit: 1
  servers:
   - 192.168.10.241:6379:1 node1
   - 192.168.10.242:6379:1 node2
 
 
 
Tp1 配置文件
 
Keepalived:
 
vrrp_script chk_t {
                script "/etc/keepalived/scripts/t_check.sh"   ###监控脚本
                interval 2                                        ###监控时间
}
vrrp_instance VI_1 {
        state BACKUP                            ###设置为BACKUP
        interface eth1                          ###监控网卡
        virtual_router_id 53
        priority 99                            ###权重值
        authentication {
                     auth_type PASS             ###加密
                     auth_pass redis            ###密码
        }
         track_interface {
            eth1
        }
        virtual_ipaddress {
             192.168.10.243                         ###VIP
        }
        notify_master /etc/keepalived/scripts/t_master.sh
        notify_backup /etc/keepalived/scripts/t_backup.sh
        notify_fault  /etc/keepalived/scripts/t_fault.sh
        notify_stop   /etc/keepalived/scripts/t_stop.sh
}
 
/usr/local/twemproxy/conf/nutcracker.yml 
redis:
  listen: 192.168.10.243:6379
  hash: fnv1a_64
  distribution: ketama
  auto_eject_hosts: true
  redis: true
  server_retry_timeout: 2000
  server_failure_limit: 1
  servers:
   - 192.168.10.241:6379:1 node1
   - 192.168.10.242:6379:1 node2
 
测试
redis-cli -h 192.168.10.243 
redis> set pp pp
OK
redis> get pp
"pp"
 
关掉tp 查看tp1 日志
tailf /var/log/keepalived-t-state.log
[backup]
Tue Dec 31 14:37:20 CST 2013
Being slave....
[master]
Tue Dec 31 14:47:28 CST 2013
Being master....
Run twemproxy ...
 
Tp2 接管 vip 继续提供服务
redis-cli -h 192.168.10.243 
redis> get pp
"pp"
 
重启tp 查看tp1 日志
tailf /var/log/keepalived-t-state.log
[backup]
Tue Dec 31 14:54:37 CST 2013
Being slave....
 
redis-cli -h 192.168.10.243 
redis> get pp
"pp"
 Tp接管主服务 tp2切换备份服务
 
 
3.可能遇到相关问题和处理办法:
1. 如果两机器不能连通,请注意关掉防火墙: 
service iptables stop #临时关闭 
 chkconfig iptables off #永久关闭
2. windows上编写的shell脚本,要注意通过dos2unix 命令转换,否则shell脚本可能有不可见符号不能正常执行。