Redis Cluster集群搭建与配置

时间:2023-02-09 17:29:54

Redis Cluster是一种服务器sharding分片技术,关于Redis的集群方案应该怎么做,请参考我的另一篇博客http://www.cnblogs.com/xckk/p/6134655.html 

本文主要介绍Redis Cluster集群的工作原理,详细讲解了Redis Cluster集群如何搭建与配置。

一、redis安装

redis官网下载后是源码包,需要make安装。

1、解压redis-3.2.5.tar.gz

2、cd redis-3.2.5

3、make

4、cd src/

5、make install

安装完成,src包下面新增redis-server、redis-cli等这些可执行脚本,表示安装完成。

 

二、启动实例集群配置

搭建Redis Cluster,建立6个redis实例,3主3从,端口号分别为7000-7005

 

创建以6个端口命名的子目录,之后我们将在6个子目录上都开启Redis实例

cd redis-3.2.5

mkdir cluster-test

cd cluster-test

mkdir 7000 7001 7002 7003 7004 7005

 

Redis实例配置

在7000到7005每一个目录中都创建redis.conf文件,配置项如下:

cd redis-3.2.5

mkdir cluster-test

cd cluster-test

mkdir 7000 7001 7002 7003 7004 7005

 

 各配置描述如下:

配置项redis.conf

示例

备注

port 7000 端口号
cluster-enabled yes

yes,表示支持Redis Cluster
no,单实例

cluster-config-file nodes-7000.conf

Redis Cluster记录的启动信息文件,文件由cluster自动生成,不需要用户编辑

cluster-node-timeout 5000 毫秒单位,失效时间
appendonly yes 支持appendonly持久化方式
protected-mode no 默认是打开的,此处关闭。因为jedis访问时,如果是保护模式,影响访问
bind 192.168.121.130 机器IP,此处最好写IP,不要写localhost或127.0.0.1,可能影响jedis客户端访问

 

Redis实例启动

将第一步安装生成的redis-server拷贝到cluster-test目录,分别后台启动6个redis实例

nohup ./redis-server 7000/redis.conf &

nohup ./redis-server 7001/redis.conf &

nohup ./redis-server 7002/redis.conf &

nohup ./redis-server 7003/redis.conf &

nohup ./redis-server 7004/redis.conf &

nohup ./redis-server 7005/redis.conf &

启动完后,cluster-test目录下生成了nodes-7000.conf等启动文件信息。
查看进程启动成功:[root@centos2 cluster-test]# ps -ef|grep redis

Redis Cluster集群搭建与配置

三、创建集群

6个redis实例已经创建,它们目前是相互独立的,没有关联。接下来是创建Redis Cluster集群,将6个实例组成一个集群。

 

redis-trib命令创建集群

src目录下make生成的redis-trib命令,可用于Redis Cluster集群创建的一些脚本启动,它是一个ruby程序。因此我们需要配置ruby环境

 

安装ruby

yum install ruby

yum install ruby-irb

具体ruby的安装可参考:http://jingyan.baidu.com/article/b7001fe173fe9a0e7382dd57.html

 

安装Gem

Gem是一个管理Ruby库和程序的标准包,它通过RubyGem(如 http://rubygems.org/ )源来查找、安装、升级和卸载软件包,非常的便捷。

配置了ruby环境后,还需要安装redis整个包才能够运行redis-trib。安装过程中,需要先安装gem库,不然会出现gem command not found错误。

[root@centos2 src]# gem install redis

-bash: gem: command not found

 

安装gem过程中,可能出现以下问题

yum install rubygems

ERROR: Could not find a valid gem 'redis' (>= 0) in any repository

 

问题原因:无法连接gem服务器,原因是该gem服务器被墙了。按如下步骤手动下载安装。

wget https://rubygems.global.ssl.fastly.net/gems/redis-3.2.1.gem

gem install -l ./redis-3.2.1.gem

安装完后,src包下就会生成redis-trib.rb文件

 

启动命令

./redis-trib.rb create --replicas 1 192.168.121.130:7000 192.168.121.130:7001 192.168.121.130:7002 192.168.121.130:7003 192.168.121.130:7004 192.168.121.130:7005

 

启动后,各Redis实例之间分别会新增一个端口用于通信,通信端口是Redis实例端口+1W,分别为17000,17001,17002,17003,17004,17005

命令的解释如下:

1、给定的命令将会被ruby程序翻译为create,这表示我们想要创建一个新的集群。

2、选项--replicas 1意思是为每一个master节点创建一个slave节点。

3、其他参数表示redis实例的地址及端口,以空格为间隔。

 

启动日志分析

出现[OK] All 16384 slots covered.表示启动成功。从启动日志可以看出,master是7000,7001,7002,对应的slave是7003,7004,7005

[root@centos1 src]# ./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005

>>> Creating cluster

>>> Performing hash slots allocation on 6 nodes...

Using 3 masters:

127.0.0.1:7000

127.0.0.1:7001

127.0.0.1:7002

Adding replica 127.0.0.1:7003 to 127.0.0.1:7000

Adding replica 127.0.0.1:7004 to 127.0.0.1:7001

Adding replica 127.0.0.1:7005 to 127.0.0.1:7002

M: 434e5ee5cf198626e32d71a4aee27bc4058b4e45 127.0.0.1:7000

slots:0-5460 (5461 slots) master

M: 048a0c9631c87e5ecc97a4ce5834d935f2f938b6 127.0.0.1:7001

slots:5461-10922 (5462 slots) master

M: 04ae4184b2853afb8122d15b5b2efa471d4ca251 127.0.0.1:7002

slots:10923-16383 (5461 slots) master

S: 499b0bfa9274425bfcb87f7aa3da76456c3397da 127.0.0.1:7003

replicates 434e5ee5cf198626e32d71a4aee27bc4058b4e45

S: 7f92536844f1698b5776c2f0823f1822f0bb88d7 127.0.0.1:7004

replicates 048a0c9631c87e5ecc97a4ce5834d935f2f938b6

S: 411bb5ea8c07d872033f7c473c35fe38416052ce 127.0.0.1:7005

replicates 04ae4184b2853afb8122d15b5b2efa471d4ca251

Can I set the above configuration? (type 'yes' to accept): yes

>>> Nodes configuration updated

>>> Assign a different config epoch to each node

>>> Sending CLUSTER MEET messages to join the cluster

Waiting for the cluster to join...

>>> Performing Cluster Check (using node 127.0.0.1:7000)

M: 434e5ee5cf198626e32d71a4aee27bc4058b4e45 127.0.0.1:7000

slots:0-5460 (5461 slots) master

1 additional replica(s)

S: 411bb5ea8c07d872033f7c473c35fe38416052ce 127.0.0.1:7005

slots: (0 slots) slave

replicates 04ae4184b2853afb8122d15b5b2efa471d4ca251

S: 7f92536844f1698b5776c2f0823f1822f0bb88d7 127.0.0.1:7004

slots: (0 slots) slave

replicates 048a0c9631c87e5ecc97a4ce5834d935f2f938b6

M: 048a0c9631c87e5ecc97a4ce5834d935f2f938b6 127.0.0.1:7001

slots:5461-10922 (5462 slots) master

1 additional replica(s)

S: 499b0bfa9274425bfcb87f7aa3da76456c3397da 127.0.0.1:7003

slots: (0 slots) slave

replicates 434e5ee5cf198626e32d71a4aee27bc4058b4e45

M: 04ae4184b2853afb8122d15b5b2efa471d4ca251 127.0.0.1:7002

slots:10923-16383 (5461 slots) master

1 additional replica(s)

[OK] All nodes agree about slots configuration.

>>> Check for open slots...

>>> Check slots coverage...

[OK] All 16384 slots covered.

 

集群启动状态检测

我们可以通过./redis-trib.rb check 192.168.121.130:7000来检测Redis Cluster的启动状态。

[root@centos2 src]# ./redis-trib.rb check 192.168.121.130:7000

>>> Performing Cluster Check (using node 192.168.121.130:7000)

M: dca47b274799927fd15aa4e6312b94752418f0c0 192.168.121.130:7000

slots:0-5460 (5461 slots) master

1 additional replica(s)

M: fe1082655e832bd2afb7d4bfabfec3ce2354868b 192.168.121.130:7002

slots:10923-16383 (5461 slots) master

1 additional replica(s)

S: 7fe93723c48b7ea33bbf21f12e97d18bdd0221ea 192.168.121.130:7003

slots: (0 slots) slave

replicates dca47b274799927fd15aa4e6312b94752418f0c0

M: 59c926f7bb1e24e293d379507742c52b6dd08cc2 192.168.121.130:7001

slots:5461-10922 (5462 slots) master

1 additional replica(s)

S: a554424a9969af2b55f4e11f9eedf1bc3bce9000 192.168.121.130:7004

slots: (0 slots) slave

replicates 59c926f7bb1e24e293d379507742c52b6dd08cc2

S: 065fb213e9db64627c27af93500420351b80ac34 192.168.121.130:7005

slots: (0 slots) slave

replicates fe1082655e832bd2afb7d4bfabfec3ce2354868b

[OK] All nodes agree about slots configuration.

>>> Check for open slots...

>>> Check slots coverage...

[OK] All 16384 slots covered.

 

登录集群

通过命令./redis-cli -c -h 192.168.121.129 -p 7000用来登录

 

重定向

从以下命令可以看出,我们登录的是192.168.121.130机器的7000端口Redis实例,set foo 123,foo经过hash后落到了12182槽,12182槽存储在192.168.121.130机器的7002端口实例,因此日志显示重定向到7002端口的实例。

[root@centos2 src]# ./redis-cli -c -h 192.168.121.130 -p 7000

192.168.121.130:7000> set foo 123

-> Redirected to slot [12182] located at 192.168.121.130:7002

OK

192.168.121.130:7002> get foo

"123"

192.168.121.130:7002>

 

redis cluster的主从同步采用异步复制(提高性能),因此会存在部分写内容丢失的情况

 

四、创建集群-通过create-cluster脚本

通过redis-trib.rb命令创建集群涉及命令较多,稍显复杂。好在redis提供了create-cluster脚本utils/create-cluster/create-cluster来创建集群,方便很多。

 

create-cluster命令介绍

从create-cluster源码看,其实现原理也是利用了redis-trib.rb命令。该脚本默认创建3个master节点,3个salve节点。

1、create-cluster start

创建6个redis实例,默认端口号是30001-30006,如果想改端口号,在create-cluster里面更改PORT值即可。

如果想要端口号是7000-7005,设置PORT=6999即可

2、create-cluster create

创建集群,3个master,3个slave

3、create-cluster stop

关闭Redis Cluster集群和Redis实例

4、create-cluster clean

清除所在Redis实例的data, logs, config文件

 

使用方法

cd /home/redis-3.2/utils/create-cluster --进入目录

./create-cluster clean ---先清除Redis实例的文件

./create-cluster start ---启动Redis实例

./create-cluster create ---创建Redis Cluster集群

 

启动完后,使用redis-cli命令登录,即可正常访问。

[root@centos1 create-cluster]# redis-cli -c -p 7000

127.0.0.1:7000> set foo hello

-> Redirected to slot [12182] located at 127.0.0.1:7002

OK

127.0.0.1:7002> set hello world

-> Redirected to slot [866] located at 127.0.0.1:7000

OK

127.0.0.1:7000> get foo

-> Redirected to slot [12182] located at 127.0.0.1:7002

"hello"

127.0.0.1:7002> get hello

-> Redirected to slot [866] located at 127.0.0.1:7000

"world"

127.0.0.1:7000>

 

五、Jedis测试程序

Jedis是Redis官方首选的 Java 客户端开发包。我们通过Junit进行单元测试

maven添加jedis和junit依赖包

<dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>

        <!-- https://mvnrepository.com/artifact/redis.clients/jedis -->
        <dependency>
            <groupId>redis.clients</groupId>
            <artifactId>jedis</artifactId>
            <version>2.8.1</version>
        </dependency>

    </dependencies>

 

java代码

package com.hk.test.helloWorld;

import java.util.HashSet;
import java.util.List;
import java.util.Set;

import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
import org.junit.Before;
import org.junit.Test;

import redis.clients.jedis.HostAndPort;
import redis.clients.jedis.JedisCluster;

public class TestJedis
{
    JedisCluster jc = null;

    @Before
    public void before()
    {
        Set<HostAndPort> jedisClusterNodes = new HashSet<HostAndPort>();
        // Jedis Cluster will attempt to discover cluster nodes automatically
        jedisClusterNodes.add(new HostAndPort("centos2", 7000));
        GenericObjectPoolConfig config;
        jc = new JedisCluster(jedisClusterNodes);
    }

    @Test
    public void testFoo()
    {
        for (int i = 0; i <= 3000; i++)
        {
            String key = "foo" + i;
            String value = String.valueOf(i);
            jc.set(key, value);
        }
    }

    @Test
    public void test_incr()
    {

        String key = "page_view";
        jc.del(key);
        jc.incr(key);
        String result = jc.get(key);
        System.out.println(result);

    }

    @Test
    public void test_setAndGetStringVal()
    {
        String key = "foo";
        String value = "bar";
        jc.set(key, value);
        String result = jc.get(key);
        System.out.println(result);

    }

    @Test
    public void test_setAndGetStringVal_and_set_expire() throws InterruptedException
    {
        String key = "hello";
        String value = "world";
        int seconds = 3;
        jc.setex(key, seconds, value);
        String result = jc.get(key);
        System.out.println(result);

        Thread.sleep(seconds * 1000);
        result = jc.get(key);
        System.out.println(result);

    }

    @Test
    public void test_setAndGetHashVal()
    {

        String key = "website";
        String field = "google";
        String value = "google.com";
        jc.del(key);
        jc.hset(key, field, value);
        String result = jc.hget(key, field);
        System.out.println(result);

    }

    @Test
    public void test_setAndGetListVal()
    {

        String key = "mylist";
        jc.del(key);
        String[] vals =
        { "a", "b", "c" };
        jc.rpush(key, vals);
        List<String> result = jc.lrange(key, 0, -1);
        System.out.println(result);

    }

    @Test
    public void test_setAndGetSetVal()
    {

        String key = "language";
        jc.del(key);
        String[] members =
        { "java", "ruby", "python" };
        jc.sadd(key, members);
        Set<String> result = jc.smembers(key);
        System.out.println(result);

    }
}

 

六、常见问题

一、Could not connect to Redis at 127.0.0.1:30001 Connection refused

原因:create-cluster脚本默认设定的端口从30001开始,由于我设置的redis端口是7000-7005,当时启动cluster时用的redis-trib,因此端口号不同,

./redis-trib.rb create --replicas 1 192.168.121.130:7000 192.168.121.130:7001 192.168.121.130:7002 192.168.121.130:7003 192.168.121.130:7004 192.168.121.130:7005

解决方法:

1、create-cluster里面将PORT由30001改为6999。

2、关闭Redis实例进程

可以通过netstat -apn|grep redis或者ps -ef|grep redis查找进程号,kill -9 {进程号}进行关闭。

3、create-cluster stop

4、create-cluster clean 清除data,log,conf文件

5、create-cluster start

6、create-cluster create

 

二、redis.clients.jedis.exceptions.JedisDataException: DENIED Redis is running in protected mode because protected mode is enabled

原因:Redis实例启动时默认设置了保护模式

解决方法:需要在启动时增加配置项 protected-mode no

1、如果是redis-trib.rb启动,启动配置文件内增加 protected-mode no

port 7000

cluster-enabled yes

cluster-config-file nodes-7000.conf

cluster-node-timeout 5000

appendonly yes

protected-mode no

bind 192.168.121.129

 

2、如果是create-cluster启动,在create-cluster脚本里start分支中增加protected-mode no

if [ "$1" == "start" ]

then

while [ $((PORT < ENDPORT)) != "0" ]; do

PORT=$((PORT+1))

echo "Starting $PORT"

../../src/redis-server --port $PORT --cluster-enabled yes --cluster-config-file nodes-${PORT}.conf --cluster-node-timeout $TIMEOUT --appendonly yes --bind 192.168.121.130 --protected-mode no --appendfilename appendonly-${PORT}.aof --dbfilename dump-${PORT}.rdb --logfile ${PORT}.log --daemonize yes

done

exit 0

fi

 

三、Jedis访问时,redis.clients.jedis.exceptions.JedisClusterMaxRedirectionsException: Too many Cluster redirections?

若用JedisCluster操作集群报以上错误,就是有Redis节点通讯被拒绝,最好是将redis.conf文件中的bind属性设置成redis节点所在服务器的物理ip地址,而不是localhost

1、更改node节点配置文件绑定地址

Redis Cluster集群搭建与配置

2、重启redis节点

3、重新create cluster

./redis-trib.rb create --replicas 192.168.121.130:7000 192.168.121.130:7001 192.168.121.130:7002 192.168.121.130:7003192.168.121.130:7004 192.168.121.130:7005

 

四、[ERR] Node 172.168.63.202:7001 is not empty. Either the nodealready knows other nodes (check with CLUSTER NODES) or contains some

解决方法:(一般1,2步即可解决)

1)、将需要新增的节点下aof、rdb等本地备份文件删除;

2)、同时将新Node的集群配置文件删除,即:删除你redis.conf里面cluster-config-file所在的文件;

3)、再次添加新节点如果还是报错,则登录新Node,./redis-cli–h x –p对数据库进行清除:

172.168.63.201:7001>  flushdb      #清空当前数据库

4)、删除后再执行第2,3步,再./create-cluster start即可

 

五、ERR Invalid node address specified

./redis-trib.rb create --replicas 1 centos2:7000 centos2:7001 centos2:7002 centos2:7003 centos2:7004 centos2:7005

报" ERR Invalid node address specified"

由于Redis-trib.rb 对域名或主机名支持不好,故在创建集群的时候要使用ip:port的方式

./redis-trib.rb create --replicas 1 192.168.121.130:7000 192.168.121.130:7001 192.168.121.130:7002 192.168.121.130:7003 192.168.121.130:7004 192.168.121.130:7005

 

六、ERR Slot 4648 is already busy (Redis::CommandError)

这是由于之前创建集群没有成功,需要

1、将nodes.conf和dir里面的文件全部删除(注意不要删除了redis.conf)

2、将redis实例关闭后重新启动

3、再执行/redis-trib.rb create 。。。脚本

Sorry, the cluster configuration file nodes.conf is already used by a different Redis Cluster node. Please make sure that different nodes use different cluster configuration files.

 

七、[ERR] Node 127.0.0.1:7000 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

解决方法:先stop和clean,再start和create

./create-cluster stop

./create-cluster clean

./create-cluster start

./create-cluster create

 

秀才坤坤出品

转载请注明来源http://www.cnblogs.com/xckk/p/6144447.html