es集群无缝迁移扩容

时间:2024-01-23 18:30:27

新节点加入老集群

cluster.name: soc-cluster
node.name: node-2test
path.data: /home/secure/elastic/data
path.logs: /home/secure/elastic/logs
network.host: 0.0.0.0
cluster.initial_master_nodes: ["node-2"]
discovery.seed_hosts: ["10.32.3.2:9300","10.32.3.18:9300"]
http.port: 9201
transport.tcp.port: 9301
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true

xpack.license.self_generated.type: basic
xpack.security.http.ssl.key: certs/instance/instance.key
xpack.security.http.ssl.certificate: certs/instance/instance.crt
xpack.security.http.ssl.certificate_authorities: certs/ca/ca.crt

xpack.security.transport.ssl.verification_mode: none
xpack.security.transport.ssl.key: certs/instance/instance.key
xpack.security.transport.ssl.certificate: certs/instance/instance.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca/ca.crt

xpack.security.http.ssl.supported_protocols: TLSv1.2
xpack.security.transport.ssl.supported_protocols: TLSv1.2

新节点配置

cluster.name: soc-cluster
node.name: node-2
cluster.initial_master_nodes: ["node-2"]
path.data: /data/soc/elastic/data
path.logs: /data/soc/elastic/logs
network.host: 0.0.0.0
discovery.seed_hosts: ["10.32.3.2", "10.32.3.18", "10.32.3.20"]
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true

xpack.license.self_generated.type: basic
xpack.security.http.ssl.key: certs/instance/instance.key
xpack.security.http.ssl.certificate: certs/instance/instance.crt
xpack.security.http.ssl.certificate_authorities: certs/ca/ca.crt

xpack.security.transport.ssl.verification_mode: none
xpack.security.transport.ssl.key: certs/instance/instance.key
xpack.security.transport.ssl.certificate: certs/instance/instance.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca/ca.crt

xpack.security.http.ssl.supported_protocols: TLSv1.2
xpack.security.transport.ssl.supported_protocols: TLSv1.2

老节点配置     

es集群无缝迁移扩容_elastic

     

es集群无缝迁移扩容_elastic_02

数据迁移

      1.旧集群添加新节点后自动会触发分片均衡策略

        

es集群无缝迁移扩容_json_03

         

es集群无缝迁移扩容_json_04

          

es集群无缝迁移扩容_elastic_05

       2.锁定分片只能分配到新节点上

          集群中每个节点都能尽量分配到相等的分片数量

         

es集群无缝迁移扩容_elastic_06

           手动干预集群的分片策略

           

es集群无缝迁移扩容_elastic_07

       3.手动驱逐旧节点上的分片到新节点上

es集群无缝迁移扩容_重启_08

es集群无缝迁移扩容_json_09

curl -u elastic:11111 --insecure https://10.32.3.2:9200/_cat/nodes

transient   //暂时配置 重启后失效
persistent  //永久配置 重启后仍然有效

1.源集群操作
  触发集群分片均衡策略,把集群老节点上的分片全部移动到新的集群节点上
 curl -u elastic:1111 --insecure -H "Content-Type: application/json" -XPUT https://10.32.3.2:9200/_cluster/settings -d '{
  "transient" : {
     "cluster.routing.allocation.include._name" : null, //设置集群分片可以包含的节点名称
     "cluster.routing.allocation.exclude._name" : "node-2,node-18,node-20,node-4"  //设置集群分片排除的节点名称
  }
}'


在分片迁移的中间状态由于分片迁移会消耗一定的性能。因此为了稳定性考虑,可以通过如下API将迁移的并发度和迁移速度调小一点。
curl -u elastic:11111 --insecure -H "Content-Type: application/json" -XPUT https://10.32.3.2:9200/_cluster/settings -d '{ "persistent": {
            "cluster.routing.allocation.node_concurrent_recoveries": 2,
            "indices.recovery.max_bytes_per_sec": "100mb"  //设置节点的分片传输速率
 }
}'


2.集群分离
  停止集群中老节点的es服务进程
  老节点下线需要滚动进行,且每个节点的下线间隔需要大于10s,否则会有失主的风险
  原来的主节点需要最后下线
  

4.还原es集群的配置
 curl -u elastic:11111 --insecure -H "Content-Type: application/json" -XPUT https://10.32.3.2:9200/_cluster/settings -d '{
  "transient" : {
     "cluster.routing.allocation.include._name" : null,  //还原集群的默认配置
     "cluster.routing.allocation.exclude._name" : null   //还原集群的默认配置
  }
}'

View Code

          1.把老节点上的分片迁移到新节点上

          

es集群无缝迁移扩容_elastic_10

           

es集群无缝迁移扩容_重启_11

       2.新创建的分片不要再路由到老节点上

         

es集群无缝迁移扩容_json_12

          等待一段时间后再观察

            

es集群无缝迁移扩容_elastic_13

               

es集群无缝迁移扩容_json_14

老节点下线

        

es集群无缝迁移扩容_elastic_15

        

es集群无缝迁移扩容_重启_16

        把所有老节点下线后,集群状态正常,原来索引中的数据也可以正常查询.整个集群的数据迁移完成

作者:不懂123