ElasticSearch-解决this action would add [5] total shards, but this cluster currently has [1000]/[1000

简介

本文介绍ElasticSearch总分片数量导致的异常的解决方案。

异常的日志为：this action would add [5] total shards, but this cluster currently has [1000]/[1000

问题描述

项目里SkyWalking使用ES来存数据，但今天我查看日志时没有今天的数据，只有昨天的，于是我查看SkyWalking的日志（logs/skywalking-oap-server.log），发现如下结果：

[4532]: index [sw_segment-20211201], type [_doc], id [1c8b09ece71a45bd8de74e27a74a6bf6.167.16383242084776012], message [ElasticsearchException[Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [5] total shards, but this cluster currently has [1000]/[1000] maximum shards open;]]]
[4533]: index [sw_log-20211201], type [_doc], id [34a07ac188a8433e865a3bbabb74673a], message [ElasticsearchException[Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [5] total shards, but this cluster currently has [1000]/[1000] maximum shards open;]]]
[4534]: index [sw_segment-20211201], type [_doc], id [1c8b09ece71a45bd8de74e27a74a6bf6.167.16383242085016022], message [ElasticsearchException[Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [5] total shards, but this cluster currently has [1000]/[1000] maximum shards open;]]]
[4535]: index [sw_segment-20211201], type [_doc], id [1c8b09ece71a45bd8de74e27a74a6bf6.167.16383242085336030], message [ElasticsearchException[Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [5] total shards, but this cluster currently has [1000]/[1000] maximum shards open;]]]
[4536]: index [sw_segment-20211201], type [_doc], id [1c8b09ece71a45bd8de74e27a74a6bf6.167.16383242085436040], message [ElasticsearchException[Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [5] total shards, but this cluster currently has [1000]/[1000] maximum shards open;]]]

其异常信息都是：this action would add [5] total shards, but this cluster currently has [1000]/[1000] maximum shards open; 

原因分析

ES7.x默认总分片只有1000个，目前已经用完了，导致已经没法创建新的索引了。

解决方案

解决方法就是提高ES的分片数量。

方法1：控制台

PUT /_cluster/settings
{
  "persistent": {
    "cluster": {
      "max_shards_per_node":10000
    }
  }
}

persistent：永久生效，transient：临时生效。

方法2：CURL命令

curl --location --request PUT 'http://127.0.0.1:9200/_cluster/settings' \
--header 'Content-Type: application/json' \
--data '{"persistent":{"cluster":{"max_shards_per_node":10000}}}'

persistent：永久生效，transient：临时生效。 

无效的方法

有人说可以修改配置文件（elasticsearch.yml），添加下边一行

cluster.max_shards_per_node: 10000

但我测试发现 ，没有生效。我通过GET /_cluster/settings?pretty查看，返回结果如下：

{
    "persistent": {},
    "transient": {}
}

我的ES版本为：7.15.0 

查看是否生效

GET /_cluster/settings?pretty

结果

{
    "persistent": {
        "cluster": {
            "max_shards_per_node": "10000"
        }
    },
    "transient": {}
}