Как решить проблему unassigned shards в кластере OpenSearch приводящей к статусу red?

Question

[dmvlch_dev]

Всем привет!

Есть кластер Opensearch состоящий из трех нод, две дата ноды и один кластер менеджер.

ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role node.roles      cluster_manager name
10.220.101.40           18          93   0    0.03    0.07     0.03 di        data,ingest     -               opensearch-d1
10.220.101.7            58          95   0    0.05    0.07     0.02 m         cluster_manager *               opensearch-cluster_manager
10.220.101.41           30          93   0    0.19    0.29     0.24 di        data,ingest     -               opensearch-d2

Статус кластера находится в "red"

{
  "cluster_name": "graylog",
  "status": "red",
  "timed_out": false,
  "number_of_nodes": 3,
  "number_of_data_nodes": 2,
  "discovered_master": true,
  "discovered_cluster_manager": true,
  "active_primary_shards": 0,
  "active_shards": 0,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 9,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 0.0
}

Конфиги трех нод

Cluster manager:

cluster.name: graylog
node.name: opensearch-cluster_manager
node.roles: [ cluster_manager ]
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: 10.220.101.7            
discovery.seed_hosts: ["10.220.101.40", "10.220.101.7", "10.220.101.41"]
cluster.initial_cluster_manager_nodes: ["opensearch-cluster_manager", "opensearch-d1", "opensearch-d2"]
plugins.security.disabled: true

dn1:

cluster.name: graylog
node.name: opensearch-d1
node.roles: [ data, ingest ]
path.data: /data/opensearch
path.logs: /var/log/opensearch
network.host: 10.220.101.40
discovery.seed_hosts: ["10.220.101.40","10.220.101.7", "10.220.101.41"]
cluster.initial_cluster_manager_nodes: ["opensearch-cluster_manager", "opensearch-d1", "opensearch-d2"]
plugins.security.disabled: true

dn2:

cluster.name: graylog
node.name: opensearch-d2
node.roles: [ data, ingest ]
path.data: /data/opensearch
path.logs: /var/log/opensearch
network.host: 10.220.101.41
discovery.seed_hosts: ["10.220.101.40", "10.220.101.7", "10.220.101.41"]
cluster.initial_cluster_manager_nodes: ["opensearch-cluster_manager", "opensearch-d1", "opensearch-d2"]
plugins.security.disabled: true

Состояние распределения данных в кластере:

[root@uxrudc1s121l ~]# curl -XGET 'http://localhost:9200/_cluster/allocation/explain?pretty=true'
{
  "index" : ".plugins-ml-config",
  "shard" : 0,
  "primary" : true,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "CLUSTER_RECOVERED",
    "at" : "2024-10-28T05:29:55.566Z",
    "last_allocation_status" : "no_valid_shard_copy"
  },
  "can_allocate" : "no_valid_shard_copy",
  "allocate_explanation" : "cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster",
  "node_allocation_decisions" : [
    {
      "node_id" : "6zVH9iXxQ5GVZhIj-mq4Te",
      "node_name" : "opensearch-d2",
      "transport_address" : "10.220.101.41:9300",
      "node_attributes" : {
        "shard_indexing_pressure_enabled" : "true"
      },
      "node_decision" : "no",
      "store" : {
        "found" : false
      }
    },
    {
      "node_id" : "zX9apdq8REyyaC_tXKhmJk",
      "node_name" : "opensearch-d1",
      "transport_address" : "10.220.101.40:9300",
      "node_attributes" : {
        "shard_indexing_pressure_enabled" : "true"
      },
      "node_decision" : "no",
      "store" : {
        "found" : false
      }
    }
  ]
}

Имеются не назначенные шарды

[root@uxrudc1s121l ~]# curl -XGET 'http://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason'
.plugins-ml-config               0 p UNASSIGNED CLUSTER_RECOVERED
.plugins-ml-config               0 r UNASSIGNED REPLICA_ADDED
.opensearch-observability        0 p UNASSIGNED CLUSTER_RECOVERED
.opensearch-observability        0 r UNASSIGNED REPLICA_ADDED
gl-system-events_deflector       0 p UNASSIGNED CLUSTER_RECOVERED
.opensearch-sap-log-types-config 0 p UNASSIGNED CLUSTER_RECOVERED
.opensearch-sap-log-types-config 0 r UNASSIGNED REPLICA_ADDED
gl-events_0                      0 p UNASSIGNED CLUSTER_RECOVERED
graylog_0                        0 p UNASSIGNED CLUSTER_RECOVERED