How to re-join exited Elasticsearch master node container with external storage configuration to the cluster again in a Docker Swarm configuration

In this guide I use my previous Elasticsearch cluster to show how to re-join exited Elasticsearch master node container with external storage configuration to the cluster again in a Docker Swarm configuration. My previous Elasticsearch cluster normal state:

• 3 Master nodes (loganalytics3 as active master)
• 3 Coordinating only nodes
Green health status

• 100% active shards

Let’s begin.
The following guide is applied only to loganalytics1 node.
1. Verify Elasticsearch cluster state
curl -XGET ‘localhost:9200/_cat/nodes?v’ -u elastic:elastic
curl -XGET ‘localhost:9200/_cat/health?v’ -u elastic:elastic
docker stack ps loganalyticsswarm
Elasticsearch cluster current state:
• 2 Master nodes (loganalytics1 as active master)
• 3 Coordinating only nodes
• Yellow health status
• 75% active shards

one of Elasticsearch master node (loganalytics3) was exited and Docker always trying to restart the container but still failing.

2.
Delete exited Elasticsearch master node (loganalytics3) data and logs.
rm -rf /cephfs/loganalytics3/data/*
rm -rf /cephfs/loganalytics3/logs/*

3.
Verify Elasticsearch cluster state again after the container started
docker stack ps loganalyticsswarm

curl -XGET ‘localhost:9200/_cat/health?v’ -u elastic:elastic
curl -XGET ‘localhost:9200/_cat/nodes?v’ -u elastic:elastic

Elasticsearch cluster back to normal state again:
• 3 Master nodes (loganalytics1 as active master)

• 3 Coordinating only nodes
• Green health status
• 100% active shards

Ade Destrianto
Just tryna git gud.

Leave a Reply

Your email address will not be published. Required fields are marked *