In this guide I use my previous Elasticsearch cluster to show how to re-join exited Elasticsearch master node container with external storage configuration to the cluster again in a Docker Swarm configuration. My previous Elasticsearch cluster normal state:
• 3 Master nodes (loganalytics3 as active master)
• 3 Coordinating only nodes
• Green health status
• 100% active shards
Let’s begin.
The following guide is applied only to loganalytics1 node.
1. Verify Elasticsearch cluster state
curl -XGET ‘localhost:9200/_cat/nodes?v’ -u elastic:elastic
curl -XGET ‘localhost:9200/_cat/health?v’ -u elastic:elastic
docker stack ps loganalyticsswarm
Elasticsearch cluster current state:
• 2 Master nodes (loganalytics1 as active master)
• 3 Coordinating only nodes
• Yellow health status
• 75% active shards
one of Elasticsearch master node (loganalytics3) was exited and Docker always trying to restart the container but still failing.
2. Delete exited Elasticsearch master node (loganalytics3) data and logs.
rm -rf /cephfs/loganalytics3/data/*
rm -rf /cephfs/loganalytics3/logs/*
3. Verify Elasticsearch cluster state again after the container started
docker stack ps loganalyticsswarm
curl -XGET ‘localhost:9200/_cat/health?v’ -u elastic:elastic
curl -XGET ‘localhost:9200/_cat/nodes?v’ -u elastic:elastic
Elasticsearch cluster back to normal state again:
• 3 Master nodes (loganalytics1 as active master)
• 3 Coordinating only nodes
• Green health status
• 100% active shards