An easy way to cycle EC2 instances where we have an elasticsearch cluster running. As an example target we have,

• Two instances i-11111111 and i-22222222, both running elastic search as a cluster with replicas set to 2, so that each has a replica of the others primary indices.
• Add one to the auto-scaling group, increasing desired to 3
• Wait for new instance i-333333 to join the cluster

$ curl -s 'http://localhost:9200/_cluster/health?pretty'
{
  "cluster_name": "mycluster",
  "status": "green",
  "number_of_nodes": 2,
}

• Blacklist the instance we’re going to terminate, i-11111111, from the elasticsearch cluster

$ curl -s 'http://localhost:9200/_cat/nodes'
  
35xx59d583f9 172.17.0.1 16 58 0.00 d * i-11111111
3259xxd31e3c 172.17.0.2  4 65 0.00 d m i-22222222

$ curl -XPUT localhost:9200/_cluster/settings -d '{
  "transient" :{
      "cluster.routing.allocation.exclude._ip" : "172.17.0.1"
   }
}'

• Wait until that node is not carrying any more indices

$ curl -s 'http://localhost:9200/_nodes/i-11111111/stats/indices?pretty' | jq '.nodes[].indices.docs.count'

• Check that the cluster is green and the new node has the indices

$ curl -s 'http://localhost:9200/_cluster/health?pretty'
  
{
  "cluster_name": "mycluster",
  "status": "green",
  "number_of_nodes": 3,
}

$ curl -s 'http://localhost:9200/_nodes/i-33333333/stats/indices?pretty' | jq '.nodes[].indices.docs.count'

123421

• Terminate the old instance

You should now have a new 2 instance elasticsearch cluster. Repeat as necessary.