An easy way to cycle EC2 instances where we have an elasticsearch cluster running. As an example target we have,

  • Two instances i-11111111 and i-22222222, both running elastic search as a cluster with replicas set to 2, so that each has a replica of the others primary indices.
  • Add one to the auto-scaling group, increasing desired to 3
  • Wait for new instance i-333333 to join the cluster
$ curl -s 'http://localhost:9200/_cluster/health?pretty'
{
  "cluster_name": "mycluster",
  "status": "green",
  "number_of_nodes": 2,
}
  • Blacklist the instance we’re going to terminate, i-11111111, from the elasticsearch cluster

    $ curl -s 'http://localhost:9200/_cat/nodes'
      
    35xx59d583f9 172.17.0.1 16 58 0.00 d * i-11111111
    3259xxd31e3c 172.17.0.2  4 65 0.00 d m i-22222222
    
    $ curl -XPUT localhost:9200/_cluster/settings -d '{
      "transient" :{
          "cluster.routing.allocation.exclude._ip" : "172.17.0.1"
       }
    }'
    
  • Wait until that node is not carrying any more indices

    $ curl -s 'http://localhost:9200/_nodes/i-11111111/stats/indices?pretty' | jq '.nodes[].indices.docs.count'
    
  • Check that the cluster is green and the new node has the indices

    $ curl -s 'http://localhost:9200/_cluster/health?pretty'
      
    {
      "cluster_name": "mycluster",
      "status": "green",
      "number_of_nodes": 3,
    }
    
    $ curl -s 'http://localhost:9200/_nodes/i-33333333/stats/indices?pretty' | jq '.nodes[].indices.docs.count'
    
    123421
    
  • Terminate the old instance

You should now have a new 2 instance elasticsearch cluster. Repeat as necessary.