429s due to replica operations

[bcastilho90]

I have a cluster with 3 master nodes and several data nodes. From time to time, we experience a brief period where certain nodes will return 429s due to replica operations:

es_rejected_execution_exception Reason: "rejected execution of primary operation [coordinating_and_primary_bytes=0, replica_bytes=2210080256, all_bytes=2210080256, primary_operation_bytes=30430, max_coordinating_and_primary_bytes=2147483648

I've scaled the cluster before, but we continue to see this intermittently since it happens on random nodes. Is there a way to have the readiness probe fail on cases like this so that requests stop being sent to a node that is overloaded?

I'm not sure how exactly to find out what is causing the back log of replica ops.

[barkbay]

I don't feel like a readiness probe should be used to deal with performance issues, also having a readiness probe failing does not mean that a Pod cannot be accessed. If a cluster is struggling to handle the load, I would first try to understand why.

I'm not sure how exactly to find out what is causing the back log of replica ops.

I would first enable stack monitoring, also check infra metrics (disk + network i/o, cpu load...) and maybe try to ask the ES team in https://discuss.elastic.co/c/elastic-stack/elasticsearch/6 with the cluster configuration, the CPU and memory allocated, metrics you have collected, how data is indexed and searched...

I don't think there is something we can do on the orchestration side, feel free to reopen if you think otherwise.