Fix e2e monitoring file/metricbeat configuration #5542
Conversation
…cloud load balancer.
config/e2e/monitoring.yaml
Outdated
| # Since E2E cluster's Elasticsearch instance may lag behind the newest version of Beat, we need to allow this scenario. | ||
| allow_older_versions: true | ||
| # The maximum size to send in a single Elasticsearch bulk API index request. Needed if Elasticsearch is behind Cloud Ingress. | ||
| bulk_max_body_size: 10M |
There was a problem hiding this comment.
I am confused: have we not been running the monitor behind an ingress for a long time? Why is this only now a problem?
There was a problem hiding this comment.
Unsure what has recently changed, or changed with what I was ingesting, but I directly ran into:
client.go:244: ERR Failed to perform any bulk index operations: 413 Request Entity Too Large
during my testing of this during a local e2e test run. This caused no ingestion at all into the ES cluster, and everything after failed.
config/e2e/monitoring.yaml
Outdated
| # The maximum number of events to bulk in a single Elasticsearch bulk API index request. Needed if Elasticsearch is behind Cloud Ingress. | ||
| bulk_max_size: 10 | ||
| # The maximum byte size to send in a single Elasticsearch bulk API index request, regardless of compression. Needed if Elasticsearch is behind Cloud Ingress. | ||
| bulk_max_bytes: 10485760 |
There was a problem hiding this comment.
I struggle to find any reference to these options in Beats documentation with the exception of allow_older_versions and bulk_max_size The only thing I could find was a user requesting for those to be added, but afaik that never happened. Do they exist?
There was a problem hiding this comment.
My apologies. I really should have referenced the beat configuration reference before adding those. I'm testing this configuration locally now again to ensure I don't see any 413 Request Entity Too Large errors again.
|
run full pr build |
* Fix e2e monitoring file/metricbeat configuration since it's behind a cloud load balancer. * remove missing options from e2e monitoring configuration file
Since e2e monitoring Elasticsearch cluster is behind a cloud ingress Load Balancer at this time, the filebeat/metricbeat configuration needs to be tweaked to work in that scenario. See elastic/beats#3688