This is a helper exporter that other exporters can depend on. Today, it primarily offers queued retry capabilities.
⚠️ This exporter should not be added to a service pipeline.
The following configuration options can be modified:
retry_on_failureenabled(default = true)initial_interval(default = 5s): Time to wait after the first failure before retrying; ignored ifenabledisfalsemax_interval(default = 30s): Is the upper bound on backoff; ignored ifenabledisfalsemax_elapsed_time(default = 300s): Is the maximum amount of time spent trying to send a batch; ignored ifenabledisfalse. If set to 0, the retries are never stopped.multiplier(default = 1.5): Factor by which the retry interval is multiplied on each attempt; ignored ifenabledisfalse
sending_queueenabled(default = true)num_consumers(default = 10): Number of consumers that dequeue batches; ignored ifenabledisfalsewait_for_result(default = false): determines if incoming requests are blocked until the request is processed or not.block_on_overflow(default = false): If true, blocks the request until the queue has space otherwise rejects the data immediately; ignored ifenabledisfalsesizer(default = requests): How the queue and batching is measured. Available options:requests: number of incoming batches of metrics, logs, traces (the most performant option);items: number of the smallest parts of each signal (spans, metric data points, log records);bytes: the size of serialized data in bytes (the least performant option).
queue_size(default = 1000): Maximum size the queue can accept. Measured in units defined bysizerbatchdisabled by default if not definedflush_timeout: time after which a batch will be sent regardless of its size.min_size: the minimum size of a batch.min_size: the maximum size of a batch, enables batch splitting.
timeout(default = 5s): Time to wait per individual attempt to send data to a backend
The initial_interval, max_interval, max_elapsed_time, and timeout options accept
duration strings,
valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
To use the persistent queue, the following setting needs to be set:
sending_queuestorage(default = none): When set, enables persistence and uses the component specified as a storage extension for the persistent queue. There is no in-memory queue when set.
The maximum number of batches stored to disk can be controlled using sending_queue.queue_size parameter (which,
similarly as for in-memory buffering, defaults to 1000 batches).
When persistent queue is enabled, the batches are being buffered using the provided storage extension - filestorage is a popular and safe choice. If the collector instance is killed while having some items in the persistent queue, on restart the items will be picked and the exporting is continued.
┌─Consumer #1─┐
│ ┌───┐ │
──────Deleted────── ┌───►│ │ 1 │ ├───► Success
Waiting in channel x x x │ │ └───┘ │
for consumer ───┐ x x x │ │ │
│ x x x │ └─────────────┘
▼ x x x │
┌─────────────────────────────────────────x─────x───┐ │ ┌─Consumer #2─┐
│ x x x │ │ │ ┌───┐ │
│ ┌───┐ ┌───┐ ┌───┐ ┌─x─┐ ┌───┐ ┌─x─┐ ┌─x─┐ │ │ │ │ 2 │ ├───► Permanent -> X
│ n+1 │ n │ ... │ 6 │ │ 5 │ │ 4 │ │ 3 │ │ 2 │ │ 1 │ ├────┼───►│ └───┘ │ failure
│ └───┘ └───┘ └───┘ └───┘ └───┘ └───┘ └───┘ │ │ │ │
│ │ │ └─────────────┘
└───────────────────────────────────────────────────┘ │
▲ ▲ ▲ ▲ │ ┌─Consumer #3─┐
│ │ │ │ │ │ ┌───┐ │
│ │ │ │ │ │ │ 3 │ ├───► (in progress)
write read └─────┬─────┘ ├───►│ └───┘ │
index index │ │ │ │
│ │ └─────────────┘
│ │
currently │ ┌─Consumer #4─┐
dispatched │ │ ┌───┐ │ Temporary
└───►│ │ 4 │ ├───► failure
│ └───┘ │ │
│ │ │
└─────────────┘ │
▲ │
└── Retry ───────┤
│
│
X ◄────── Retry limit exceeded ───┘
Example:
receivers:
otlp:
protocols:
grpc:
exporters:
otlp:
endpoint: <ENDPOINT>
sending_queue:
storage: file_storage/otc
extensions:
file_storage/otc:
directory: /var/lib/storage/otc
timeout: 10s
service:
extensions: [file_storage]
pipelines:
metrics:
receivers: [otlp]
exporters: [otlp]
logs:
receivers: [otlp]
exporters: [otlp]
traces:
receivers: [otlp]
exporters: [otlp]