Clarify how Receivers must handle errors returned by Consume*() funct…

…ions (open-telemetry#4267)

This is important to make sure failed Consume*() calls are correctly handled
by Receivers and that the Receives in turn correctly respond back to the senders
from which they received data.

We may need to audit all receivers to make sure they work according to these
requirements.

component/receiver.go

@@ -23,7 +23,34 @@ import (
 
 // Receiver allows the collector to receive metrics, traces and logs.
 //
-// The receivers that receive data via a protocol that support acknowledgements
+// Receiver receives data from a source (either from a remote source via network
+// or scrapes from a local host) and pushes the data to the pipelines it is attached
+// to by calling the nextConsumer.Consume*() function.
+//
+// Error Handling
+// ==============
+// The nextConsumer.Consume*() function may return an error to indicate that the data
+// was not accepted. There are 2 types of possible errors: Permanent and non-Permanent.
+// The receiver must check the type of the error using IsPermanent() helper.
+//
+// If the error is Permanent than the nextConsumer.Consume*() call should not be
+// retried with the same data. This typically happens when the data cannot be
+// serialized by the exporter that is attached to the pipeline or when the destination
+// refuses the data because it cannot decode it. The receiver must indicate to
+// the source from which it received the data that the received data was bad, if the
+// receiving protocol allows to do that. In case of OTLP/HTTP for example, this means
+// that HTTP 400 response is returned to the sender.
+//
+// If the error is non-Permanent then the nextConsumer.Consume*() call may be retried
+// with the same data. This may be done by the receiver itself, however typically it is
+// done by the original sender, after the receiver returns a response to the sender
+// indicating that the Collector is currently overloaded and the request must be
+// retried. In case of OTLP/HTTP for example, this means that HTTP 429 or 503 response
+// is returned.
+//
+// Acknowledgment Handling
+// =======================
+// The receivers that receive data via a network protocol that support acknowledgments
 // MUST follow this order of operations:
 // - Receive data from some sender (typically from a network).
 // - Push received data to the pipeline by calling nextConsumer.Consume*() function.