loggingexporter: Error: failed to shutdown service: failed to shutdown pipelines: failed to shutdown exporters: sync /dev/stderr: bad file descriptor

[pmalek-sumo]

Describe the bug

When trying to run collector using the following piece of code:

import (
	"net"
	"testing"
	"time"

	"github.com/stretchr/testify/require"
	"go.opentelemetry.io/collector/component"
	"go.opentelemetry.io/collector/service"
)

func TestConfig(t *testing.T) {
	// components() generated with opentelemetry builder
	factories, err := components()
	require.NoError(t, err)

	t.Log("Creating new app...")
	app, err := service.New(service.CollectorSettings{
		BuildInfo: component.DefaultBuildInfo(),
		Factories: factories,
	})
	require.NoError(t, err)

	app.Command().Root().SetArgs(
		[]string{
			"--config=testdata/filelog.yaml",
			"--metrics-addr=" + GetAvailableLocalAddress(t),
			"--log-level=debug",
		},
	)

	go func() {
		<-time.NewTimer(3 * time.Second).C
		t.Log("Calling .Shutdown() on the app...")
		app.Shutdown()
	}()

	t.Log("Calling .Run() on the app...")
	require.NoError(t, app.Run())
}

one can observe the following error:

2021-09-30T17:11:41.596+0200    info    service/service.go:152  Stopping extensions...
2021-09-30T17:11:41.596+0200    info    service/collector.go:353        Shutdown complete.
Error: failed to shutdown service: failed to shutdown pipelines: failed to shutdown exporters: sync /dev/stderr: bad file descriptor

Steps to reproduce

Use the above piece of code

What did you expect to see?

No error.

What did you see instead?

bad file descriptor error

What version did you use?
Version: 2116719 vel v0.35.0

What config did you use?

The test load the following config:

receivers:
  filelog:
    include: [ "*_dummy.log" ]
    start_at: end

processors:
  attributes/123:
    actions:
      - action: insert
        key: host
        value: 123

exporters:
  logging:
    loglevel: debug

service:
  pipelines:
    logs:
      receivers:
      - filelog
      processors:
      - attributes/123
      exporters:
      - logging

Environment
OS: mac
Compiler(if manually compiled): "go 1.17.1"

Additional context

When debugging the above mentioned piece of code with e.g. delve I do not get the error because the underlying error is
unix.ENOTTY which is in the set of known sync errors

(dlv) n
> go.opentelemetry.io/collector/exporter/loggingexporter.loggerSync.func1() /Users/pmalek/.gvm/pkgsets/go1.17.1/global/pkg/mod/go.opentelemetry.io/[email protected]/exporter/loggingexporter/logging_exporter.go:154 (PC: 0x49f37d1)
   149:              // Since these are not actionable ignore them.
   150:              err := logger.Sync()
   151:              if osErr, ok := err.(*os.PathError); ok {
   152:                      wrappedErr := osErr.Unwrap()
   153:                      if knownSyncError(wrappedErr) {
=> 154:                              err = nil
   155:                      }
   156:              }
   157:              return err
   158:      }
   159: }
(dlv) p osErr
*io/fs.PathError {
        Op: "sync",
        Path: "/dev/stderr",
        Err: error(syscall.Errno) golang.org/x/sys/unix.ENOTTY (25),}

My proposal for this issue is to add syscall.EBADF to knownSyncError() (unless there's someone to point out a better way to approach this).

[bogdandrutu]

@pmalek-sumo can you run with the latest version?

[pmalek-sumo]

Ok so ... with the following go.mod

module github.com/SumoLogic/opentelemetry-collector-builder

go 1.17

require (
	github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage v0.36.1-0.20210929182127-182b5c2a7138
	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/filelogreceiver v0.36.1-0.20210929182127-182b5c2a7138
	github.com/stretchr/testify v1.7.0
	go.opentelemetry.io/collector v0.36.1-0.20210927193005-ebb0fbd6f23e
)

require (
.... deps deps deps
)

replace go.opentelemetry.io/collector => go.opentelemetry.io/collector v0.36.1-0.20210927193005-ebb0fbd6f23e

and the following collector config

receivers:
  filelog:
    include: [ "*_dummy.log" ]
    start_at: end

exporters:
  logging:
    loglevel: debug

service:
  pipelines:
    logs:
      receivers:
      - filelog
      exporters:
      - logging

I get the same result with the adjusted code for newest API

func TestX(t *testing.T) {
	factories, err := components()
	require.NoError(t, err)

	t.Log("Creating new app...")
	col, err := service.New(service.CollectorSettings{
		BuildInfo: component.NewDefaultBuildInfo(),
		Factories: factories,
	})
	require.NoError(t, err)

	cmd := service.NewCommand(col)
	cmd.Root().SetArgs(
		[]string{
			"--config=testdata/filelog.yaml",
			"--metrics-addr=" + GetAvailableLocalAddress(t),
			"--log-level=debug",
		},
	)

	go func() {
		<-time.NewTimer(1 * time.Second).C
		t.Log("Calling .Shutdown() on the app...")
		col.Shutdown()
	}()

	t.Log("Calling .Run() on the app...")
	require.NoError(t, cmd.ExecuteContext(context.Background()))
}

[mx-psi]

I can reproduce this on macOS on the Datadog Agent (which uses the logging exporter). Opened #5585 to fix this with the approach proposed by @pmalek-sumo, since I haven't been able to find a better solution. Note that the correct error code seems to be syscall.EBADF ("Bad file descriptor") instead of syscall.EBADFD ("File descriptor in bad state"). See here for details.