Tempo Destination Error with Invalid Port

[jlimai]

Describe the bug
We have qRYN self-hosted installed. We are able to send Logs and Metrics, but failing with Tempo.
This is related to issue #1545, the fix solved the URL port parsing, but we are getting another error.

Error: invalid configuration exporters:otlp/tempo-odigos-qryn: invalid port "3100/tempo/api/push"

We successfully sent the following curl request to qRYN Tempo, which worked as expected. Therefore, we believe the issue may be with Odigos.

curl -X POST http://qryn.ns.svc.cluster.local:3100/tempo/api/push -H 'Content-Type: application/json' -d '[{
 "id": "1234",
 "traceId": "d6e9329d67b6146b",
 "timestamp": '$(date +%s%N | cut -b1-16)',
 "duration": 1000,
 "name": "1st span from bash!",
 "tags": {
    "http.method": "POST",
    "http.path": "/parent"
  },
  "localEndpoint": {
    "serviceName": "shell script child 1"
  }
}]'

To Reproduce
Open Odigos UI, add a Tempo destination. Add application to source and wait for data to be sent to Tempo endpoint. We got error from Odigos Gateway

Expected behavior
Expected Odigos to honor the Tempo endpoint destination as configured.

Additional context
Odigos version: 1.0.109
EKS version: 1.25

[jlimai]

@alonkeyval , thanks for your earlier fix on issue #1545, but after applying the latest version, we got this error.

[alonkeyval]

Hey @jlimai, thanks again for the feedback. We will take a look and update you when it's fixed.

[yodigos]

Hey @jlimai ,
Can you please share the Service of your Tempo Destination and the endpoint that you fill?
I see in Tempo the common way is to use otpl protocol with port 4317/4318 - grpc/http.
With Loki you usually use port 3100 and this endpoint to push traces.
Can you elaborate a bit more on your use-case.

[jlimai]

We are using qRYN as our storage backend. We have this 3 endpoints expose from qRYN

• Traces: http://qryn:3100/tempo/api/push
• Metrics: http://qryn:3100/api/v1/prom/remote/write
• Logs: http://qryn:3100/loki/api/v1/push

All our applications metrics and traces are sending to Odigos 4317 gRPC receiver and I believe logs are scrape through filelog. We then expect Odigos to process and export it to the backend. We configure the Odigos destination with the above endpoint. We saw logs and metrics. Just tempo is failing with the error I described earlier. Let me know if I'm missing anything in our setup. Thanks

[yodigos]

Hey @jlimai ,
Based on qryn documentation it appears that the endpoint you're using for traces /tempo/api/push is expecting Tempo formatted Traces, while looking on the source code it appears that actually Zipkin formatted data .
We should address this in Odigos and update the documentation.

Tempo Destination in Odigos uses OTLP gRPC.
In the meantime, please try using the OTLP-http destination in Odigos and send traces to /v1/traces endpoint instead of tempo/api/push.

In the future addition, as you know we currently support qryn Cloud, we soon will work to add qryn self-managed.

If you encounter any further issues, feel free to reach out to me directly (Yoav Levy) on the Odigos community Slack.

[jlimai]

@yodigos thanks for the quick turnaround and suggestion. I will give it a try. Thanks

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.