[BUG] OTEL Trace submission in ECS Fargate tasks - unable to connect to localhost:5003

[mlindes]

Agent version

7.72.2 / 7.74.0-devel

Bug Report

When DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT is set on the datadog-agent container, the agent tries to connect to port 5003 but can not. The following error logs and traces do not get sent:

2025-11-14 22:55:38 UTC | CORE | ERROR | (exporterhelper@v0.138.0/internal/base_exporter.go:114 in Send) | service.instance.id:c064b890-a9d8-4739-a017-d8d52d951300,service.name:agent,service.version:7.74.0-devel+git.204.3828d01,otelcol.component.id:otlp,otelcol.component.kind:exporter,otelcol.signal:traces,error:request will be cancelled before next retry: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:5003: connect: connection refused",rejected_items:1 | Exporting failed. Rejecting data. Try enabling sending_queue to survive temporary failures.

Similarly, with the agent running as a sidecar to an application written with the opentelemetry-go library are unable to send to the datadog agent when this variable is set. They too pick up on the 5003 port as it must be exported by the datadog agent. With

Reproduction Steps

If the DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT variable is set, the 5003 connection errors log. If the variable is not set, I can't figure out how to send traces from an otel client.
I was running public.ecr.aws/datadog/agent:latest of the datadog agent but updated to 7-beta to test. No change.

Agent configuration

# agent config
api_key: '***************************...'
apm_config:
  enabled: true
  receiver_port: 8126
container_exclude:
  - name:datadog-agent
dogstatsd_non_local_traffic: true
env: dev
logs_enabled: false
otlp_config:
  receiver.protocols.grpc.endpoint: 0.0.0.0:4317
process_config:
  container_collection:
    enabled: false
  enabled: "true"
  process_collection:
    enabled: true
secret_backend_command_allow_group_exec_perm: true
security_agent:
  internal_profiling:
    api_key: '***************************...'
    env: dev
    site: datadoghq.com
site: datadoghq.com
tags:
  - env:dev
  - service:test-service

Operating System

Linux ip-96-103-146-197.ec2.internal 5.10.245-241.976.amzn2.aarch64 #1 SMP Tue Oct 21 22:09:01 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux

Other environment details

AWS ECS Fargate

[liustanley]

Hi @mlindes, port 5003 is the default internal port used by the trace agent OTLP receiver. Are you running the trace agent alongside the core agent?

[mlindes]

I'm using the Datadog agent as the trace agent. Maybe I don't understand your question.
The Datadog agent has an OTEL Agent that, by default, listens on port 4317.. set with DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT env var. If this variable is set, the datadog agent tries to connect to port 5003 - but nothing listening there.