[BUG] Failed to start datadog-agent-installer.service - Datadog Installer.

[dwhjames]

Agent version

7.72.2

Bug Report

The systemd service datadog-agent-installer.service fails on startup.

journalctl -u datadog-agent-installer.service

Nov 13 22:05:53 ip-10-156-68-100 systemd[1]: datadog-agent-installer.service: Scheduled restart job, restart counter is at 4.
Nov 13 22:05:53 ip-10-156-68-100 systemd[1]: Stopped datadog-agent-installer.service - Datadog Installer.
Nov 13 22:05:53 ip-10-156-68-100 systemd[1]: Started datadog-agent-installer.service - Datadog Installer.
Nov 13 22:05:53 ip-10-156-68-100 agent[21295]: {"agent":"installer","time":"2025-11-13T22:05:53Z","level":"DEBUG","file":"pkg/util/log/log.go","line":"696","func":"func1","msg":"Executable folder is /opt/datadog-agent/bin/agent"}
Nov 13 22:05:53 ip-10-156-68-100 agent[21295]: {"agent":"installer","time":"2025-11-13T22:05:53Z","level":"INFO","file":"pkg/util/log/log.go","line":"736","func":"func1","msg":"Using '/opt/datadog-agent/embedded' as Python home"}
Nov 13 22:05:53 ip-10-156-68-100 agent[21295]: {"agent":"installer","time":"2025-11-13T22:05:53Z","level":"INFO","file":"pkg/util/log/log.go","line":"731","func":"func1","msg":"Starting to load the configuration"}
Nov 13 22:05:53 ip-10-156-68-100 agent[21295]: {"agent":"installer","time":"2025-11-13T22:05:53Z","level":"WARN","file":"pkg/util/log/log.go","line":"776","func":"func1","msg":"Unknown environment variable: DD_BUNDLED_AGENT"}
Nov 13 22:05:53 ip-10-156-68-100 agent[21295]: {"agent":"installer","time":"2025-11-13T22:05:53Z","level":"WARN","file":"pkg/util/log/log.go","line":"776","func":"func1","msg":"Unknown environment variable: DD_PROFILING_STACK_V2_ENABLED"}
Nov 13 22:05:53 ip-10-156-68-100 agent[21295]: {"agent":"installer","time":"2025-11-13T22:05:53Z","level":"INFO","file":"pkg/util/log/log.go","line":"731","func":"func1","msg":"Loading proxy settings"}
Nov 13 22:05:53 ip-10-156-68-100 agent[21295]: {"agent":"installer","time":"2025-11-13T22:05:53Z","level":"DEBUG","file":"pkg/util/log/log.go","line":"696","func":"func1","msg":"'use_proxy_for_cloud_metadata' is enabled: adding cloud provider URL >
Nov 13 22:05:53 ip-10-156-68-100 agent[21295]: {"agent":"installer","time":"2025-11-13T22:05:53Z","level":"DEBUG","file":"pkg/util/log/log.go","line":"696","func":"func1","msg":"Updating setting 'proxy.http' for source 'agent-runtime' with the sam>
Nov 13 22:05:53 ip-10-156-68-100 agent[21295]: {"agent":"installer","time":"2025-11-13T22:05:53Z","level":"DEBUG","file":"pkg/util/log/log.go","line":"696","func":"func1","msg":"Updating setting 'proxy.https' for source 'agent-runtime' with the sa>
Nov 13 22:05:53 ip-10-156-68-100 agent[21295]: {"agent":"installer","time":"2025-11-13T22:05:53Z","level":"DEBUG","file":"pkg/util/log/log.go","line":"696","func":"func1","msg":"Updating setting 'proxy.no_proxy' for source 'agent-runtime' with new>
Nov 13 22:05:53 ip-10-156-68-100 agent[21295]: {"agent":"installer","time":"2025-11-13T22:05:53Z","level":"INFO","file":"pkg/util/log/log.go","line":"731","func":"func1","msg":"Starting to resolve secrets"}
Nov 13 22:05:53 ip-10-156-68-100 agent[21295]: {"agent":"installer","time":"2025-11-13T22:05:53Z","level":"INFO","file":"pkg/util/log/log.go","line":"731","func":"func1","msg":"Finished resolving secrets"}
Nov 13 22:05:53 ip-10-156-68-100 agent[21295]: {"agent":"installer","time":"2025-11-13T22:05:53Z","level":"DEBUG","file":"pkg/util/log/log.go","line":"696","func":"func1","msg":"Updating setting 'api_key' for source 'file' with the same value, ski>
Nov 13 22:05:53 ip-10-156-68-100 agent[21295]: {"agent":"installer","time":"2025-11-13T22:05:53Z","level":"DEBUG","file":"pkg/util/log/log.go","line":"691","func":"func1","msg":"FIPS mode is disabled"}
Nov 13 22:05:53 ip-10-156-68-100 agent[21295]: {"agent":"installer","time":"2025-11-13T22:05:53Z","level":"INFO","file":"pkg/util/log/log.go","line":"736","func":"func1","msg":"Agent did not find PodResources socket at /var/lib/kubelet/pod-resourc>
Nov 13 22:05:53 ip-10-156-68-100 agent[21295]: {"agent":"installer","time":"2025-11-13T22:05:53Z","level":"DEBUG","file":"pkg/util/log/log.go","line":"696","func":"func1","msg":"Agent did not find NVML library: could not locate libnvidia-ml.so.1: >
Nov 13 22:05:53 ip-10-156-68-100 agent[21295]: {"agent":"installer","time":"2025-11-13T22:05:53Z","level":"INFO","file":"pkg/util/log/log.go","line":"736","func":"func1","msg":"0 Features detected from environment: "}
Nov 13 22:05:53 ip-10-156-68-100 agent[21295]: {"agent":"installer","time":"2025-11-13T22:05:53Z","level":"WARN","file":"pkg/util/log/setup/log.go","line":"83","func":"SetupLogger","msg":"config key \"installer.log_file\" is unknown"}
Nov 13 22:05:53 ip-10-156-68-100 agent[21295]: {"agent":"installer","time":"2025-11-13T22:05:53Z","level":"INFO","file":"comp/core/pid/pidimpl/pid.go","line":"54","func":"newPID","msg":"pid '21295' written to pid file '/opt/datadog-agent/run/insta>
Nov 13 22:05:53 ip-10-156-68-100 agent[21295]: Error: remote config is required to create the updater
Nov 13 22:05:53 ip-10-156-68-100 systemd[1]: datadog-agent-installer.service: Main process exited, code=exited, status=255/EXCEPTION
Nov 13 22:05:53 ip-10-156-68-100 systemd[1]: datadog-agent-installer.service: Failed with result 'exit-code'.
Nov 13 22:05:54 ip-10-156-68-100 systemd[1]: datadog-agent-installer.service: Scheduled restart job, restart counter is at 5.
Nov 13 22:05:54 ip-10-156-68-100 systemd[1]: Stopped datadog-agent-installer.service - Datadog Installer.
Nov 13 22:05:54 ip-10-156-68-100 systemd[1]: datadog-agent-installer.service: Start request repeated too quickly.
Nov 13 22:05:54 ip-10-156-68-100 systemd[1]: datadog-agent-installer.service: Failed with result 'exit-code'.
Nov 13 22:05:54 ip-10-156-68-100 systemd[1]: Failed to start datadog-agent-installer.service - Datadog Installer.

Error comes from here

https://github.com/DataDog/datadog-agent/blob/7.72.2/comp/updater/updater/updaterimpl/updater.go#L48-L50

I had wondered if the following code should have handled this
https://github.com/DataDog/datadog-agent/blob/7.72.2/cmd/installer/subcommands/daemon/run_nix.go#L46-L51

Note:

"msg":"Unknown environment variable: DD_BUNDLED_AGENT"

which seems surprising that the process running in the service does not recognize the env var the service file defines for it.

Reproduction Steps

No response

Agent configuration

/etc/datadog-agent/datadog.yaml has

…<snip>…

remote_configuration:
  enabled: false
remote_updates: false

…<snip>…

/lib/systemd/system/datadog-agent-installer.service matches

• https://github.com/DataDog/datadog-agent/blob/7.72.2/pkg/fleet/installer/packages/embedded/templates/gen/debrpm/datadog-agent-installer.service

Operating System

Debian 6.1.148-1 (2025-08-26) x86_64 GNU/Linux

Other environment details

No response

[arbll]

Hey @dwhjames,

There’s definitely a missing cleanup in the systemd unit that’s causing that noisy log line. We’ll get that fixed in the next release.

Just to clarify, that’s not what’s preventing the service from starting. The real issue is: Error: remote config is required to create the updater. The installer service is responsible for remotely upgrading or modifying your agent configuration from the Fleet Automation UI. Since those features are disabled in your setup, the service won’t start as expected.

Let me know if you have any follow-up questions.

Cheers,
Arthur

[dwhjames]

Hi @arbll

The real issue is: Error: remote config is required to create the updater. The installer service is responsible for remotely upgrading or modifying your agent configuration from the Fleet Automation UI. Since those features are disabled in your setup, the service won’t start as expected.

It had seemed like in some codepaths, e.g.,

https://github.com/DataDog/datadog-agent/blob/7.72.2/cmd/installer/subcommands/daemon/run_nix.go#L46-L51

there is an attempt to gracefully exit if a feature is not configured for use.

So, maybe not a bug in the strictness sense, but given the intent in the code, I was expecting/hoping that the installer would indeed gracefully terminate, rather than error out here

https://github.com/DataDog/datadog-agent/blob/7.72.2/comp/updater/updater/updaterimpl/updater.go#L48-L50