Grafana shows N/A and Prometheus won't stay running (duplicate command-line flag)

When Prometheus exits on startup with a "flag cannot be repeated" error, Grafana panels show N/A or no data. This is usually caused by the same flag being set in two places. This article shows how to confirm it and fix it so it survives future updates.

Symptoms

• Grafana panels show N/A or display no statistics.
• The Prometheus service won't stay running; systemctl status prometheus shows it failed.
• The service log shows a restart loop ending in a start-limit message:

Error parsing commandline arguments: flag 'storage.tsdb.retention.size' cannot be repeated
prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
prometheus.service: Scheduled restart job, restart counter is at 5.
prometheus.service: Start request repeated too quickly.
Failed to start Monitoring system and time series database.

Prometheus also dumps its full --help output on a parse error; that text is noise — the line that matters is "cannot be repeated." Grafana shows N/A because Prometheus never starts, so the data source returns nothing.

Why it happens

Prometheus builds its command line from two sources:

• The systemd unit's ExecStart
• The expanded $ARGS (or similar variable) from its environment file, typically /etc/default/prometheus
  Most flags can appear only once. If a package update adds a flag to the unit's ExecStart that is already set in ARGS, the assembled command line contains it twice, and Prometheus exits with status 2 before it opens its database. systemd retries, hits the start limit, and gives up.

Confirm the cause

# Show the unit plus any drop-in overrides (each with a "# path" header)
systemctl cat prometheus

# Show the admin args
cat /etc/default/prometheus

# Confirm the offending flag
journalctl -u prometheus -n 50 --no-pager

Look for the same flag in both the unit's ExecStart line and the ARGS value. For example, the unit may hardcode --storage.tsdb.retention.size=512MB while ARGS already sets --storage.tsdb.retention.size=15GB.

Fix

The goal is to leave the flag set once, keeping the value you actually want. Where you make the change determines whether it survives the next update:

systemctl edit prometheus

Add exactly this:

[Service]
ExecStart=
ExecStart=/usr/bin/prometheus $ARGS

The empty ExecStart= clears the inherited line; the second line redefines it without the hardcoded flag. The single flag remaining in ARGS now applies. This writes to /etc/systemd/system/prometheus.service.d/override.conf, which package updates do not touch.

Apply and verify

systemctl daemon-reload
systemctl reset-failed prometheus
systemctl restart prometheus
systemctl --no-pager status prometheus

Confirm Prometheus is actually serving. Use its configured listen address — check --web.listen-address in ARGS (the ggRock default is often localhost:5010, not the upstream default 9090):

curl -s http://localhost:5010/-/healthy
curl -s 'http://localhost:5010/api/v1/targets?state=active' | head

A response of Prometheus is Healthy. and active targets means Grafana panels will repopulate on the next scrape.

Notes

• The same duplicate can be introduced by any update that rewrites a managed config. After a ggRock or distribution update, re-check systemctl cat prometheus against /etc/default/prometheus if the service fails to start.
• The drop-in override pattern (ExecStart= then a redefined ExecStart=) works for removing any unwanted hardcoded flag from a package-managed unit, not just this one.