Articles on: ggRock

Grafana shows N/A and Prometheus won't stay running (duplicate command-line flag)

When Prometheus exits on startup with a "flag cannot be repeated" error, Grafana panels show N/A or no data. This is usually caused by the same flag being set in two places. This article shows how to confirm it and fix it so it survives future updates.

Symptoms

Grafana panels show N/A or display no statistics.
The Prometheus service won't stay running; systemctl status prometheus shows it failed.
The service log shows a restart loop ending in a start-limit message:

Error parsing commandline arguments: flag 'storage.tsdb.retention.size' cannot be repeated
prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
prometheus.service: Scheduled restart job, restart counter is at 5.
prometheus.service: Start request repeated too quickly.
Failed to start Monitoring system and time series database.

Prometheus also dumps its full --help output on a parse error; that text is noise — the line that matters is "cannot be repeated." Grafana shows N/A because Prometheus never starts, so the data source returns nothing.

The flag name in the error can vary. The example here is storage.tsdb.retention.size, but the same failure applies to any non-repeatable flag that is set twice.

Why it happens

Prometheus builds its command line from two sources:

The systemd unit's ExecStart
The expanded $ARGS (or similar variable) from its environment file, typically /etc/default/prometheus

Most flags can appear only once. If a package update adds a flag to the unit's ExecStart that is already set in ARGS, the assembled command line contains it twice, and Prometheus exits with status 2 before it opens its database. systemd retries, hits the start limit, and gives up.

Confirm the cause

# Show the unit plus any drop-in overrides (each with a "# path" header)
systemctl cat prometheus

# Show the admin args
cat /etc/default/prometheus

# Confirm the offending flag
journalctl -u prometheus -n 50 --no-pager

Look for the same flag in both the unit's ExecStart line and the ARGS value. For example, the unit may hardcode --storage.tsdb.retention.size=512MB while ARGS already sets --storage.tsdb.retention.size=15GB.

Fix

The goal is to leave the flag set once, keeping the value you actually want. Where you make the change determines whether it survives the next update:

Location	Type	Edit here?
`/lib/systemd/system/prometheus.service`	Package-managed unit	No — overwritten on update
`/etc/default/prometheus` (`ARGS`)	Admin config	Yes
`/etc/systemd/system/prometheus.service.d/override.conf`	Drop-in override	Yes — survives updates

Decide which value to keep before editing. storage.tsdb.retention.size is enforced at startup — if the smaller value wins, Prometheus will trim the database and delete old blocks. Keep the value you intend (usually the one in ARGS).

The cleanest, update-safe fix is a drop-in override that removes the hardcoded flag from ExecStart and lets ARGS be authoritative:

systemctl edit prometheus

Add exactly this:

[Service]
ExecStart=
ExecStart=/usr/bin/prometheus $ARGS

The empty ExecStart= clears the inherited line; the second line redefines it without the hardcoded flag. The single flag remaining in ARGS now applies. This writes to /etc/systemd/system/prometheus.service.d/override.conf, which package updates do not touch.

Alternatively, you can delete the duplicate flag directly from the unit's ExecStart. This works immediately, but the next package update will overwrite the unit and re-introduce the duplicate. The drop-in override avoids that.

Apply and verify

systemctl daemon-reload
systemctl reset-failed prometheus
systemctl restart prometheus
systemctl --no-pager status prometheus

reset-failed clears the start-limit lockout left by the repeated failed starts. Without it, the service may refuse to restart even after the fix.

Confirm Prometheus is actually serving. Use its configured listen address — check --web.listen-address in ARGS (the ggRock default is often localhost:5010, not the upstream default 9090):

curl -s http://localhost:5010/-/healthy
curl -s 'http://localhost:5010/api/v1/targets?state=active' | head

A response of Prometheus is Healthy. and active targets means Grafana panels will repopulate on the next scrape.

Notes

There will be a data gap for the entire outage window. Prometheus stores only what it scrapes live and cannot backfill missed time. Data from before the failure is intact; expect a gap covering the downtime.

The same duplicate can be introduced by any update that rewrites a managed config. After a ggRock or distribution update, re-check systemctl cat prometheus against /etc/default/prometheus if the service fails to start.
The drop-in override pattern (ExecStart= then a redefined ExecStart=) works for removing any unwanted hardcoded flag from a package-managed unit, not just this one.

Updated on: 26/06/2026

Was this article helpful?

Thank you!