> ## Knowledge Base Index
> Fetch the complete knowledge base index at: https://helpdesk.ggcircuit.com/sitemap.xml
> Use this file to discover available pages before exploring further.
> Pure-Markdown content can be obtained by appending a '.md' suffix to the content URLs listed in the sitemap (without the trailing slash).

# Grafana shows N/A and Prometheus won't stay running (duplicate command-line flag)

# Grafana shows N/A and Prometheus won't stay running (duplicate command-line flag)
**When Prometheus exits on startup with a "flag cannot be repeated" error, Grafana panels show N/A or no data. This is usually caused by the same flag being set in two places. This article shows how to confirm it and fix it so it survives future updates.**

## Symptoms
* Grafana panels show **N/A** or display no statistics.
* The Prometheus service won't stay running; `systemctl status prometheus` shows it failed.
* The service log shows a restart loop ending in a start-limit message:
```
Error parsing commandline arguments: flag 'storage.tsdb.retention.size' cannot be repeated
prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
prometheus.service: Scheduled restart job, restart counter is at 5.
prometheus.service: Start request repeated too quickly.
Failed to start Monitoring system and time series database.
```
Prometheus also dumps its full `--help` output on a parse error; that text is noise — the line that matters is **"cannot be repeated."** Grafana shows N/A because Prometheus never starts, so the data source returns nothing.
|| The flag name in the error can vary. The example here is `storage.tsdb.retention.size`, but the same failure applies to any non-repeatable flag that is set twice.

## Why it happens
Prometheus builds its command line from **two** sources:
* The systemd unit's `ExecStart`
* The expanded `$ARGS` (or similar variable) from its environment file, typically `/etc/default/prometheus`
Most flags can appear only once. If a package update adds a flag to the unit's `ExecStart` that is **already** set in `ARGS`, the assembled command line contains it twice, and Prometheus exits with status 2 **before** it opens its database. systemd retries, hits the start limit, and gives up.

## Confirm the cause
```
# Show the unit plus any drop-in overrides (each with a "# path" header)
systemctl cat prometheus

# Show the admin args
cat /etc/default/prometheus

# Confirm the offending flag
journalctl -u prometheus -n 50 --no-pager
```
Look for the **same flag in both** the unit's `ExecStart` line and the `ARGS` value. For example, the unit may hardcode `--storage.tsdb.retention.size=512MB` while `ARGS` already sets `--storage.tsdb.retention.size=15GB`.

## Fix
The goal is to leave the flag set **once**, keeping the value you actually want. Where you make the change determines whether it survives the next update:
| Location | Type | Edit here? |
| ---- |
| `/lib/systemd/system/prometheus.service` | Package-managed unit | **No** — overwritten on update |
| `/etc/default/prometheus` (`ARGS`) | Admin config | Yes |
| `/etc/systemd/system/prometheus.service.d/override.conf` | Drop-in override | Yes — survives updates |
||| Decide which value to keep before editing. `storage.tsdb.retention.size` is enforced at startup — if the smaller value wins, Prometheus will trim the database and delete old blocks. Keep the value you intend (usually the one in `ARGS`).
The cleanest, update-safe fix is a **drop-in override** that removes the hardcoded flag from `ExecStart` and lets `ARGS` be authoritative:
```
systemctl edit prometheus
```
Add exactly this:
```
[Service]
ExecStart=
ExecStart=/usr/bin/prometheus $ARGS
```
The empty `ExecStart=` clears the inherited line; the second line redefines it **without** the hardcoded flag. The single flag remaining in `ARGS` now applies. This writes to `/etc/systemd/system/prometheus.service.d/override.conf`, which package updates do not touch.
|| Alternatively, you can delete the duplicate flag directly from the unit's `ExecStart`. This works immediately, but the next package update will overwrite the unit and re-introduce the duplicate. The drop-in override avoids that.

## Apply and verify
```
systemctl daemon-reload
systemctl reset-failed prometheus
systemctl restart prometheus
systemctl --no-pager status prometheus
```
| `reset-failed` clears the start-limit lockout left by the repeated failed starts. Without it, the service may refuse to restart even after the fix.
Confirm Prometheus is actually serving. Use **its configured listen address** — check `--web.listen-address` in `ARGS` (the ggRock default is often `localhost:5010`, not the upstream default `9090`):
```
curl -s http://localhost:5010/-/healthy
curl -s 'http://localhost:5010/api/v1/targets?state=active' | head
```
A response of `Prometheus is Healthy.` and active targets means Grafana panels will repopulate on the next scrape.

## Notes
||| There will be a data gap for the entire outage window. Prometheus stores only what it scrapes live and **cannot backfill** missed time. Data from before the failure is intact; expect a gap covering the downtime.
* The same duplicate can be introduced by any update that rewrites a managed config. After a ggRock or distribution update, re-check `systemctl cat prometheus` against `/etc/default/prometheus` if the service fails to start.
* The drop-in override pattern (`ExecStart=` then a redefined `ExecStart=`) works for removing any unwanted hardcoded flag from a package-managed unit, not just this one.