Upgrading to Debian 13 (From Debian 11 or Debian 12)

ggCircuit provides an upgrade script that moves a ggRock server from Debian 11 (Bullseye) or Debian 12 (Bookworm) to Debian 13 (Trixie). It upgrades one release at a time, reboots, and carries on by itself until it reaches Trixie. This article covers how to prepare, run the upgrade, check the result, and what to do if something stops.

At a glance

• Works from: Debian 11 (Bullseye) or Debian 12 (Bookworm). To see which you have, run grep VERSION_CODENAME /etc/os-release.
• Time: about 40 to 75 minutes from Bullseye (two steps) and about 20 to 30 minutes from Bookworm (one step). Servers with slow storage, many old kernels or long hardware checks at startup take longer.
• Reboots: one at the end of each step.
• You need: root (or sudo) on the server, 10 GB free on /, and internet access from the server.
• Your terminal: you can close it. The upgrade runs on the server and does not depend on your connection.

Before you start

1. Pick a window and shut the stations down. Tell anyone who uses the center.

2. Make sure you can reach the server if its network drops. Use IPMI, iDRAC, iLO, a hypervisor console, or a keyboard and monitor. Test that it actually works before you begin, not just that you have the password.

3. Check for held packages. The result should be empty. If packages are listed, the script stops before it changes anything and prints the exact command to clear them. These holds usually come from the temporary workaround ggCircuit support gave for the end of Debian 11 updates, and clearing them before the upgrade is the intended next step.

   apt-mark showhold
   

4. Check for virtual machines. The script shuts running VMs down cleanly (up to 2 minutes each) and stops if one will not shut down. It never forces a VM off, and it never restarts them afterwards. Warn anyone who depends on them.

   virsh list --all
   

5. Back up the ggRock database. Run both lines as root, then copy the file off the server. If the server is a virtual machine, take a snapshot too.

   pg_lsclusters
   

   sudo -u postgres pg_dumpall -p 5432 > /root/pre-upgrade-backup-$(date +%Y%m%d%H%M%S).sql
   

6. Leave Debian 11 alone until you run the script. Do not run apt upgrade, apt full-upgrade, the ggRock installer or the ggRock configurator on a Debian 11 server yourself. Debian 11 updates have ended and its security repository no longer carries some packages, so a manual upgrade can fail partway and remove packages ggRock needs. The script is built to handle this.

7. Check free space. The script needs 10 GB free on /. If root is on LVM and the volume group has room, it grows root for you. If not, it stops and tells you how to expand it.

   df -h /
   

Run the upgrade

1. Connect over SSH, or use the server console, and become root.

   sudo -i
   

2. Start the upgrade.

   wget -O - https://images.ggleap.com/ManagedServices/Scripts/ggrock-upgrade-debian.sh | bash -
   

3. Check the Current release and Next hop the script shows, then answer y.

4. A progress bar appears in your terminal and updates until the server reboots. The percentage covers the whole upgrade: hop 1/2 means step 1 of 2, so the bar reaches 50% when the first step finishes, and a Bookworm server has one step and runs from 0 to 100%. Ctrl-C or closing the terminal only stops the display. The upgrade keeps running.

   [##############------------------]  41%  hop 1/2  12m08s  Upgrading packages (full-upgrade): Unpacking libssl3
   

Watch from another session

Open a second session as root and use any of these.

• Follow the full log:

  tail -f /var/log/ggrock-upgrade-debian.log
  

• Follow only the progress lines (they look like [ 46%] hop 1/2: ...):

  tail -f /var/log/ggrock-upgrade-debian.log | grep --line-buffered '%\]'
  

• Bring the progress bar back, for example after a reboot:

  /usr/local/sbin/ggrock-upgrade-debian.sh --watch
  

• Check that the upgrade is still active:

  systemctl status ggrock-debian-upgrade.scope --no-pager
  

The script refuses to start if an upgrade is already running, so you cannot stack two by accident.

What happens, in order

Each step does the following. The first checks are read-only, so if one fails nothing has been changed.

1. Safety checks. It waits up to 3 minutes for any other package-manager activity to finish, and stops if a package is half-installed or half-configured. It warns about network settings that will not apply as written, then checks for held packages and running VMs.
2. Old kernel cleanup. It keeps the running kernel and one older kernel as a fallback, and removes the rest along with their header packages (including leftover shared ones) and ZFS module entries.
3. Disk space. It needs 10 GB free on / and grows an LVM root automatically if it can.
4. Package sources. It backs them up, switches them to the next release and refreshes them. Translation files are never downloaded, so a web filter that blocks them cannot stop the upgrade. If the refresh fails, your original sources are restored automatically. Third-party repositories that publish nothing for the new release are disabled and listed in the log. A PostgreSQL entry that points at PostgreSQL's archive host is pointed at the live PostgreSQL repository first, because the archive has no packages for the new release. Debian, ggCircuit and PostgreSQL repositories are never disabled, so if one of those fails the step stops instead.
5. The upgrade. This is the longest part, roughly three quarters of the total time.
6. Final checks. It confirms the ggRock packages are still installed (and reinstalls them if not), reinstalls the Python packages ggRock's tools need, stops NetworkManager from taking over your network ports, adds the VM console fix if it is missing, and sets up one-time checks for the next startup.
7. Reboot. After the reboot, the next step starts by itself about a minute later. You do not need to run anything.

What is normal

• Long stretches where the screen barely changes during the upgrade.
• A pause of 5 to 10 minutes near the end of a step, with the log sitting on Installing apt-transport-https.... ggRock's package runs a repository setup that waits on the package manager. Let it finish.
• The web console (port 9090) refusing logins or being unavailable until the upgrade completes.
• A command you start yourself failing for a few seconds with a loader error saying a GLIBC_2.35 version was not found. That happens while the system library is being replaced. The upgrade is not affected. Wait a few seconds and run the command again.
• Your SSH session ending at every reboot. Reconnect after a few minutes. Servers with long hardware checks take a while to come back.

Check the result

The upgrade is finished when the log shows Reached Trixie. Cleaning up resume unit and rebooting to finish. followed by Rebooting now. and the server has come back from that last reboot. Then run this as root (it is one line):

echo "--- release"; grep PRETTY_NAME /etc/os-release; uname -r; echo "--- package problems (want nothing)"; dpkg --audit; echo "--- ggRock packages"; dpkg -l ggrock ggrock-linux-configurator | grep ^ii; echo "--- apt sources in use (want only trixie)"; apt-cache policy 2>/dev/null | grep -E '^ +[0-9]+ https?://' | grep -oE 'bullseye|bookworm|trixie' | sort | uniq -c; echo "--- network"; ip -br addr; ping -c3 $(ip route | awk '/^default/ {print $3}'); echo "--- services (want active)"; systemctl is-active ggrock postgresql; systemctl --failed --no-pager; echo "--- PostgreSQL"; pg_lsclusters; apt-cache policy postgresql-17 | sed -n 2,6p; echo "--- stations in the database: $(runuser -u postgres -- psql -p 5432 -d ggrock -Atc 'select count(*) from tb_machine;' < /dev/null 2>/dev/null)"; echo "--- storage"; zpool status -x; zfs version; echo "--- upgrade leftovers (want No such file twice)"; ls /var/lib/ggrock-upgrade-hop.state /etc/systemd/system/ggrock-debian-upgrade-resume.service 2>&1

What good looks like

• Release: Debian GNU/Linux 13 (trixie) and a kernel starting with 6.12.
• Package problems: nothing printed.
• ggRock packages: both lines start with ii.
• Apt sources: only trixie lines.
• Network: the gateway replies. Some networks block ping, so if only the ping fails, use the internet check right after this list.
• Services: active for ggrock and postgresql, and no failed units.
• PostgreSQL: the newest cluster (17) shows online on port 5432. An older cluster (12) showing down on port 5433 is normal. The postgresql-17 package should list a repository next to its version, not only /var/lib/dpkg/status (see Troubleshooting).
• Stations: the count matches the number of stations you expect.
• Storage: all pools are healthy, and the zfs and zfs-kmod versions match.
• Leftovers: both files report No such file or directory. If the first one exists, the upgrade never reached its reboot (see Troubleshooting).

If only the ping failed, test the internet with this line. Any HTTP status line means the internet works.

curl -sI --max-time 6 https://api.ggleap.com | head -1

Then start your VMs if you use any, and boot one station to confirm it loads its image.

Messages that look alarming but are harmless

• openipmi.service failing on a virtual machine. There is no IPMI hardware to attach to.
• NetworkManager-wait-online.service timing out after about 60 seconds on a server with unused network ports.
• A collation version mismatch warning from PostgreSQL. See Cleanup below.
• ZFS reporting Some supported and requested features are not enabled on the pool. Leave it alone. Running zpool upgrade cannot be undone, so only do it if ggCircuit support asks you to.

The one-time checks at startup

After each reboot, three checks run before ggRock starts. Each one logs what it found and then removes itself.

• Network port names (ggrock-netfix). Newer kernels can rename network ports, for example eno3 becomes eno3np0. Before each reboot the script records which physical port each name belongs to. If a port in /etc/network/interfaces has a new name, the check finds the same physical port, updates the file and keeps a backup at /etc/network/interfaces.pre-netfix.<time>. If a name now points at different hardware, or it cannot tell, it changes nothing and logs a warning.
• PostgreSQL (ggrock-pgcheck). It gives the database up to 30 seconds to start on its own, starts it if needed, and checks that the ggrock database is in the cluster it relied on.
• Bonded network ports (ggrock-bondcheck). It only does anything on servers with bonding. It confirms the bond is up and attached to the bridge, and repairs it safely if not. For LACP bonds it also checks that the switch is answering, and only logs a warning if it is not (see Troubleshooting).

Read what they logged during the current boot:

journalctl -b -t ggrock-netfix -t ggrock-pgcheck -t ggrock-bondcheck --no-pager | cut -c1-300

Cleanup in the following days

Remove the old PostgreSQL cluster

First confirm ggRock is using the new cluster. You should see connections to the ggrock database:

runuser -u postgres -- psql -p 5432 -d ggrock -Atc "select usename, datname, count(*) from pg_stat_activity where datname is not null group by 1,2" < /dev/null

The old cluster may not be a separate copy of your data. When the upgrade shares data files between the old and new clusters, your own backup (the pg_dumpall file from before you started) is the real fallback, and you must never start the old cluster or rename its files while the new one is running. To see whether files are shared, run this. A 1 followed by a number above 0 means they are:

ls /var/lib/postgresql/12/main/global/ | grep -c pg_control.old; find /var/lib/postgresql/17/main/base -type f -links +1 | wc -l

If the old cluster is down and the connections are on port 5432, wait several days of normal operation. Then remove it. This example removes version 12. Use the old version number that pg_lsclusters shows if it differs.

pg_dropcluster 12 main

Clear the database collation warning

The C library changed during the upgrade, so PostgreSQL warns about a collation version mismatch on every connection until the indexes are rebuilt. It is harmless, but you can clear it. Do this when no stations are booting, because REINDEX locks tables. Warnings printed while it runs are expected, and each database should end with REINDEX and ALTER DATABASE.

for d in ggrock postgres template1; do runuser -u postgres -- psql -p 5432 -d $d -c "REINDEX DATABASE $d;" -c "ALTER DATABASE $d REFRESH COLLATION VERSION;" < /dev/null; done

Check for disabled repositories

If the script disabled a third-party repository, it says so in the log. Tell whoever owns that service. If the vendor now publishes packages for Debian 13, you can restore the file from /var/backups/ggrock-upgrade/disabled-repos/.

grep -n "Disabled third-party" /var/log/ggrock-upgrade-debian.log

Remove leftover kernel headers

Servers upgraded with an older copy of the script can keep shared header packages named like linux-headers-4.19.0-10-common after their kernels were removed. They are harmless but use about 250 MB. Current copies remove them automatically. To list the leftovers, which are the shared header packages with no matching kernel on the server:

for p in $(dpkg -l 'linux-headers-*-common' | awk '/^ii/ {print $2}'); do v=${p#linux-headers-}; v=${v%-common}; ls /lib/modules | grep -q "^$v-" || echo $p; done

If it prints nothing, there is nothing to remove. Otherwise remove only the packages it printed, using their names in place of the example below. Read the list apt shows before you confirm, and do not run apt autoremove here.

apt-get purge linux-headers-4.19.0-10-common linux-headers-4.19.0-11-common

Troubleshooting

To see why the script stopped, look for the reason in the log:

grep -nE 'ABORTING|ACTION REQUIRED' /var/log/ggrock-upgrade-debian.log | tail -20

The script stopped with ABORTING before upgrading

These checks happen before any package is upgraded, and your package sources are unchanged. (Running VMs may already have been shut down, and old kernels may already have been removed. That is harmless.) Fix the cause, then run the same upgrade command again. Nothing needs undoing.

• Another package process is running, or a package is half-configured. This has its own section below.
• Held packages. Run the apt-mark unhold command the message prints, confirm apt-mark showhold is empty, then run the script again.
• A VM did not shut down. Find out why. If it is safe for what is running inside, you can force it off with virsh destroy and the VM's name, but that can lose data inside the VM.
• Not enough free space. Expand root, then check with df -h /. On LVM, lvs shows the volume names, and the command looks like lvextend -L +20G -r /dev/mapper/VG-root. Replace VG-root with the name df -h / shows.
• Package indexes could not be refreshed. The message names the repository that failed. Check the server can reach it (internet, DNS, proxy). The script has already restored your original sources. If the message shows 403 Forbidden for a Translation-en file, a web filter on the network is blocking Debian's translation files. Current copies of the script skip those files, so run the latest upgrade command again. If other files are blocked too, ask whoever manages the network to allow Debian's package servers.
• tzdata is newer than the next release offers. This one happens on the Debian 12 to Debian 13 step and has its own section below.

The script stopped because dpkg or apt is busy, or a package is half-configured

Both checks run before anything is changed, so nothing needs undoing. Look at the ABORTING line in the log.

• dpkg or apt has been busy. Another package process held the package manager for 3 minutes. The log names it, how long it has been running and what started it. A dpkg --configure -a or apt command someone started by hand is the usual cause. Let it finish, because a ggRock package can pause 5 to 10 minutes. Do not kill it or delete lock files. Then run the upgrade command again.

• A package is half-configured. The log lists the packages. Finish the configuration, confirm dpkg --audit prints nothing, then run the upgrade command again.

  dpkg --configure -a; apt-get -f install; dpkg --audit
  

If another process takes the package manager after these checks passed, the step stops at the upgrade with full-upgrade could not start, puts your original package sources back and names the process. The fix is the same: let it finish, then run the upgrade command again. If dpkg --configure -a fails or hangs, contact support.

The script stopped because tzdata is newer than the next release offers

This happens on the step that moves from Debian 12 to Debian 13. Debian ships timezone updates for Debian 12 a little ahead of Debian 13. If your server picked up one of those updates during the first step, the script cannot install tzdata-legacy (the package that keeps old timezone names such as US/Eastern working), because that would mean replacing your timezone package with an older one. The script stops before it upgrades anything and puts your package sources back, so nothing needs undoing. The message names the version you have, the newest version Debian 13 offers, and two ways forward.

• Wait a few days, until Debian 13 ships the same timezone release, then run the upgrade command again. You lose nothing by waiting.

• Accept a temporary downgrade, if you cannot wait. Your timezone rules stay one release behind until Debian 13 catches up, and then they update on their own. If your location had a recent timezone rule change, read the release notes for the newer release first (https://www.iana.org/time-zones/releases) and wait if the change you need is only in the newer one. To go ahead, run the upgrade again with the downgrade allowed:

  wget -O - https://images.ggleap.com/ManagedServices/Scripts/ggrock-upgrade-debian.sh | bash -s -- --allow-tzdata-downgrade
  

The script only downgrades the timezone packages (tzdata and tzdata-legacy), and only when you ask it to.

The script stopped partway (ACTION REQUIRED)

This means the package upgrade already ran, but a later step failed. Common causes are the network or DNS dropping while the Python packages install, or a package that could not build a kernel module. The server now has the new release's packages but is still running the old kernel.

1. Do not reboot.

2. Fix the cause. For a network problem, check the addresses and then DNS.

   ip -br addr; getent hosts pypi.org
   

3. Run the same upgrade command again. The script notices where it stopped, finishes the remaining steps and reboots, without upgrading any packages a second time.

If the log says an earlier step was interrupted before the package upgrade finished, contact support before doing anything else.

The script says "Already on Trixie. Nothing to do."

Your server is already on Trixie and nothing more is needed. If uname -r still shows an old kernel and the log has an ACTION REQUIRED line, contact support instead.

You cannot reach the server after a reboot

The most likely cause is a renamed network port. Go to the server console and check what the system calls its ports now, then check what the network file expects.

ip -br link

grep -nE '^(auto|iface)|bridge[-_]ports|bond[-_]slaves' /etc/network/interfaces

If a name in the file is not in the list, find the new name for that port with dmesg | grep -i renamed, then replace the old name in the file. This example changes eno3 to eno3np0. Use your own names.

sed -i 's/\beno3\b/eno3np0/g' /etc/network/interfaces

• Servers without bonding. Restart the bridge.

  ifdown vmbr0; ifup vmbr0
  

• Servers with bonding. Never run ifup, ifdown or systemctl restart networking on a live bond. A known bug can switch off both network ports. Recreate the bond instead, then re-attach the bridge and check both.

  ip link set bond0 down; ip link delete bond0; ifup bond0; ifdown vmbr0; ifup vmbr0; bridge link show; cat /proc/net/bonding/bond0
  

If that does not bring the network back, contact support.

The log warns that a network setting is ignored

At the start of each step the script reads /etc/network/interfaces and logs a warning if a static address will not come up as written. It only warns and never changes the file. To see the warning:

grep -n 'classful default' /var/log/ggrock-upgrade-debian.log

The usual cause is a subnet 255.255.255.0 line. subnet is not a keyword the system understands, so it is ignored and a default mask based on the address is used instead. For 172.21.90.6 that is /16, not /24. The server normally keeps working, so find out which mask your router really uses before changing anything. If the file is right, change the keyword to netmask.

sed -i 's/^\(\s*\)subnet /\1netmask /' /etc/network/interfaces

Then restart the bridge from the server console during a quiet period, because it briefly drops the stations' connection to ggRock. Servers with bonding can restart the bridge the same way, but never the bond itself.

ifdown vmbr0; ifup vmbr0

The log warns that a bond has no LACP partner

The log line starts with WARNING: bond 'bond0' is in 802.3ad (LACP) mode but no LACP partner has answered. This is about the network switch, not about the upgrade, and the server keeps working. The two cables are not acting as one connection, so one carries all the traffic, there is no redundancy, and the second sits idle. The server's own settings are almost always correct. Check what the bond sees:

grep -E 'Bonding Mode|Partner Mac Address|Number of ports|Aggregator ID' /proc/net/bonding/bond0

A Partner Mac Address of 00:00:00:00:00:00 and Number of ports: 1 mean no switch is answering. Ask whoever manages the switch whether both cables are on the same switch (or a stacked or paired switch) and whether the two ports are set up as one LACP group in active mode. When they are, the bond joins within seconds and the server needs no change or reboot. If the switch cannot do LACP, contact support to move the server to active-backup mode, which needs a scheduled reboot. Never change the bond settings on a live bond.

PostgreSQL has no package source after the upgrade

This affects servers whose PostgreSQL repository pointed at PostgreSQL's archive host, which only holds retired releases. Current copies of the script point it at the live host automatically. A server upgraded with an older copy keeps PostgreSQL on its Debian 11 build with no source for updates. PostgreSQL keeps working, but it will not receive updates. To check:

apt-cache policy postgresql-17 | sed -n 2,6p

If the only source shown is /var/lib/dpkg/status, or the version contains pgdg11 or pgdg12, add the live repository. This is the same line servers with a working PostgreSQL source already use:

echo 'deb [signed-by=/etc/apt/keyrings/pgdg.gpg] https://apt.postgresql.org/pub/repos/apt trixie-pgdg main' > /etc/apt/sources.list.d/pgdg.list

apt-get update && apt-cache policy postgresql-17 | head -8

The candidate should now come from the PostgreSQL repository you just added, with a pgdg13 version. Then update PostgreSQL's packages when no stations are in use. This restarts PostgreSQL, so ggRock is briefly interrupted. Preview the change first. You should see only Inst lines:

apt-get -s install --only-upgrade postgresql-17 postgresql-client-17 libpq5 | grep -E '^(Inst|Remv)' | cut -c1-130

Take a backup, then update:

runuser -u postgres -- pg_dumpall -p 5432 > /root/pre-pgdg-swap-$(date +%Y%m%d%H%M%S).sql

apt-get install --only-upgrade postgresql-17 postgresql-client-17 libpq5

Confirm PostgreSQL and ggRock are running, and that nothing PostgreSQL-related is left to update:

pg_lsclusters; systemctl is-active ggrock; apt list --upgradable 2>/dev/null | tail -n +2

If the old cluster's packages (postgresql-12, postgresql-client-12) or postgresql-common are listed as upgradable from trixie-pgdg, update them the same way. Do not run apt autoremove afterwards. The list it offers includes a fallback kernel and packages ggRock may need.

ggRock or the database is not running

systemctl status ggrock --no-pager; pg_lsclusters

If the newest cluster shows down, start it (this example uses version 17), then restart ggRock.

pg_ctlcluster 17 main start; systemctl restart ggrock

If they will not start, contact support and include the output of journalctl -u ggrock -n 40 --no-pager.

VM consoles show "Failed to connect to server"

The upgrade adds a fix for this automatically. If consoles still fail, run the standalone fix. It adds the missing nginx setting, tests the configuration, reloads nginx, and rolls back by itself if the test fails.

wget -O - https://images.ggleap.com/ManagedServices/Scripts/fix-novnc-websockify.sh | bash -

Your VMs are off

This is expected. The script shuts them down before the upgrade and never restarts them. Start each one with virsh start and its name.

What the upgrade changes on your server

• Debian release. Your package sources are switched to the next release. The originals are saved in /var/backups/ggrock-upgrade/.
• Old kernels. Everything except the running kernel and one older kernel is removed, along with its headers (including leftover shared header packages) and ZFS module entries. If a module cannot be built for the new kernel, it is removed so the upgrade can finish. This is recorded in /var/lib/ggrock-dkms-removed.log.
• Third-party repositories. Any that cannot serve the new release are disabled. The originals are kept in /var/backups/ggrock-upgrade/disabled-repos/.
• NetworkManager. It is masked so it cannot take over network ports that ggRock manages.
• Network port names. /etc/network/interfaces is edited only if a port was renamed, and a backup is kept. The script also logs a warning for settings that will not apply as written (an ignored subnet line, or an address with no netmask), but never edits the file for that reason. A small hardware map, /var/lib/ggrock-nic-map, exists only between a reboot and the startup check that uses it.
• VM console. If it is missing, a /vnc/websockify block is added to /etc/nginx/conf.d/ggrock.conf, and a backup is kept next to it.
• Python packages. debinterface and rtslib_fb are reinstalled for the new Python version.
• The user account. It is created if it does not exist, added to the sudo group, and its password is set to match root's current password on every run. Change it afterwards with passwd user if you want a different one.
• PostgreSQL. ggRock moves to the newest cluster (17). An older cluster, if there is one, is left stopped. It may share data files with the new one, so do not start it. A PostgreSQL repository entry on the archive host is pointed at the live host.
• The log. Everything is recorded in /var/log/ggrock-upgrade-debian.log.

Frequently asked questions

• Can I go back to the previous version?
  • There is no automatic rollback. The script keeps backups of the files it edits, but it does not downgrade the system. Your database backup and VM snapshot are the way back, so contact support before attempting a recovery.
• Do I need to run anything between steps?
  • No. After the first reboot the next step starts by itself.
• Can I run the script again?
  • Yes. It detects the current release, picks up from where it stopped, and says Already on Trixie when there is nothing left to do.
• What if the power fails or the server restarts during the upgrade?
  • Do not run any apt commands. Let the server start, then check dpkg --audit and the end of the log, and contact support.

Contact support

If you are stuck, contact ggCircuit support via the purple chat bubble at the bottom-right of ggcircuit.com and include the output of these commands:

tail -100 /var/log/ggrock-upgrade-debian.log

grep -E 'VERSION_CODENAME' /etc/os-release; uname -r; dpkg --audit; pg_lsclusters; ip -br addr

Tell us which step you were on, whether it stopped or only seems slow, and whether any stations or VMs are affected.