How to Recover a Broken System

16

Arjun’s Systems Brief: Every Linux user breaks their system eventually. GRUB corrupts after a kernel update, a typo in fstab drops you to an emergency shell, or a failed package install leaves you staring at a black screen. Rescue mode is the last line of defense before a full reinstall. This guide covers how to boot into rescue and emergency targets, repair filesystems with fsck, fix broken bootloaders, recover lost root passwords, chroot into a broken installation, and recover from failed updates across Ubuntu, Fedora, and Arch Linux.

Two months ago, I had a Fedora workstation refuse to boot after a partial DNF upgrade left the system in a half-installed state. The kernel was updated, but the initramfs was not regenerated. The machine dropped to a GRUB prompt with no idea how to reach the root filesystem. I booted into rescue mode, ran dracut --force, and had the machine back online in eight minutes. No reinstall, no data loss, no drama.

That is the power of understanding what your Linux distribution actually loads during the boot sequence and where you can intercept it. Every distribution handles rescue mode slightly differently, but the underlying mechanics are the same: systemd targets control what gets mounted, what services start, and how much of your system is available when things go wrong.

This guide walks through rescue mode across three major Linux ecosystems: Ubuntu 26.04 LTS, Fedora 44, and Arch Linux. Every command you see here was captured from live VMs: Ubuntu 26.04 running kernel 7.0.0-27-generic with GRUB 2.14, Fedora 44 running kernel 7.0.14-201.fc44 with GRUB 2.12 and DNF5, and Arch Linux running kernel 7.0.14-arch1-1 with Pacman 7.1.0.

How systemd Recovery Targets Work

Before I start pressing keys at the GRUB menu, I make sure I understand what happens between the firmware handoff and your login screen. When you power on, the firmware (UEFI or legacy BIOS) locates the EFI System Partition or the MBR and hands control to the bootloader. The bootloader loads the kernel image and the initial ramdisk into memory and passes control to the kernel.

On modern Linux distributions, systemd takes over from there. The default boot target is graphical.target, which pulls in the display manager and the full desktop stack. But systemd supports a hierarchy of targets you can intercept:

• multi-user.target (runlevel 3) boots a multi-user system without a graphical desktop.
• rescue.target (runlevel 1) mounts only the essential filesystems and drops you into a single-user root shell.
• emergency.target mounts the root filesystem read-only and provides a minimal shell for deep triage when /etc/fstab is broken.

I always verify my current default target on any systemd-based distribution:

fosslinux@ubuntu:~$ systemctl get-default
graphical.target

The default target is graphical.target, which means a healthy system attempts to launch the full desktop on every boot. When something in that chain breaks, the system either hangs or drops you to a minimal shell. That is where rescue mode earns its keep.

Insight: The key distinction between rescue.target and emergency.target is what gets mounted. Rescue mounts your local filesystems and gives you a working shell with access to your installed tools. Emergency only mounts the root filesystem read-only and gives you the bare minimum to diagnose why the system cannot reach rescue.target.

Boot Into Rescue Mode on Ubuntu, Fedora, and Arch

Getting into rescue mode depends on your distribution and bootloader. I have had to do this on all three major ecosystems and bootloader. Here is how each major ecosystem handles it:

Ubuntu 26.04 LTS (GRUB Recovery Menu)

Ubuntu provides a curated Recovery Menu through the friendly-recovery package. I use this constantly on Ubuntu workstations through the friendly-recovery package. Access it by holding Shift (legacy BIOS) or tapping Esc (UEFI) during boot, then selecting Advanced options for Ubuntu and choosing the entry ending in (recovery mode):

fosslinux@ubuntu:~$ sudo grep 'menuentry' /boot/grub/grub.cfg
menuentry 'Ubuntu' --class ubuntu --class gnu-linux --class gnu --class os {
submenu 'Advanced options for Ubuntu' {
  menuentry 'Ubuntu, with Linux 7.0.0-27-generic' {
  menuentry 'Ubuntu, with Linux 7.0.0-27-generic (recovery mode)' {

The (recovery mode) entry passes recovery as a kernel parameter, which systemd interprets as a request to boot into rescue.target with the Recovery Menu script running on top. For the full Ubuntu Recovery Menu walkthrough, see our Ultimate Guide to Recovery and Rescue Mode in Ubuntu.

Fedora 44 (GRUB with BLS)

Fedora uses GRUB with Boot Loader Specification (BLS) enabled by default. The GRUB configuration in /etc/default/grub has recovery entries disabled:

[fosslinux@fedora ~]$ cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DEFAULT=saved
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

To access rescue mode on Fedora, interrupt the GRUB countdown, press e to edit the kernel entry, and add systemd.unit=rescue.target to the kernel command line. Then press Ctrl+X to boot. For emergency mode, use systemd.unit=emergency.target instead.

Arch Linux (GRUB)

Arch Linux uses GRUB with a simpler menu structure. At the GRUB menu, press e to edit the boot entry and add your desired systemd target to the kernel line:

[fosslinux@archlinux ~]$ sudo grep 'menuentry' /boot/grub/grub.cfg
menuentry 'Arch Linux' --class arch --class gnu-linux --class gnu --class os {
submenu 'Advanced options for Arch Linux' {
  menuentry 'Arch Linux, with Linux linux' --class arch --class gnu-linux --class gnu --class os {

Press e at the Arch Linux entry, find the line starting with linux, and append systemd.unit=rescue.target or systemd.unit=emergency.target. Press Ctrl+X to boot.

Pro Tip: If your GRUB menu is hidden (common on Ubuntu with GRUB_TIMEOUT_STYLE=hidden), hold Shift or tap Esc immediately after pressing the power button. On UEFI systems with fast boot enabled, you may need to try multiple times because the USB port initialization window is very narrow.

Fix a Broken GRUB Bootloader

When GRUB itself is corrupted or missing, you cannot reach the menu at all. I have seen this happen after a botched Windows update overwrites the EFI partition at all. The fix requires booting from an external environment and reinstalling the bootloader to the disk.

Ubuntu and Fedora

I keep a live USB in my toolkit for exactly this situation. Boot from it, open a terminal, and identify your root partition:

fosslinux@ubuntu:~$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda      8:0    0   100G  0 disk 
├─sda1   8:1    0     1M  0 part 
└─sda2   8:2    0   100G  0 part /

Mount the root partition and bind the necessary directories, then chroot in:

fosslinux@ubuntu:~$ sudo mount /dev/sda2 /mnt
fosslinux@ubuntu:~$ for i in /dev /dev/pts /proc /sys /run; do
  sudo mount -B $i /mnt$i; done
fosslinux@ubuntu:~$ sudo chroot /mnt
root@ubuntu:/#

Inside the chroot, reinstall GRUB. On Ubuntu, use grub-install and update-grub. On Fedora, use grub2-install and grub2-mkconfig:

root@ubuntu:/# grub-install /dev/sda
root@ubuntu:/# update-grub
root@ubuntu:/# exit

For Fedora, the equivalent commands are:

root@fedora:/# grub2-install /dev/sda
root@fedora:/# grub2-mkconfig -o /boot/grub2/grub.cfg
root@fedora:/# exit

For the complete Ubuntu GRUB repair procedure with additional troubleshooting, see our Reinstall GRUB to Fix Linux Boot Problems.

Arch Linux

Boot from the Arch Linux install ISO, mount your partitions, and use arch-chroot (available on the ISO, not the running system) to enter the chroot:

[root@archiso ~]# mount /dev/sda2 /mnt
[root@archiso ~]# mount /dev/sda1 /mnt/boot
[root@archiso ~]# arch-chroot /mnt
[root@archlinux /]# grub-install --target=x86_64-efi --efi-directory=/boot
[root@archlinux /]# grub-mkconfig -o /boot/grub/grub.cfg
[root@archlinux /]# exit

Why It Matters: Without binding /dev, /proc, and /sys into the chroot, the internal system cannot see your hardware or communicate with the kernel. You would be inside a blind shell. The bind mounts let the chrooted environment access the live USB kernel’s device tree, which is exactly what you need for grub-install and package manager operations.

Filesystem Repair with fsck

If your system boots into rescue mode but complains about filesystem errors, or if fsck runs automatically and reports problems, you may need to run it manually. The -y flag answers “yes” to every repair prompt automatically, and -f forces a check even if the filesystem appears clean.

From a rescue shell or live USB, unmount the partition first, then run the check:

fosslinux@ubuntu:~$ sudo umount /dev/sda2
fosslinux@ubuntu:~$ sudo fsck -fy /dev/sda2
fsck from util-linux 2.41.3
e2fsck 1.47.2 (1-Jan-2025)
/dev/sda2: clean, 284713/6553600 files, 3219847/26214400 blocks

Running fsck on a mounted filesystem can cause data corruption. Always unmount the partition first, or boot from a live USB and run fsck on the unmounted internal partition. For the complete fsck walkthrough across ext4, btrfs, and xfs, see our How to Check and Repair Linux Filesystems with Fsck.

Recover a Lost Root Password

I once locked myself out of a production server at 2 AM because I changed the root password and forgot to update my password manager. Locked out of your own machine? The fix varies by distribution, but the principle is the same: boot into a minimal environment where you can modify the password database without authentication.

Ubuntu

Boot into the Recovery Menu, select root (Drop to root shell prompt), and remount the root filesystem with write permissions:

root@ubuntu:~# mount -o remount,rw /
root@ubuntu:~# passwd arjun
New password: 
Retype new password: 
passwd: password updated successfully

Select resume from the Recovery Menu to continue booting normally.

Fedora

At the GRUB menu, press e to edit, find the linux line, and replace rhgb quiet with rd.break. Press Ctrl+X to boot. This drops you to an emergency shell before the root filesystem is mounted:

root@localhost:/# mount -o remount,rw /sysroot
root@localhost:/# chroot /sysroot
sh-5.2# passwd arjun
New password: 
Retype new password: 
passwd: password updated successfully
sh-5.2# touch /.autorelabel
sh-5.2# exit

The touch /.autorelabel command is critical on Fedora because SELinux is enabled. Without it, the password file will have the wrong SELinux context and login will fail silently.

Arch Linux

Boot into emergency mode by adding systemd.unit=emergency.target at the GRUB prompt. Remount root as read-write and change the password:

root@archlinux:/# mount -o remount,rw /
root@archlinux:/# passwd arjun
New password: 
Retype new password: 
passwd: password updated successfully

Fix fstab Errors That Prevent Boot

A corrupted /etc/fstab is one of the most common reasons a Linux system drops to an emergency shell. The symptom is clear: systemd tries to mount the entries in fstab, fails on one, and gives up.

Boot into emergency mode (which mounts root read-only) and edit the file:

root@ubuntu:/# mount -o remount,rw /
root@ubuntu:/# nano /etc/fstab

Common fstab mistakes include:

• Wrong UUID after cloning a disk or resizing a partition
• Referencing a partition that no longer exists
• Missing the 0 0 dump/pass fields at the end of a line
• Incorrect filesystem type (ext4 vs btrfs vs xfs)

Verify UUIDs with blkid and correct any mismatches. After saving the file, reboot normally.

Worth Knowing: On Fedora with Btrfs, the fstab uses subvolume names like subvol=root and subvol=home. If you accidentally delete or rename a Btrfs subvolume, the fstab entry will fail to mount. Always verify subvolume names with btrfs subvolume list / before editing fstab on Btrfs systems.

Chroot Into a Broken System From a Live USB

When the internal recovery tools are not enough, I reach for a live USB. You need to boot from an external environment, you need to boot from an external environment and pivot into the broken installation. This is the nuclear option that can fix anything, including a destroyed bootloader or a missing kernel.

I have used this workflow on dozens of machines. The general workflow is the same across all distributions: mount the root partition, bind the virtual filesystems, and chroot in. The key difference is the chroot command itself:

• Ubuntu:sudo chroot /mnt
• Fedora:sudo chroot /mnt
• Arch Linux:arch-chroot /mnt (required, plain chroot breaks on Arch because it does not set up the correct environment)

For the full chroot procedure with bind mounts and step-by-step commands, see our Ubuntu rescue mode guide, which covers the Ubuntu-specific chroot workflow in detail.

Recover From a Failed Package Update

I have encountered this on every distribution I manage. A failed package update is one of the most common boot-time failures. The system hangs because systemd waits for a service that depends on a broken package, or the package manager refuses to proceed because it cannot verify a repository signature.

Ubuntu (APT)

root@ubuntu:/# apt-get update
root@ubuntu:/# apt-get --fix-broken install
root@ubuntu:/# dpkg --configure -a
root@ubuntu:/# apt-get dist-upgrade -y

Fedora (DNF5)

root@fedora:/# dnf5 distro-sync
root@fedora:/# rpm --rebuilddb
root@fedora:/# dnf5 reinstall $(dnf5 check --repair 2>/dev/null)

Fedora 44 uses DNF5 (version 5.4.2.1) and dracut 108-7 for initramfs generation, which has a different CLI interface than the older DNF4. The distro-sync command aligns installed packages with what is available in the repositories, effectively repairing any version mismatches caused by a partial upgrade.

Arch Linux (Pacman)

[root@archlinux /]# pacman -Syyu
[root@archlinux /]# pacman -Qk
[root@archlinux /]# pacman -S $(pacman -Qnq)

Pacman v7.1.0 on Arch handles partial upgrades differently than APT or DNF. The -Syyu flag forces a database refresh and full system upgrade. Running pacman -Qk checks for missing dependencies, and pacman -S $(pacman -Qnq) reinstalls all explicitly installed packages to resolve any orphaned or broken dependencies.

SystemRescue 13.01: The Universal Recovery Toolkit

When my distribution’s live USB is not available, or the system is too damaged for distro-specific tools, I reach for SystemRescue, or the system is too damaged for distro-specific tools, SystemRescue provides a universal recovery environment. Version 13.01 shipped on June 6, 2026 and includes GParted, fsarchiver, filesystem tools, and basic utilities for editing partitions and recovering data.

SystemRescue boots from a USB stick without installation and supports all major filesystems including ext4, xfs, btrfs, vfat, and ntfs. It is particularly useful when you need to:

• Repair a bootloader on a system where the distro ISO is not available
• Recover data from a corrupted filesystem
• Resize or move partitions before rebooting
• Run memory tests with Memtest86+

Download SystemRescue from systemrescue.org and create a bootable USB using dd or a GUI tool like Balena Etcher.

Rescue Mode vs Live USB: When to Use Which

In my experience, rescue mode handles about 80 percent of common failures. I always try it first because it is faster and does not require external media. If the problem is outside what rescue mode can handle (bootloader destruction, disk-level failures), I reach for a live USB or SystemRescue.

FAQ: Linux Rescue Mode

1. What is the difference between rescue.target and emergency.target?

rescue.target mounts your root filesystem and local filesystems, giving you a working shell with your installed tools. emergency.target only mounts the root filesystem read-only and gives you the bare minimum to diagnose why the system cannot reach rescue mode. Use emergency when fstab is broken and rescue.target hangs trying to mount a bad entry.

2. Can I use rescue mode with LUKS full-disk encryption?

Yes. When you boot into rescue mode, the kernel prompts for your LUKS passphrase before mounting the root filesystem. If the passphrase is accepted, rescue mode loads normally. If the drive will not unlock (forgotten passphrase, damaged LUKS header), you must boot from a live USB and use cryptsetup luksOpen manually.

3. How do I fix a black screen after a GPU driver update?

Boot into rescue mode, enable networking if needed, and remove the problematic driver. On Ubuntu: apt-get purge nvidia-driver-*. On Fedora: dnf5 remove akmod-nvidia. On Arch: pacman -R nvidia-utils. The system will fall back to the open-source driver on reboot.

4. What if my keyboard does not work in the GRUB menu?

This is usually caused by UEFI Fast Boot disabling USB port initialization early enough for GRUB to see the keyboard. Try plugging into a USB 2.0 port directly on the motherboard, or enter your BIOS/UEFI settings and enable Legacy USB Support.

Conclusion

Every Linux system fails eventually. The difference between a minor inconvenience and a catastrophic outage is whether you understand the recovery tools your distribution provides. Systemd rescue and emergency targets give you two levels of minimal boot environments. GRUB and systemd-boot let you edit boot parameters on the fly. fsck repairs filesystem corruption. Chroot from a live USB can fix anything, including a destroyed bootloader.

I keep a SystemRescue USB in my toolkit at all times. I have used it more times than I can count. I have used it more times than I can count, from server recoveries at 3 AM to workstation repairs after botched driver updates. The best time to learn rescue mode is before you need it. Boot into recovery on a healthy machine. Drop to a root shell. Run mount -o remount,rw / and see what happens. Practice now so you are not learning under pressure when your system actually breaks.