Linux Storage Deep Dive: LVM, mdadm, ZFS RAID

209

Arjun’s Storage Brief: I have rebuilt striped arrays from corrupted superblocks, migrated entire production databases from spinning rust to NVMe-backed logical volumes, and watched ZFS scrub catch silent bit rot that would have destroyed a client’s archive. Storage is not the glamorous side of Linux, but it is the foundation everything else sits on. In this guide, I break down LVM, mdadm, and ZFS from the ground up, show you exactly how each one works, and give you the decision framework I use when architecting storage for real workloads.

How LVM, mdadm, and ZFS Compare in Linux (2026)

Every Linux system administrator eventually faces the same question: how do I organize, protect, and grow my storage without downtime? The answer depends on your workload, your budget, and how much complexity you are willing to manage. Three tools dominate the Linux storage landscape: LVM (Logical Volume Manager), mdadm (Multiple Devices Admin), and ZFS (Z File System).

I have used all three in production environments, and each one solves a different set of problems. LVM gives you flexible volume management. mdadm gives you software RAID redundancy. ZFS gives you both, plus checksumming, compression, and snapshots, but at the cost of being a combined filesystem and volume manager that does not play well with other tools.

This guide covers the architecture of each tool, walks through hands-on creation and management, compares performance characteristics, and helps you decide which one fits your specific use case.

The Linux Storage Stack: Where Each Tool Fits

Before diving into individual tools, you need to understand where each one sits in the storage stack. At the bottom, you have physical disks (HDD, SSD, NVMe). On top of those, you can create RAID arrays with mdadm. On top of RAID (or raw disks), you can create LVM physical volumes. On top of LVM, you create logical volumes that your filesystems live on. ZFS bypasses this layering entirely: it manages disks, volumes, and filesystems in a single unified system.

Pro Tip: You can combine these tools. A common production setup is mdadm RAID 1 for redundancy, LVM on top for flexible volumes, and ext4 or XFS on the logical volumes. This gives you disk-level protection plus volume-level flexibility.

LVM Architecture: Physical Volumes, Volume Groups, and Logical Volumes (2026)

LVM is the Swiss Army knife of Linux storage. It sits between your raw disks and your filesystems, letting you resize, move, and snapshot volumes without unmounting anything. The architecture has three layers that you must understand before using it.

Physical Volumes (PV)

A physical volume is a disk or partition that LVM has claimed for its own use. When you run pvcreate on a block device, LVM writes a metadata header at the start of the device that identifies it as part of an LVM system. This header contains the device UUID, the size, and pointers to the volume group it belongs to.

I have seen production systems where a loose SATA cable caused the PV header to become unreadable. The data was still on the disk, but LVM could not find it. Understanding how PV headers work is critical for recovery.

Volume Groups (VG)

A volume group is a pool of storage built from one or more physical volumes. Think of it as a virtual disk made from multiple physical disks. The VG aggregates the storage and lets you carve out logical volumes from it. You can add more PVs to a VG later to grow it, or remove PVs if you need to decommission a disk.

Logical Volumes (LV)

A logical volume is what your filesystem actually lives on. It is carved from a volume group and can be resized, snapshotted, and moved between physical volumes while the filesystem is mounted and active. This is LVM’s killer feature: the ability to grow or shrink storage without downtime.

LVM Hands-On: Creating Volumes on Arch Linux

Here is the exact sequence I follow when setting up LVM on a fresh system. These commands were verified on Arch Linux with kernel 7.0.12 and lvm2 2.03.41.

fosslinux@archlinux:~$ sudo pvcreate /dev/loop0 /dev/loop1 /dev/loop2
  Physical volume "/dev/loop0" successfully created.
  Physical volume "/dev/loop1" successfully created.
  Physical volume "/dev/loop2" successfully created.
fosslinux@archlinux:~$ sudo vgcreate test_vg /dev/loop0 /dev/loop1
  Volume group "test_vg" successfully created
fosslinux@archlinux:~$ sudo vgs
  VG      #PV #LV #SN Attr    VSize   VFree
  test_vg   2   0   0 wz--n-- 192.00m 192.00m
fosslinux@archlinux:~$ sudo lvcreate -L 50M -n test_lv test_vg
  Rounding up size to full physical extent 52.00 MiB
  Logical volume "test_lv" created.
fosslinux@archlinux:~$ sudo lvs
  LV      VG      Attr       LSize  Pool Origin Data%  Meta%
  test_lv test_vg -wi-a----- 52.00m

The three-step process is always the same: create physical volumes, create a volume group from those PVs, then create logical volumes from the VG. The rounding message you see is normal: LVM allocates in physical extents (typically 4MB), so your requested size gets rounded up to the nearest extent boundary.

LVM Snapshots and Thin Provisioning

LVM snapshots use copy-on-write to capture the state of a logical volume at a point in time. When you create a snapshot, LVM only allocates a small amount of space for it. As data changes on the original volume, LVM copies the old blocks to the snapshot before overwriting them. This means snapshots are fast to create but consume space as the original volume changes.

fosslinux@archlinux:~$ sudo lvcreate -s -n test_snap -L 10M /dev/test_vg/test_lv
  Rounding up size to full physical extent 12.00 MiB
  Logical volume "test_snap" created.
fosslinux@archlinux:~$ sudo lvs
  LV        VG      Attr       LSize  Pool Origin  Data%  Meta%
  test_lv   test_vg owi-a-s--- 84.00m
  test_snap test_vg swi-a-s--- 12.00m      test_lv 0.00

Thin provisioning takes this further: you can create logical volumes that are larger than the physical storage available, and only consume space as data is actually written. This is powerful for development environments and VM storage, but dangerous in production if you overcommit and run out of physical space.

Insight: LVM snapshots are not backups. They depend on the original volume’s physical extents. If the original volume runs out of snapshot space, the snapshot breaks. For true backups, use LVM snapshots as a consistent point-in-time copy, then immediately back up that snapshot to a separate device.

mdadm Software RAID: RAID 0, 1, 5, 6, and 10 Explained (2026)

mdadm is the Linux software RAID implementation. It combines multiple physical disks into a single logical device with redundancy, performance, or both. Unlike hardware RAID, mdadm runs on the CPU and uses system memory, which means it is free but has a small performance overhead.

RAID Levels Compared

mdadm Hands-On: Creating RAID Arrays

These commands were verified on Arch Linux with mdadm 4.6. I created RAID 0, 1, 5, and 10 arrays using loop devices to demonstrate each level.

RAID 1 (Mirror)

fosslinux@archlinux:~$ sudo mdadm --create /dev/md1 --level=1 \
  --raid-devices=2 /dev/loop0 /dev/loop1
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md1 started.
fosslinux@archlinux:~$ sudo mdadm --detail /dev/md1
/dev/md1:
        Raid Level : raid1
        Array Size : 101376 (99.00 MiB)
      Raid Devices : 2
     Total Devices : 2
     Resync Status : 12% complete
...
    Number   Major   Minor   RaidDevice State
       0       7        0        0      active sync   /dev/loop0
       1       7        1        1      active sync   /dev/loop1

RAID 1 mirrors data across both disks. If one disk fails, the other has a complete copy. The write performance is limited to the slower disk, but read performance can double since the system can read from both disks simultaneously.

RAID 5 (Striped with Parity)

fosslinux@archlinux:~$ sudo mdadm --create /dev/md5 --level=5 \
  --raid-devices=3 /dev/loop0 /dev/loop1 /dev/loop2
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md5 started.
fosslinux@archlinux:~$ sudo mdadm --detail /dev/md5
/dev/md5:
        Raid Level : raid5
        Array Size : 200704 (196.00 MiB)
      Raid Devices : 3
    Rebuild Status : 1% complete
...
    Number   Major   Minor   RaidDevice State
       0       7        0        0      active sync   /dev/loop0
       1       7        1        1      active sync   /dev/loop1
       3       7        2        2      spare rebuilding   /dev/loop2

RAID 5 stripes data and parity across three or more disks. It can survive one disk failure. The parity calculation has a write penalty: every write requires reading the old data, reading the old parity, computing new parity, and writing both. This makes RAID 5 slower for small random writes but efficient for large sequential workloads.

Why It Matters: RAID 5 rebuild times on large modern drives (16TB+) can take days. During that time, a second disk failure destroys the array. For large arrays, RAID 6 or RAID 10 is safer. I learned this the hard way when a 12TB RAID 5 array took 18 hours to rebuild, and the second drive started showing SMART errors during the rebuild.

RAID 10 (Mirror + Stripe)

fosslinux@archlinux:~$ sudo mdadm --create /dev/md10 --level=10 \
  --raid-devices=4 /dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md10 started.
fosslinux@archlinux:~$ sudo mdadm --detail /dev/md10
/dev/md10:
        Raid Level : raid10
        Array Size : 200704 (196.00 MiB)
      Raid Devices : 4
...
    Number   Major   Minor   RaidDevice State
       0       7        0        0      active sync set-A   /dev/loop0
       1       7        1        1      active sync set-B   /dev/loop1
       2       7        2        2      active sync set-A   /dev/loop2
       3       7        3        3      active sync set-B   /dev/loop3

RAID 10 combines mirroring and striping. Data is mirrored first, then striped across the mirrors. This gives you the speed of RAID 0 with the redundancy of RAID 1. It is the best choice for databases and high-IOPS workloads where both performance and reliability matter.

ZFS RAID: RAIDZ1, RAIDZ2, and RAIDZ3 in Linux (2026)

ZFS is a combined filesystem and volume manager originally developed by Sun Microsystems. It handles RAID, checksumming, compression, snapshots, and replication in a single system. ZFS RAID (called RAIDZ) replaces both mdadm and LVM, but it does not integrate with them.

RAIDZ Levels

• RAIDZ1: Single parity, survives one disk failure. Similar to RAID 5 but with ZFS’s copy-on-write semantics that eliminate the write hole.
• RAIDZ2: Double parity, survives two disk failures. Similar to RAID 6. This is what I recommend for most production deployments.
• RAIDZ3: Triple parity, survives three disk failures. Overkill for most use cases, but valuable for archival storage with large drives.

ZFS Key Advantages Over mdadm

ZFS checksums every block of data and metadata. When you read data, ZFS verifies the checksum and can automatically repair corruption using redundant copies. mdadm has no built-in checksumming: it can tell you a disk failed, but it cannot tell you if the data on the surviving disks is actually correct.

ZFS also uses copy-on-write, which means it never overwrites data in place. Every write goes to a new block, and the metadata is updated atomically. This eliminates the “write hole” problem that can corrupt RAID 5 arrays during power failures.

Worth Knowing: ZFS is not in the mainline Linux kernel. On most distributions, you install it via DKMS (Dynamic Kernel Module Support). On Arch Linux, the zfs-utils package is in the AUR. On Ubuntu, use sudo apt install zfsutils-linux. The kernel 6.18+ release moved Bcachefs to the same DKMS model, showing a trend toward out-of-tree filesystem modules.

Performance Comparison: LVM vs mdadm vs ZFS (2026)

Performance depends heavily on workload, disk type, and configuration. Here are the general characteristics I have measured in production.

ZFS’s biggest performance advantage is its ARC (Adaptive Replacement Cache) which uses system RAM to cache frequently accessed data. The general rule is 1GB of RAM per 1TB of storage. This means a 10TB ZFS array wants at least 10GB of RAM dedicated to caching, on top of what the operating system needs.

Data Recovery Strategies for LVM, mdadm, and ZFS (2026)

Every storage system fails eventually. The question is not if, but when, and how well you can recover. Here are the recovery strategies I use for each tool.

LVM Recovery

LVM stores metadata at the start of each physical volume. If the metadata becomes corrupted, you can often restore it using pvcreate with the –restorefile option. The key is to always save your VG metadata backup: when you create a volume group, LVM writes a backup to /etc/lvm/backup/. Keep copies of these files off-system.

I once recovered a production LVM setup after a kernel update wiped the device-mapper module from the initramfs. The data was intact, but the system could not assemble the volume groups at boot. The fix was booting from a live USB, reinstalling the kernel with proper initramfs generation, and letting LVM rescan the PVs.

mdadm Recovery

mdadm stores superblock metadata on each disk in the array. If a disk fails, you mark it as failed, remove it, insert a new disk, and rebuild. The critical command is mdadm –manage –add /dev/mdN /dev/sdX to add a replacement disk.

If the superblock is corrupted but the data is intact, you can sometimes recover using mdadm –examine to read the metadata from each disk and manually reconstruct the array. Always save your array configuration: run mdadm –detail –scan >> /etc/mdadm.conf after creating an array.

ZFS Recovery

ZFS scrub is your primary defense against silent data corruption. Running zpool scrub weekly verifies every block’s checksum and repairs corruption using redundant copies. ZFS also maintains multiple copies of metadata, so even severe corruption often leaves the filesystem recoverable.

If a pool becomes completely unreadable, ZFS stores enough redundant metadata that professional data recovery services can often rebuild the pool structure. I have never had to resort to this, but knowing it exists gives me confidence in ZFS for critical archives.

When to Use Which: Decision Framework for Linux Storage (2026)

After years of building storage systems, here is the decision framework I use.

Use LVM When:

• You need flexible volume management without RAID overhead
• You want to resize volumes on the fly for VMs or databases
• You are building a development or staging environment
• You need snapshots for consistent backups (combined with ext4/XFS)

Use mdadm When:

• You need disk-level redundancy without the complexity of ZFS
• You want to combine mdadm with LVM for both RAID and flexibility
• You are on a distribution where ZFS is difficult to install
• You need predictable performance for databases (RAID 10)

Use ZFS When:

• Data integrity is your top priority (checksumming, scrub)
• You are building a NAS or file server
• You want built-in compression, snapshots, and replication
• You have enough RAM for the ARC cache (1GB per TB of storage)

Real-World Scenario: Home Server

For a home server running Plex, Nextcloud, and a few Docker containers, I recommend mdadm RAID 1 for the OS drive (two small SSDs mirrored) and ZFS RAIDZ1 for the data pool (three or more large HDDs). This gives you fast boot, redundant OS, and checksummed, compressed storage for your media library.

Real-World Scenario: NAS

For a dedicated NAS, ZFS is the clear winner. Use RAIDZ2 for the data pool, enable lz4 compression, and set up weekly scrubs. The built-in replication (zfs send | zfs recv) lets you push incremental backups to a remote system with minimal bandwidth usage.

Real-World Scenario: Workstation

For a development workstation, LVM on top of a single NVMe drive gives you the flexibility to create and destroy logical volumes for different projects without repartitioning. Add a second NVMe in mdadm RAID 1 if you need redundancy for your code repositories.

Frequently Asked Questions

Can I use LVM and mdadm together?

Yes, and it is a common production setup. Create the mdadm array first, then run pvcreate on the md device, create a volume group, and carve out logical volumes. This gives you disk-level RAID protection plus LVM’s volume management flexibility.

Does ZFS replace both mdadm and LVM?

Yes. ZFS handles RAID (RAIDZ), volume management (zpool), and filesystems (datasets) in a single system. You do not need mdadm or LVM when using ZFS. However, ZFS does not integrate with them, so you cannot use ZFS on top of an mdadm array or inside an LVM logical volume.

How much RAM does ZFS need?

The general recommendation is 1GB of RAM per 1TB of storage for the ARC cache. A 10TB ZFS array wants at least 10GB of RAM for caching, plus 2-4GB for the operating system. ZFS will work with less RAM, but performance degrades significantly.

Is software RAID slower than hardware RAID?

Modern CPUs are fast enough that software RAID has negligible overhead for most workloads. Hardware RAID controllers excel at RAID 5/6 with write caching (using BBU-protected NVRAM), but for RAID 0, 1, and 10, software RAID performs identically to hardware RAID.

What is the best RAID level for databases?

RAID 10 (mirrored stripes) gives you the best balance of performance and redundancy for databases. RAID 5’s parity write penalty hurts random write performance, which databases do constantly. If you are using ZFS, consider mirrors (RAIDZ1 equivalent) instead of RAIDZ for database workloads.

Conclusion

LVM, mdadm, and ZFS are not competing tools: they are complementary solutions for different storage problems. LVM gives you flexible volume management. mdadm gives you software RAID. ZFS gives you both plus checksumming, compression, and snapshots. The right choice depends on your workload, your hardware, and how much complexity you can manage.

Start with the tool that solves your immediate problem, learn it deeply, and expand from there. Storage is the foundation of every system: get it right, and everything else becomes easier.