Production Server Setup Notes, Ep. 1 - What Is RAID?

Also available in Korean: 운영계 세팅 회고 ep1 - RAID란?

What I found

I took over a new production server and started setting it up.
Before installing anything I checked the disk layout, and lsblk showed this.

nvme0n1       931.5G disk  isw_raid_member
├─md126         1.7T raid0
│ ├─md126p3     1.7T part  LVM2_member
│ │ ├─rl-root    70G lvm   xfs   /
│ │ └─rl-home   1.6T lvm   xfs   /home
nvme1n1       931.5G disk  isw_raid_member
├─md126         1.7T raid0
sda               7T disk
└─sda1            7T part  xfs   /data/ssd
sdb             7.3T disk
└─sdb1          7.3T part  xfs   /data/hdd

The two NVMe drives holding the OS were striped together as raid0.
If either one fails, the whole OS goes with it.
I asked the people who handed over the server why they had chosen RAID 0, and the answer was that they had no idea it was set up that way. (ㅡ..ㅡ)

So I asked for the OS disks to be rebuilt as RAID 1.
What follows are the RAID concepts that decision rests on.

What RAID is

RAID (Redundant Array of Independent Disks) combines several disks so that the operating system sees them as one.
There are two reasons to do it: to get a bigger, faster disk, or to keep data when a disk fails.

How the disks are combined comes down to mixing three ingredients.

Ingredient	What it does	What you gain	What you give up
striping	Splits data into pieces and writes them across disks in turn	Speed, combined capacity	Losing one disk loses everything
mirroring	Writes the same data to two disks	Survives one disk failure	Half the capacity
parity	Stores a recovery value computed from the data	Survives a failure with little extra space	Every write needs a calculation

Each combination has a number: the RAID level.

RAID levels

The figure below shows how each level lays data out across disks, and what is left when the second disk fails.
A1, A2… are pieces of one file, and cells ending in p hold parity.

Parity is the XOR of the data pieces in the same row, so if one piece is lost, XOR-ing the remaining pieces with the parity brings it back.

What is left at each level when the second disk fails. Only RAID 0 loses data.

RAID 0

Striping only.
A file is split and written across both disks in turn, so you get the full capacity of both and reads and writes are spread out and faster.

The catch is that redundancy is zero, which is where the 0 comes from.
Since the pieces of every file are scattered across both disks, losing one disk leaves nothing that can rebuild any file.
With two disks, the chance that at least one fails is roughly double that of a single disk.
It suits scratch space or caches you can afford to lose.

RAID 1

Mirroring.
Both disks get the same writes, so if one dies the other is still a complete original.
You can use only one disk’s worth of capacity.
When the failed disk is replaced, the surviving disk is copied onto it.
It is simple, which is why it is common for OS disks and other small places that must not stop.

RAID 5

Striping plus one parity.
With three disks, two disks’ worth holds data and one disk’s worth holds parity.
Parity is not kept on one disk; its position rotates from row to row.
When one disk dies, its contents are computed from the remaining data and parity.

It uses capacity well, but while a new disk is being filled in after a failure (the rebuild), there is no spare left.
If another disk dies during that time, everything is lost.
Bigger disks mean longer rebuilds, which is why RAID 6 is recommended when combining several large HDDs.

RAID 6

RAID 5 with a second parity.
It survives two simultaneous failures and needs at least four disks.
Every write computes two parities, so writes are slower than RAID 5.

RAID 10

First build RAID 1 pairs, then stripe across the pairs.
With four disks you get two mirror pairs.
One disk per pair can die, but if both disks of the same pair die, data is lost.
Capacity is halved, but there is no parity to compute, so writes are fast, which makes it common for write-heavy databases.

Comparing levels

For n disks of the same size, each with capacity C:

Level	Minimum disks	Usable capacity	Failures tolerated	Typical use
RAID 0	2	n × C	0	Scratch space, caches
RAID 1	2	C	n−1	OS disks
RAID 5	3	(n−1) × C	1	Read-heavy data
RAID 6	4	(n−2) × C	2	Many large disks
RAID 10	4	n/2 × C	1 per pair	Write-heavy databases

RAID only protects against disk failure.
Data deleted by mistake or encrypted by ransomware is written to every disk just the same, so those incidents need a separate backup.

Who builds the RAID

The same RAID can be built by three different layers.
This server had both hardware RAID (the data SSDs) and firmware RAID (the OS NVMe drives).

The orange outline marks where RAID is computed. With hardware RAID the OS does not know about the RAID, so its health has to be watched from outside the OS.

Type	Built by	On this server
Hardware RAID	A dedicated controller card	A `MegaRAID SAS-3 3108` controller combines three SSDs into RAID 5, and the OS sees a single 7T disk, `sda`
Firmware RAID	A chipset-defined format plus an OS driver	The two NVMe drives show up as `isw_raid_member`. Linux `mdadm` reads the Intel chipset (IMSM) RAID metadata and assembles it as `md126`
Software RAID	The OS (`mdadm`, LVM, ZFS)	None

mdadm is the tool that creates and manages Linux software RAID, the kernel's md (Multiple Devices) devices. It can also read the IMSM format used by Intel chipset RAID, which is how firmware RAID gets assembled on Linux.

LVM (Logical Volume Manager) is a Linux tool that builds virtual block devices from physical disks, letting you pool several disks and carve them into the sizes you need.

ZFS combines a file system and a volume manager: it groups disks itself (RAIDZ and others), keeps a checksum for every block to detect corruption on read, and repairs it when redundancy is available.

The difference shows when something breaks.
With hardware RAID the OS only sees the combined disk, so when a disk inside it dies, everything looks fine from the OS.
You have to watch it separately, through the controller’s own tool (storcli) or alerts from the server management board.
Firmware and software RAID can be checked directly from the OS with cat /proc/mdstat.
Two healthy disks show as [UU], and a missing one as [U_].

Summary

Choosing a RAID level means choosing what to gain and what to give up among speed, capacity, and tolerance to failure.
RAID 0 gains speed and capacity but survives no failure at all, while RAID 1 gives up half the capacity to survive one failure.
RAID 5 and 6 save capacity with parity while still surviving failures, and RAID 10 gives up capacity to get both write speed and resilience.

For an OS disk, staying up matters more than capacity or speed.
So the OS disks should have been RAID 1, not RAID 0.

References

mdadm(8) manual https://man7.org/linux/man-pages/man8/mdadm.8.html
Linux kernel documentation, RAID arrays (md) https://docs.kernel.org/admin-guide/md.html
lvm(8) manual https://man7.org/linux/man-pages/man8/lvm.8.html
LVM2 project https://sourceware.org/lvm2/
OpenZFS documentation, Basic Concepts https://openzfs.github.io/openzfs-docs/Basic%20Concepts/index.html
OpenZFS documentation, Checksums and Their Use in ZFS https://openzfs.github.io/openzfs-docs/Basic%20Concepts/Data%20Storage/Checksums.html