Prepare the disks

First we need to prepare the disks for the RAID, to do that we will use the System Rescue CD as it has all the tools we will need, we start with parted:

(here we use MB units to ensure partition alignment, which we can check for in the end, although the partition actual size may differ, for small sizes)

root# parted /dev/sda
(parted) mklabel gpt
(parted) unit MB
(parted) mkpart ESP 0\% 100MB
(parted) set 1 esp on
(parted) mkpart primary linux-swap 100MB 16.1GB
(parted) mkpart primary 16.1GB 56.1GB
(parted) mkpart primary 56.1GB 100\%

(parted) print

Number Start End Size File System Name Flags
1 1049kB 99.6MB 98.6MB fat32 EFI msftdata
2 99.6MB 16.1GB 16.0GB linux-swap Linux RAID
3 16.1GB 56.1GB 40GB Linux RAID
4 56.1GB 4001GB 3945GB Linux RAID

(parted) align-check optimal 1
1 aligned
(parted) align-check optimal 2
2 aligned
(parted) align-check optimal 3
3 aligned
(parted) align-check optimal 4
4 aligned

Copy partition layouts using sgdisk:

root# sgdisk --replicate=/dev/sd{b..d} /dev/sda

Randomize UUID for each partition:

root# sgdisk -G /dev/sd{b..d}

Note that the recommended size for the EFI Partition ranges from 100 to 250 MB.

Create RAID array:

root# mdadm --create --metadata=1.0 --name=hostname:efi /dev/md/efi --level=mirror --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
root# mdadm --create --metadata=1.0 --name=hostname:swap /dev/md/swap --level=mirror --raid-devices=4 /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2
root# mdadm --create --metadata=1.2 --name=hostname:root /dev/md/root --level=mirror --raid-devices=4 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3

The other partition is left alone as it is out of the scope of this guide.

Time to reboot to the Ubuntu Server image.

Install Ubuntu 16.04

After choosing/preseeding the usual installer checkboxes, we arrive at partman stage, here we can usually see our raid arrays already assemble. In order to choose the right md device we can switch to a separate tty (usually by Alt+F<2-6>, or Alt+<left/right>) and either check in /etc/mdadm/mdadm.conf for the mdXXX name corresponding to our array, or run:

root# cat /proc/mdstat

followed by:

root# mdadm --detail /dev/mdXXX

so that we can check the name of each md device.

After we are sure of which is the ESP, the swap and the root partitions, we can choose accordingly back in the installer menu (Alt+F1, or Alt+F7)

Here we choose EFI System Partition for the efi, SWAP space for the swap, and ext4 mounted on / for the root filesystem.

All done!

Troubleshooting

But wait, sometimes you can (likely) come across issues when the installer tries to run grub-install.

This may be because you did not boot the installer in UEFI.

If you did and still get the error, keep reading.

Let's reboot into rescue-mode from the Ubuntu installer by choosing rescue a broken system

Now lets choose to execute a shell into the installed system, with the root md device (see above to be sure how to sure the root md device) as the root filesystem.

root# mount /boot/efi

If you get an error, confirm that your UUID is correct in /etc/fstab

Check for installed files in /boot and /boot/efi.

If you have no files there:

root# update-grub
root# grub-install --efi-directory=/boot/efi --uefi-secure-boot

If your UEFI boots without NVRAM entries, you can disable the NVRAM writing via the “Update NVRAM variables to automatically boot into Debian?” debconf prompt when running [ref]:

root# dpkg-reconfigure -p low grub-efi-amd64

If not, you can use efibootmgr to edit the EFI boot entries.

In order to use efibootmgr the system has to be in EFI mode, run the following command to check for the existence of efivarfs:

root# mount | grep efivars
efivarfs on /sys/firmware/efi/efivars type efivarfs (ro,relatime)

In this case it is mounted read-only through the sysfs init script, so it needs to be remounted rw manually using the following command:

root# mount -o remount,rw -t efivarfs efivarfs /sys/firmware/efi/efivars

Let's take a look at the EFI entries

root# efibootmgr -v
...

If there is no entry corresponding to /dev/disk/by-partuuid/** -> /dev/sd{a..d}, we need to add them manually:

root# for i in a b c d; do efibootmgr -c -d /dev/sd\${i} -p 1 -L "ubuntu" -l "\\efi\\ubuntu\\shimx64.efi"; done
root# efibootmgr

For more info about efibootmgr check out the Gentoo wiki

Now we should be ready to go!

But wait, there is more...

... if the UEFI writes anything to one of the drives, it may lead to corrupted results once Linux mounts the RAID (since the member drives won’t have identical block-level copies of the FAT32 any more).

To deal with this "external write" situation, and since mdadm has the "--update=resync" assembly option, we can use the following approach, :

Prefer one RAID member’s copy of /boot/efi and rebuild the RAID at every boot. If there were no external writes, there’s no issue. My approach here is I really do not care which version of the ESP gets copied over, if there is something to be written in the ESP and it gets copied over to all RAID members we should be fine.

This requires updating /etc/mdadm/mdadm.conf.

I've tried adding on the RAID’s ARRAY line to keep it from auto-starting:

1

ARRAY  metadata=1.0 UUID=XXX

however it seems that mdadm refuses to assemble the array as it is explicitly ignored. So I just delete the ARRAY line corresponding to the efi md device.

We now add the efi partition to the fstab (or modify the existing one):

1

UUID=XXXX-XXXX /boot/efi vfat noauto,defaults 0 0

And finally we can add a systemd oneshot service that will assemble the RAID with resync and mount it:

/lib/systemd/system/efimount.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

[Unit]
Description=Resync /boot/efi RAID
DefaultDependencies=no
After=local-fs.target

[Service]
Type=oneshot
ExecStart=/sbin/mdadm -A /dev/md/efi --uuid=YY:Y:Y:Y:YYY --update=resync
ExecStart=/bin/mount /boot/efi
RemainAfterExit=yes

[Install]
WantedBy=sysinit.target

We also run:

root# update-initramfs -u

so the initramfs has an updated copy of /etc/mdadm/mdadm.conf

NOTE: XXXX-XXXX is the MD device's uuid, however YY:Y:Y:Y:YYY is the UUID common for each partition /dev/sd{a..d}1 (our ESP partition) that you can get by running:

root# blkid | egrep "/dev/sd.?1"

Now yes, we can reboot and provision the system!

A big thanks to Kees Cook as his blog was one of the very limited resources about this use case on the web.