Following my previous failure to get my Pi 5 booting Ubuntu from an NVMe SSD, I did some more debugging and (eventually) got it working.

I was going to give up and just run Raspberry Pi OS. However, after flashing the image to an SD card I noticed the Pi was taking a long time and failing to boot.

It turns out the solution to my problem all along was connecting a monitor. When I did this I saw,

Gave up waiting for root file system device.  Common problems:
  - Boot args (cat /proc/cmdline)
    - Check rootdelay= (did the system wait long enough?)
  - Missing modules (cat /proc/modules; ls/dev)
ALERT!  LABEL=writable does not exist.  Dropping to a shell!

That is interesting. Something was looking for a partition with the label writeable. I checked the fstab file in the recovery shell.

(initramfs) mkdir mnt
(initramfs) mount /dev/nvme0n1p2 mnt
(initramfs) cat mnt/etc/fstab
LABEL=writeable	/	ext4	defaults	0	1
LABEL=system-boot	/boot/firmware	vfat	defaults	0	1

Ok, the entries don’t correspond to labels on the NVMe SSD. I went back to raspberry Pi OS to edit fstab.

jim@chosokabe:~ $ sudo mkdir /mnt/clone
jim@chosokabe:~ $ sudo mount /dev/nvme0n1p2 /mnt/clone
jim@chosokabe:~ $ sudo blkid
/dev/nvme0n1p1: UUID="6237-F650" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="e9a45259-01"
/dev/nvme0n1p2: LABEL="root" UUID="6336b841-9d44-4726-b26c-956c14bb2c16" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="e9a45259-02"
/dev/mmcblk0p1: LABEL_FATBOOT="bootfs" LABEL="bootfs" UUID="5DF9-E225" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="565217e7-01"
/dev/mmcblk0p2: LABEL="rootfs" UUID="3b614a3f-4a65-4480-876a-8a998e01ac9b" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="565217e7-02"

I used the PARTUUIDs.

PARTUUID=e9a45259-02 / ext4 defaults 0 1
PARTUUID=e9a45259-01 /boot/firmware vfat defaults 0 1

No joy, I got the same error. Failing to find a partition with the expected label must happen before mounting drives. Next I gave the root partition on the NVMe SSD the label writable.

jim@chosokabe:~ $ sudo e2label /dev/nvme0n1p2 writable
Recovering journal.
jim@chosokabe:~ $ lsblk -f
NAME        FSTYPE FSVER LABEL    UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
mmcblk0
├─mmcblk0p1 vfat   FAT32 bootfs   5DF9-E225                               448M    12% /boot/firmware
└─mmcblk0p2 ext4   1.0   rootfs   3b614a3f-4a65-4480-876a-8a998e01ac9b   25.5G     7% /
nvme0n1
├─nvme0n1p1 vfat   FAT32          6237-F650
└─nvme0n1p2 ext4   1.0   writable 6336b841-9d44-4726-b26c-956c14bb2c16

Now it boots and I can see the init process going through its steps.

There was another problem though. systemd-networkd-wait-online.service/start runs for a long time and eventually fails. After that, however, I get the login prompt and am able to login.

The problem was, the Pi wasn’t connected to ethernet and wifi was failing to connect. I found that there was a file at /etc/netplan/50-cloud-init.yaml specifying network configuration. This isn’t something I’ve seen or used before. However, the file did look odd to me.

…
network:
    ethernets:
        wlan0:
            dhcp4: true
            match:
                macaddress: d8:3a:dd:a5:9a:30
            set-name: wlan0
    version: 2

wlan0 in ethernets despite being a wifi device. eth0 is not specified at all. I decided not to fix this, as long term I didn’t want to use wifi anyway. I Connected an ethernet cable and the network wait service didn’t hold up the system.

jim@chosokabe:~$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0         7:0    0   169M  1 loop /snap/lxd/25953
loop1         7:1    0  68.5M  1 loop /snap/core22/867
loop2         7:2    0 134.1M  1 loop /snap/lxd/27054
loop3         7:3    0  35.5M  1 loop /snap/snapd/20298
loop4         7:4    0  35.2M  1 loop /snap/snapd/20674
nvme0n1     259:0    0 476.9G  0 disk
├─nvme0n1p1 259:1    0   512M  0 part /boot/firmware
└─nvme0n1p2 259:2    0 476.4G  0 part /

🎉🥳

I installed network-manager and used nmcli to disable with wlan0 to stop the network wait service causing problems in the future.

What was Happening?⌗

I thought it was strange that after cloning the Ubuntu image from the SD card to the NVMe SSD,

  • When there was an SD card with Ubuntu
    • The Pi continued to boot into the system on the SD card even when the SSD was first in the boot order.
  • When there was no SD card
    • The Pi tries to boot from SSD and eventually fails
  • When there was an SD card with Raspberry Pi OS
    • The Pi tries to boot from SSD and eventually fails

I now think that the Pi was booting from the NVMe drive all along. However, because it was looking for a partition with the label writeable,

  • With the Ubuntu SD card
    • It finds such a partition and mounts it.
  • With Raspberry Pi OS, or no SD card
    • There is no such partition to mount, so boot fails.

The question I asked was, how do I boot from the NVMe SSD? However, the question I should have been asking was, how do I mount the partitions on the NVMe SSD after booting from it?

Post Post Update⌗

Part of running kubernetes on the Raspberry Pi required changing the kernel parameters. On the Pi these are defined at firmware/cmdline.txt on the boot partition. This file defines the root partition, which was root=LABEL=writeable. I changed this to use PARTUUID as it feels more stable than using labels.

I should have remembered this from configuring systemd-boot loaders on my Arch systems, but I didn’t 😅. This is probably the better way to fix booting after rpi-clone rather than changing partition labels. I suspect you may still have to update fstab in either case.