My Pi 5 NVMe Boot Struggles
Background⌗
When Raspberry Pi 5 pre-orders became available, I was quick to order one. I tend to think of myself as frugal and not prone to impulse purchases. However, I do sometimes buy SBCs or microcontrollers with the expectation I will find a use for them. To be fair, I have 4 (soon to be 5) out of 6 Raspberry Pis in use. On the other hand, microcontrollers, 0 out of 6 😅.
More recently, I was looking on Pimoroni after recommending it to a friend. I saw that an NVMe base, enabling storage over the Pi 5’s PCIe interface was available. So, I decided it was time to put my Pi 5 into production. NVMe storage should give a significant improvement in speed and reliability over SD cards or USB storage. My idea is to migrate my blog and Nextcloud to the Pi.
To fit the ribbon cable through the official case, which I wanted to keep due to the active cooling, I made possibly the worst case mod ever. I don’t own a rotary tool and didn’t want to wait for the library of things to open at the weekend, so made do with a combination of my drill, snips and files.
Recognising NVMe Storage⌗
I wanted to run Ubuntu as that is what I’m most familiar with on my servers. The only release currently built for the Pi5 is Mantic (23.10).
I read through Pimoroni’s instructions, Jeff Geerling’s blog post and the Raspberry Pi 5 docs on PCIe boot.
First, after flashing Ubuntu 23.10 to an SD card I looked to see if the NVMe drive was recognised.
jim@chosokabe:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 68.5M 1 loop /snap/core22/867
loop1 7:1 0 169M 1 loop /snap/lxd/25953
loop2 7:2 0 134.1M 1 loop /snap/lxd/27054
loop3 7:3 0 35.5M 1 loop /snap/snapd/20298
loop4 7:4 0 35.2M 1 loop /snap/snapd/20674
mmcblk0 179:0 0 29.7G 0 disk
├─mmcblk0p1 179:1 0 512M 0 part /boot/firmware
└─mmcblk0p2 179:2 0 29.2G 0 part /
That’s a no.
The PCIe interface isn’t enabled by default, so I added a line to /boot/firmware/config.txt to enable the interface at PCIe gen 3 speeds.
[all]
dtparam=pciex1_gen=3
I also checked the firmware after an apt update.
jim@chosokabe:~$ sudo rpi-eeprom-update
BOOTLOADER: up to date
CURRENT: Mon Sep 25 10:44:03 UTC 2023 (1695638643)
LATEST: Mon Sep 25 10:44:03 UTC 2023 (1695638643)
RELEASE: default (/lib/firmware/raspberrypi/bootloader-2712/default)
Use raspi-config to change the release.
That is old and Pimoroni suggests you need a firmware more recent than December 2023 for their base to work.
However, it is the latest firmware in Ubuntu’s rpi-eeprom package, so this requires getting the firmware from another source.
I cloned the rpi-eeprom repository and installed the latest firmware from there, sudo rpi-eeprom-update -f ./rpi-eeprom/firmware-2712/default/pieeprom-2024-02-16.bin
After a reboot, the firmware is now up to date.
jim@chosokabe:~$ sudo rpi-eeprom-update
BOOTLOADER: up to date
CURRENT: Fri Feb 16 15:28:41 UTC 2024 (1708097321)
LATEST: Fri Feb 16 15:28:41 UTC 2024 (1708097321)
RELEASE: default (/lib/firmware/raspberrypi/bootloader-2712/default)
Use raspi-config to change the release.
and …
jim@chosokabe:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 68.5M 1 loop /snap/core22/867
loop1 7:1 0 169M 1 loop /snap/lxd/25953
loop2 7:2 0 134.1M 1 loop /snap/lxd/27054
loop3 7:3 0 35.5M 1 loop /snap/snapd/20298
loop4 7:4 0 35.2M 1 loop /snap/snapd/20674
mmcblk0 179:0 0 29.7G 0 disk
├─mmcblk0p1 179:1 0 512M 0 part /boot/firmware
└─mmcblk0p2 179:2 0 29.2G 0 part /
nvme0n1 259:0 0 476.9G 0 disk
├─nvme0n1p1 259:1 0 512M 0 part
└─nvme0n1p2 259:2 0 476.4G 0 part
Success!
Failing to Boot from NVMe⌗
I changed the boot order to 0xf416 as per the docs.
jim@chosokabe:~ $ sudo rpi-eeprom-config --edit ./rpi-eeprom/firmware-2712/default/pieeprom-2024-02-16.bin
Updating bootloader EEPROM
image: ./rpi-eeprom/firmware-2712/default/pieeprom-2024-02-16.bin
config_src: blconfig device
config: /tmp/tmp2a1wti_t/boot.conf
################################################################################
[all]
BOOT_UART=1
BOOT_ORDER=0xf416
POWER_OFF_ON_HALT=0
################################################################################
*** To cancel this update run 'sudo rpi-eeprom-update -r' ***
*** CREATED UPDATE /tmp/tmp2a1wti_t/pieeprom.upd ***
CURRENT: Fri 16 Feb 15:28:41 UTC 2024 (1708097321)
UPDATE: Fri 16 Feb 15:28:41 UTC 2024 (1708097321)
BOOTFS: /boot/firmware
'/tmp/tmp.SuzyeZcVRu' -> '/boot/firmware/pieeprom.upd'
Copying recovery.bin to /boot/firmware for EEPROM update
EEPROM updates pending. Please reboot to apply the update.
To cancel a pending update run "sudo rpi-eeprom-update -r".
Then I copied the installation from the SD card to the NVMe. Thanks to Jeff Geerling (yet again) for taking over maintenance of rpi-clone.
jim@chosokabe:~$ sudo rpi-clone nvme0n1 -s chosokabe
Booted disk: mmcblk0 31.9GB Destination disk: nvme0n1 512.1GB
---------------------------------------------------------------------------
Part Size FS Label Part Size FS Label
1 /boot/firmware 512.0M fat32 -- 1 512.0M fat32 --
2 root 29.2G ext4 writable 2 476.4G ext4 root
---------------------------------------------------------------------------
== SYNC mmcblk0 file systems to nvme0n1 ==
/boot/firmware (124.0M used) : SYNC to nvme0n1p1 (512.0M size)
/ (2.4G used) : SYNC to nvme0n1p2 (476.4G size)
---------------------------------------------------------------------------
-e clone fstab edit : edit mmcblk0p device entries to nvme0n1p.
Run setup script : rpi-clone-setup chosokabe
Verbose mode : no.
-----------------------:
Ok to proceed with the clone? (yes/no): yes
Syncing file systems (can take a long time)
Syncing mounted partitions:
Mounting /dev/nvme0n1p2 on /mnt/clone
=> rsync // /mnt/clone with-root-excludes ...
Mounting /dev/nvme0n1p1 on /mnt/clone/boot/firmware
=> rsync /boot/firmware/ /mnt/clone/boot/firmware ...
Running setup script: rpi-clone-setup chosokabe
chosokabe - target hostname
/mnt/clone/etc/hostname - set new hostname:
chosokabe
/mnt/clone/etc/hosts - set new hostname "chosokabe" in lines:
127.0.1.1 chosokabe chosokabe
===============================
Done with clone to /dev/nvme0n1
Start - 16:31:45 End - 16:31:54 Elapsed Time - 0:09
Cloned partitions are mounted on /mnt/clone for inspection or customizing.
Hit Enter when ready to unmount the /dev/nvme0n1 partitions ...
unmounting /mnt/clone/boot/firmware
unmounting /mnt/clone
===============================
Reboot and … it is still booting from the SD card 😞.
jim@chosokabe:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 68.5M 1 loop /snap/core22/867
loop1 7:1 0 169M 1 loop /snap/lxd/25953
loop2 7:2 0 134.1M 1 loop /snap/lxd/27054
loop3 7:3 0 35.5M 1 loop /snap/snapd/20298
loop4 7:4 0 35.2M 1 loop /snap/snapd/20674
mmcblk0 179:0 0 29.7G 0 disk
├─mmcblk0p1 179:1 0 512M 0 part /boot/firmware
└─mmcblk0p2 179:2 0 29.2G 0 part /
nvme0n1 259:0 0 476.9G 0 disk
├─nvme0n1p1 259:1 0 512M 0 part
└─nvme0n1p2 259:2 0 476.4G 0 part
Did I do something wrong? The NVMe drive is recognised. The firmware is up to date.
jim@chosokabe:~$ sudo rpi-eeprom-config
[all]
BOOT_UART=1
POWER_OFF_ON_HALT=0
BOOT_ORDER=0xf416
The boot order seems correct, with 6 in the least significant bit. According to the docs that should put NVMe boot first.
Adding PCI_PROBE=1, as the docs suggest, didn’t have an effect.
Later, it didn’t turn out to be necessary to boot Raspberry Pi OS with the Pimoroni NVMe base.
So, is the Ubuntu partition on the NVMe drive bootable?
jim@chosokabe:~$ sudo fdisk -l
…
Disk /dev/nvme0n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors
Disk model: Patriot M.2 P300 512GB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xe9a45259
Device Boot Start End Sectors Size Id Type
/dev/nvme0n1p1 * 2048 1050623 1048576 512M c W95 FAT32 (LBA)
/dev/nvme0n1p2 1050624 1000215215 999164592 476.4G 83 Linux
…
It seemed so 🤷.
What’s Next?⌗
- Is there a log showing an attempt to boot from NVMe. I suppose this is before the kernel starts, so I’m really not sure what that would be. If there is, that could reveal why booting two identical images works for the SD card, but fails for the NVMe drive.
- Can I get it to work with Raspberry Pi OS? (The answer is yes, that blog post may appear in the future)