nvme show-regs fails on Raspbian Bookworm.

[pkpfr]

A warning is thrown from line when the script is run on a Raspberry Pi upgraded to Bookworm:

bash: line 398: warning: command substitution: ignored null byte in input

This then causes a failure to submit the benchmarked results.

[rglidden]

I did a little digging, and this (and #29 and #32) appear to be caused by a couple things.

1. The version of nvme-cli shipping with Raspberry Pi OS Bookworm has a bug

On bullseye, calling nvme cli with an invalid device displays a sensible error message:

root@bullseye:~# nvme --version
nvme version 1.12
root@bullseye:~# nvme show-regs /dev/mmcblk0 -H
mmcblk0 is not an nvme device

However, on bookworm, the same command results in erroneous output:

root@pi:~# nvme --version
nvme version 2.3 (git 2.3)
libnvme version 1.3 (git 1.3)
root@pi:~# nvme show-regs /dev/mmcblk0 -H
Unable to find mmcblk0
get-property: Invalid argument
cap : 5555e8db1de0
 Controller Ready With Media Support (CRWMS): Not Supported
 Controller Ready Independent of Media Support (CRIMS): Not Supported
 Controller Memory Buffer Supported (CMBS): The Controller Memory Buffer is Not Supported
 Persistent Memory Region Supported (PMRS): The Persistent Memory Region is Not Supported
 Memory Page Size Maximum (MPSMAX): 4096 bytes
 Memory Page Size Minimum (MPSMIN): 4096 bytes
 Boot Partition Support (BPS): No
 Command Sets Supported (CSS): NVM command set is Not Supported
 One or more I/O Command Sets are Not Supported
 Only Admin Command Set Supported
 NVM Subsystem Reset Supported (NSSRS): Yes
 Doorbell Stride (DSTRD): 128 bytes
 Timeout (TO): 116000 ms
 Arbitration Mechanism Supported (AMS): Weighted Round Robin with Urgent Priority Class is supported
 Contiguous Queues Required (CQR): Yes
 Maximum Queue Entries Supported (MQES): 7649

version : 1f680b90
 NVMe specification 8040.11

cc : 0
 Controller Ready Independent of Media Enable (CRIME): Disabled
 I/O Completion Queue Entry Size (IOCQES): 1 bytes
 I/O Submission Queue Entry Size (IOSQES): 1 bytes
 Shutdown Notification (SHN): No notification; no effect
 Arbitration Mechanism Selected (AMS): Round Robin
 Memory Page Size (MPS): 4096 bytes
 I/O Command Set Selected (CSS): NVM Command Set
 Enable (EN): No

csts : 0
 Processing Paused (PP): No
 NVM Subsystem Reset Occurred (NSSRO): No
 Shutdown Status (SHST): Normal operation (no shutdown has been requested)
 Controller Fatal Status (CFS): False
 Ready (RDY): No

double free or corruption (!prev)
Aborted

Something in this unexpected output is causing the "ignored null byte in input" error that is causing this command to fail.

This appears to be fixed in later versions of nvme-cli, as the version on Debian sid gives more sensible output when run on an unsupported device:

root@rex:~# nvme --version
nvme version 2.6 (git 2.6)
libnvme version 1.6 (git 1.6)
root@rex:~# nvme show-regs /dev/sda -H
Failed to open ns sda, errno 22
Unable to find sda
get-property: Invalid argument
root@rex:~#

2. nvme show-regs may not work anyway due to Kernel changes

As noted in linux-nvme/nvme-cli#2092 (comment), the nvme show-regs command may not work at all, depending on how the kernel is compiled.

This is also a known configuration issue. When the Linux kernel is configured with CONFIG_IO_STRICT_DEVMEM, the kernel prevents user space to map IO memory to userspace. Either compile a kernel without this option enable or set the kernel command line option io_memory=relaxed.

I tried adding the io_memory=relaxed option to the Raspberry Pi OS Bookworm cmdline.txt file, but it still results in the same behaviour.

Temporary Workaround

Instead of running the command

sudo curl https://raw.githubusercontent.com/TheRemote/PiBenchmarks/master/Storage.sh | sudo bash

like instructed on the web site, download the file with

curl -o Storage.sh https://raw.githubusercontent.com/TheRemote/PiBenchmarks/master/Storage.sh

then open the file in your favourite text editor and comment out the offending show-regs line in the script by adding a # at the start of the line.

Test_nvme=$(nvme list -o json 2>&1)
# Test_nvme+=$(nvme show-regs "$BootDrive" -H 2>&1 | sed 's/;/!/g')
Test_smartctl=$(smartctl -x "$BootDrive" 2>&1 | sed 's/;/!/g' | sed '/^[[:space:]]*$/d')

Once you've made the change, run it with sudo bash ./Storage.sh. The results should appear on the web site. If you are using an NVMe boot drive, some details may be missing, but at least the results will appear properly.

[TheRemote]

Thanks for reporting this. I've just made a change to the script that should hopefully catch these null bytes.

Can you try running it again the normal way on the same device and make sure that the submission still happens?

[rglidden]

I can confirm the fix works for me.

[igaw]

FWIW, nvme show-regs is almost always wrong to use. Reading these register has side effects which can lead to data loss. Basically, if you are not debugging/developing firmware don't use it.

I haven't checked what your scripts are doing, but I guess you are interested in some info of the device. In this case you can use nvme id-ctrl and nvme id-ns. Note these commands also have the options to write JSON output and it avoids screen scrapping.

nvme list -o json | jq .Devices[].DevicePath -r