Category: TECHNICAL

I’ve recently experienced a number of errors in my proxmox server related to the NVME drives.

{1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 514
{1}[Hardware Error]: It has been corrected by h/w and requires no further action
{1}[Hardware Error]: event severity: corrected
{1}[Hardware Error]:  Error 0, type: corrected
{1}[Hardware Error]:  fru_text: PcieError
{1}[Hardware Error]:   section_type: PCIe error
{1}[Hardware Error]:   port_type: 0, PCIe end point
{1}[Hardware Error]:   version: 0.2
{1}[Hardware Error]:   command: 0x0406, status: 0x0010
{1}[Hardware Error]:   device_id: 0000:a1:00.0
{1}[Hardware Error]:   slot: 0
{1}[Hardware Error]:   secondary_bus: 0x00
{1}[Hardware Error]:   vendor_id: 0x2646, device_id: 0x5013
{1}[Hardware Error]:   class_code: 010802
{1}[Hardware Error]:   bridge: secondary_status: 0x0000, control: 0x0000
nvme 0000:a1:00.0: AER: aer_status: 0x00002001, aer_mask: 0x00000000
nvme 0000:a1:00.0:    [ 0] RxErr                  (First)
nvme 0000:a1:00.0:    [13] NonFatalErr           
nvme 0000:a1:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID

After looking into this. I’ve found a series of articles and posts about this issue:

• https://gist.github.com/zekome/35db528b33206e68f18439ad7fabfcd5
• https://forums.unraid.net/topic/118286-nvme-drives-throwing-errors-filling-logs-instantly-how-to-resolve/
• https://forum.proxmox.com/threads/aer-corrected-error-received-should-i-be-worried.127067/
• https://forums.debian.net/viewtopic.php?t=155031
• https://forum.proxmox.com/threads/help-with-console-message-nvme-aer-error-of-this-agent-is-reported-first.123699/

and sadly none of these posts seem to have a solid answer…. and even worse, most replies are just telling people to silence the errors, or even disabling the error recovery features entirely (Absolutely bonkers).

Follow up

For my server, I’ve found this issue is almost entirely related to the NVME drives overheating.. I’ve installed some thin heatsinks, and the issue has almost completely resolved itself.

I cannot confirm if this is true for the others… but I’ve got a feeling it’s the root cause for my server considering how hot these NVME storage devices are getting on GEN 4 / GEN 5.

Welcome to techno hell. Everything needs a heatsink now.

I grabbed 4 1TB Kingston KC3000 NVME disks, and as I plan on using them in a proxmox server. I’ve investigated their LBA (Logical block addressing) format support to gain better performance by using the larger native block size.

Bonus fact: Some NVME drives also have LBA formats that support on-disk metadata, and this can be bypassed for ZFS or similar filesystems that maintain metadata.

Prerequisite

Install nvme cli for debian/ubuntu (Optional)

apt install nvme-cli

nvme-cli (with human readable output)

nvme id-ns -H /dev/nvme0n1

The LBA part is the important part for now.

Notice: There are 2 LAB Formats (512 and 4096). 512 is the current in use format, but 4096 is marked as giving better relative performance.

smartctl (with all information)

smartctl -a /dev/nvme0

Again, the support LBA size is the important part.

Format the NVME disk to use a different LBA

nvme format --lbaf=1 /dev/nvme0n1

After that. test the result from the previous

Additional Resources

https://wiki.archlinux.org/title/Advanced_Format