So, after a few days of trying I had to get it working in Debian 11 instead of Debian 12. Bookworm was just being difficult. So, I decided to try Debian 11 and it worked right on the first try. So, I am documenting my changes both to ProxMox and also the VM that I installed from a fresh Debian 11 cloud image.

ProxMox

So, this is a good place to start about the changes to the Host server running ProxMox. In this case, I have a SuperMicro X10DRi-T4+ and it has been a great host sever the week I have had it running. However, I had some issues when I initially tried to add the NVIDIA Tesla K80 I got off ebay. Oh these things do not fit into a normal desktop case as they are designed for servers and had to remove the back bracket. I will probably make a bracket to fix the looseness that this causes but at this point I just wanted to test running the Tesla in ProxMox and into a Docker container. Anyway make sure

Above 4G Decoding

I did need to enable this feature because the Tesla K80 is a beast and as soon as I plugged it into my Sever it cause the system to run out of PCI resources. So, simple fix was to extend above the 4G address space. Thank God this board supports this feature! Once I enabled this on the board all was good and system would boot without any further issues.

Fix ProxMox Grub

First, have to modify /etc/default/grub so that it has proper iommu for Intel or AMD

nano /etc/default/grub

#Intel
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
#AMD
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"

Next, it is time to run upgrade grub command.

update-grub

Blacklist Graphics Cards on ProxMox

Next, blacklist the graphics so ProxMox does not touch them!

cat << 'EOF' > /etc/modprobe.d/blacklist.conf
blacklist amdgpu
blacklist nouveau
blacklist nvidia
blacklist radeon
EOF

Once this is done this can be added to the modprob via cmd

Load Needed Modules

So, it is important to load the vivo modules so I am going to create a vivo.conf in modules-load.d

cat << 'EOF' > /etc/modules-load.d/vfio.conf
#Added as part of GPU Passthrough
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
EOF

Stop Annoying Warnings on ProxMox

I was getting notifications of errors related to rdmsr and wtmsr so the ProxMox Forum recommended a fix.

echo "options kvm ignore_msrs=1 report_ignored_msrs=0" > /etc/modprobe.d/kvm.conf

Now I am not getting the messages any longer.

Time To Reboot

With all of these changes it is time to reboot!

reboot