So, after a few days of trying I had to get it working in Debian 11 instead of Debian 12. Bookworm was just being difficult. So, I decided to try Debian 11 and it worked right on the first try. So, I am documenting my changes both to ProxMox and also the VM that I installed from a fresh Debian 11 cloud image.
So, this is a good place to start about the changes to the Host server running ProxMox. In this case, I have a SuperMicro X10DRi-T4+ and it has been a great host sever the week I have had it running. However, I had some issues when I initially tried to add the NVIDIA Tesla K80 I got off ebay. Oh these things do not fit into a normal desktop case as they are designed for servers and had to remove the back bracket. I will probably make a bracket to fix the looseness that this causes but at this point I just wanted to test running the Tesla in ProxMox and into a Docker container. Anyway make sure
Above 4G Decoding
I did need to enable this feature because the Tesla K80 is a beast and as soon as I plugged it into my Sever it cause the system to run out of PCI resources. So, simple fix was to extend above the 4G address space. Thank God this board supports this feature! Once I enabled this on the board all was good and system would boot without any further issues.
Fix ProxMox Grub
First, have to modify /etc/default/grub so that it has proper iommu for Intel or AMD
Next, it is time to run upgrade grub command.
Blacklist Graphics Cards on ProxMox
Next, blacklist the graphics so ProxMox does not touch them!
cat << 'EOF' > /etc/modprobe.d/blacklist.conf
Once this is done this can be added to the modprob via cmd
Load Needed Modules
So, it is important to load the vivo modules so I am going to create a vivo.conf in modules-load.d
cat << 'EOF' > /etc/modules-load.d/vfio.conf
#Added as part of GPU Passthrough
Stop Annoying Warnings on ProxMox
I was getting notifications of errors related to rdmsr and wtmsr so the ProxMox Forum recommended a fix.
echo "options kvm ignore_msrs=1 report_ignored_msrs=0" > /etc/modprobe.d/kvm.conf
Now I am not getting the messages any longer.
Time To Reboot
With all of these changes it is time to reboot!