> but if you boot up Ubuntu in Docker, wouldn't the NVIDIA drivers still load?

Even if the drivers loaded, they can't talk to the GPU from within docker (unless one implements PCI passthrough). MacOS owns the PCI bus in this scenario.