Reduce the number of Physical Device enumerations #10782
Description
After we merge #10738
We need to reduce how many places we call EnumeratePhysicalDevices, EnumerateDeviceExtensionProperties, etc ... really should only have to be done once for everyone