Help needed: Loading two models on separate devices

[BaleChen]

Hey, thanks for the great project.

My question is that if I have two LLMs for inference, is there a way to load model A on GPU 0&1, and model B on GPU 2&3? I am using offline inference engine. Is it possible to do it in one script, or do I need to launch the two engines in two processes and use CUDA_VISIBLE_DEVICES to control it?