How much vram needed to run the model??

[nendonerd]

I tried with sam3-audio-large on RTX Pro 6000 with 96G vram, and still got "cuda out of memory" error. Although on hf it's just 16G, it downloads and runs multiple other models together. So how much vram do we need for each size of the sam3-audio models?

[mrfireflygenai]

Now configs are broken.
I haven't seen a single successful confirmation of this model running.
Alas, I wasted 14 hours on this mess.

[Eydahn8]

Lucky you guys, you can at least test it😂 they still haven’t approved many of us on Hugging Face. Funny how they called this project “open source”, then randomly deny people access to the models on Hugging Face with no clear criteria.

[aziz-hakiri]

@nendonerd 96G vram only for loading the models ? How long is your Input data?
I saw that "Longform audio separation with Multi-diffusion" is mentioned in the paper but I do not find the implementation in the code.

[nendonerd]

@nendonerd 96G vram only for loading the models ? How long is your Input data? I saw that "Longform audio separation with Multi-diffusion" is mentioned in the paper but I do not find the implementation in the code.

my input is just 8 mins of mp3 audio. the error says i do not have enough vram to load the models

[nendonerd]

tried the sam-audio-small one and got "cuda out of memory" again. it's annoying that "open source" models don't test reproducibility, don't publish hardware requirements, and assume people have lab grade massive gpu clusters.

[chevalierNoir]

@nendonerd 8-minute audio is too long. you can try chunking it into 30s and run each chunk through the model separately.

[chevalierNoir]

@nendonerd 96G vram only for loading the models ? How long is your Input data? I saw that "Longform audio separation with Multi-diffusion" is mentioned in the paper but I do not find the implementation in the code.

multi-diffusion isn't put in this repo yet.

[notedit]

30G+

[notedit]

✓ 模型加载成功 (GPU 使用: 30.90GB)
✓ 音频处理完成
✓ 输出文件生成:
- output_speech.wav (1.4MB) - 提取的语音
- output_residual.wav (1.4MB) - 剩余音频

[nendonerd]

✓ 模型加载成功 (GPU 使用: 30.90GB) ✓ 音频处理完成 ✓ 输出文件生成: - output_speech.wav (1.4MB) - 提取的语音 - output_residual.wav (1.4MB) - 剩余音频

which model did you use and what's the format and duration of your audio input?

[Cazforshort]

I can confirm both the small and large models work on a 4090 large was much slower though.at 1min/4s audio, but large was significantly improved over small.

[nendonerd]

got the sam-audio-small working, using the examples/assets/office.mp4 as input, taking 28G of vram at peak

[nendonerd]

sam-audio-large takes 38G vram at peak for examples/assets/office.mp4 as input

[katealhola]

I am able to run with Ubuntu + RTX5090 with 32G of VRAM . However maximum audio clip size was around 20s. Even sam-audio-small is 7GM loading model requires 21.6G of VRAM and sam-audio-base is 24G VRAM. Then loading 20s audio for processing takes rest of VRAM. I tried quantisation to FP16 and model does not work any more, it runs but does not separate audio correctly. May be that there is only 10 bit mantissa in FP16. May be i need lo took if some unised models like video processing is loaded when only audio is processed ?

[chevalierNoir]

@katealhola current model loads the span prediction model by default, which has several billion parameter itself. It may not be always helpful (say when you extract instruments, see section 7.4 in the paper for more details). removing it from the config will save you quite a bit of memory.