SAM Audio Large performance on VGGClean

[Whale36377]

First of all, thank you very much for this excellent and inspiring work. I believe SAM Audio is a highly influential contribution to audio-visual separation, and I really appreciate you making the code and benchmark publicly available.
I would like to ask two questions while I was experimenting with SAM Audio Large.
1. Performance of SAM Audio Large on VGGClean test
I evaluated SAM Audio Large on a source separation task using the VGGClean dataset, which, similar to sam_audio_bench, is derived from real-world data but uses relatively 'coarse' audio mixing strategies.
Surprisingly, I found that the separation results are very poor, and in fact most metrics are worse than the input mixture itself. This was not what I initially expected.I used text-only inference with the following setting: result = model.separate(inputs, predict_spans=True, reranking_candidates=1)
This makes me wonder whether:Using only textual prompts is insufficient for this scenario? The model is more sensitive to the mixing strategy used in VGGClean?Or whether this behavior is expected for SAM Audio Large in such settings?I also wanted to ask whether you have tested SAM Audio on VGGClean .
Additionally, I noticed that the paper does not seem to include comparisons with more recent separation-focused models (e.g., MMAudioSep). I was curious whether such comparisons were considered or explored internally.
2. Availability of video IDs in sam_audio_bench
While downloading the videos listed in sam_audio_bench, I noticed that a subset of video IDs (around 90) are no longer accessible.This makes it difficult to fully reproduce the benchmark results. I was wondering whether you have considered release processed version of the dataset?
Again, thank you very much for your outstanding work and sorry for this long text.

[Whale36377]

(text + video)	FAD ↓	IS ↑	CLAP ↑	CLAP-A ↑	IB-Score ↑	DeSync ↓
MIX	2.51	7.53	17.05	59.34	18.85	0.906
SAM_text	5.89	9.35	18.40	48.84	16.46	1.00

[chevalierNoir]

@Whale36377

Hi, thank you for your interest and feedback.

1. Text-only prompting / VGGClean

(a). For text-only inference, we strongly recommend using reranking_candidates=8 (our default). Setting it to 1 can degrade performance quite a bit in generally.
(b). We haven’t evaluated SAM Audio on VGGClean formally. Our experiments mainly focus on real mixtures rather than synthetic mixtures. That being said, in internal tests on synthetic mixtures, model performance is generally better than on real mixtures.
(c). Also please note that objective metrics does not always reflect model performance. For example of synthetic mixture in vggsound, many audios contains ambient sound (e.g., wind blowing) while the text prompt is typically only describing foreground sound (e.g., dog barking). In this case, FID score may be biased as the source audio isn't really the true target. Also CLAP score can be boosted to very high by reranking despite of the artifacts. We recommend subjective evaluation as the primary way of quality assessment.

2. Comparison with MMAudioSep

Thanks for the pointer. We weren’t aware of MMAudioSep at the time, but are happy to look into it.

3. sam_audio_bench video availability

Unfortunately, we can’t release the processed audio/videos since we don’t own the raw media. The data are sourced from the following datasets: AudioSet, VGGSound, MUSIC, MUSIC-AVQA, AVSpeech, and CondensedMovies (Section 5.1.1 of the paper), so you may be able to recover them from the original datasets.

[GuillaumeCarbajal]

@Whale36377 I can't find anything on VGGclean, do you have any reference?