Add support for VLLM as whisper execution engine

[p88h]

Runs VLLM in greedy decoding mode with high batch parallelism. Tested up to batch 128 on an RTX 4080.

For AMI dataset with large-v3 model, this configuration achieves:
WER: 16.0 % RTFx: 63.56

It seems a bit faster than transformers backend, mostly thanks to wider possible batch size (that maxes out at 32 on the same GPU, achieving RTFx of 53.76).

Should scale proportionally on better hardware, and allow even wider batch sizes with more GPU memory.

It achieves slightly higher WER at the moment (16 vs 15.94), at least for this model.
For distil-large-v2, the results were almost identical for WER, again with some performance advantage.

BTW this particular dataset is likely not very representative of AMI in general and the (very low) WER results don't translate well to that whole original dataset (with very long recordings). When testing on 30s chunks, most models perform at ~25-ish WER, rather than ~15.

[p88h]

Hey - any updares here ? @Vaibhavs10 @Deep-unlearning

[p88h]

ping ?

[nithinraok]

@Deep-unlearning

[Deep-unlearning]

Hi @p88h and @nithinraok, thanks for the ping! I’ll be running the VLLM benchmarks on the datasets shortly and will share the verified WER / RTFx numbers here shortly. Once that’s done I’ll go ahead and add the results to the Whisper leaderboard.

[Deep-unlearning]

I ran the evaluation with openai/whisper-large-v3 and got the following results:

Composite Results:
openai/whisper-large-v3: WER = 7.45 %
openai/whisper-large-v3: RTFx = 129.66

Breakdown per dataset:

• AMI: 16.01 %, RTFx = 67.41
• Earnings22: 11.29 %, RTFx = 132.12
• GigaSpeech: 10.08 %, RTFx = 110.92
• LibriSpeech (clean): 2.04 %, RTFx = 135.80
• LibriSpeech (other): 3.89 %, RTFx = 124.96
• SPGISpeech: 2.95 %, RTFx = 148.46
• TEDLIUM: 3.84 %, RTFx = 131.54
• VoxPopuli: 9.52 %, RTFx = 163.54

The WERs are in line with the transformers backend, but I would have expected a higher RTFx (faster inference) from the vLLM backend. Are we sure the backend switch is properly impacting decoding speed?

I also tested openai/whisper-tiny.en, and the results seem unusually bad:

Composite Results:
openai/whisper-tiny.en: WER = 91.78 %
openai/whisper-tiny.en: RTFx = 584.87

Breakdown per dataset:

• AMI: WER = 99.10 %, RTFx = 241.97
• Earnings22: WER = 93.63 %, RTFx = 476.19
• GigaSpeech: WER = 87.53 %, RTFx = 507.63
• LibriSpeech (clean): WER = 80.68 %, RTFx = 537.27
• LibriSpeech (other): WER = 88.43 %, RTFx = 516.17
• SPGISpeech: WER = 96.19 %, RTFx = 727.25
• TEDLIUM: WER = 96.50 %, RTFx = 527.08
• VoxPopuli: WER = 92.22 %, RTFx = 669.52

This looks like something went wrong, possibly a decoding issue or mismatch in config. Could you help investigate?

[p88h]

Interesting - I haven't played much with the tiny model but that looks like there is some issue indeed.
I'll have a look over the weekend.