How to generate english subtitles, with translation of foreign languages, via the CLI

[balupton]

I'm a bit confused about these options:

  --task <task>           Task to perform (transcribe or translate) (default: transcribe)
  --language <language>   Language spoken in the audio

I want to essentially generate two subtitles:

1. A srt file that is only english dialogue is transcribed, without any (speaking in french) etc subtitles, for when there are hardcoded translations on the screen
2. A srt file that is only english, with any foreign language (if any) translated to english

Currently, with:

whisperkit-cli transcribe --verbose --task transcribe --report --skip-special-tokens --chunking-strategy vad --language en --model medium.en

I'm getting .srt files for french documentaries that are mute for like a minute, then some (speaking in french) and then several minutes in starts becoming english translations.

Is it something like where I need to use the large or turbo model, omit --task and --language, but do --prompt like:

1. Transcribe english dialogue only, ignore all other dialogue and sounds.
2. Transcribe all dialogue into english; including translating all non-english dialogue into english. Do not transcribe sounds, only transcribe and translate speech.

Would that work?

[balupton]

Okay this did not work, the transcription is still in french:

whisperkit-cli transcribe --verbose --task transcribe --report --skip-special-tokens --chunking-strategy vad --model large-v3 --prompt Transcribe\ all\ dialogue\ into\ english\;\ including\ translating\ all\ non-english\ dialogue\ into\ english.\ Do\ not\ transcribe\ sounds\,\ only\ transcribe\ and\ translate\ speech. --audio-path 

trying:

whisperkit-cli transcribe --verbose --task translate --report --skip-special-tokens --chunking-strategy vad --model large-v3 --prompt Transcribe\ all\ dialogue\ into\ english\;\ including\ translating\ all\ non-english\ dialogue\ into\ english.\ Do\ not\ transcribe\ sounds\,\ only\ transcribe\ and\ translate\ speech. --audio-path 

now

[balupton]

Okay, that worked, however I tried two prompts:

Transcribe all dialogue into english; including translating all non-english dialogue into english. Do not transcribe sounds, only transcribe and translate speech.

Transcribe all spoken dialogue from the audio. For non-English dialogue, provide an English translation. Do not include sound effects or speaker labels. Format as dialogue subtitles, with English for both original English and translated non-English lines.

Both had weird hallucinations.

I'm now trying without any prompt, and just --task translate.

If that fails, I guess because the french transcription is really good, I'll just generate the french transcription then use a different AI to translate it into english.

[balupton]

Okay without any prompt also failed. Will just do a straight transcription with WhisperKit then send the output to Apple Intelligence, which from a quick experiment, seems to work. Doing a proper test now.

[balupton]

Unfortunately, seems --report-path does not support relative paths. Trying now with an absolute path.

[balupton]

--report-path with an absolute path also didn't work, it keeps doing a NSURL=file:///the-path-to-the.srt/and-again-the-path-to-the.srt

[balupton]

Okay, for the full french subtitle file, apple intelligence doesn't like it:

Error: The request exceeded the maximum allowed length.

So I need to figure something else out.

[ZachNagengast]

Prompting whisper is a little different than prompting an LLM, mainly its more of an "example" of how the output should look, such as certain spellings or capitalization, versus a logical sentence with instructions. Read more info here: https://cookbook.openai.com/examples/whisper_prompting_guide.

For english only text from french audio, you can use
swift run whisperkit-cli transcribe --model large-v3 --task translate --language fr ...
For french text, use
swift run whisperkit-cli transcribe --model large-v3 --task transcribe --language fr ...

[balupton]

Hrmm.. So to recap the frenzy of my individual comments above:

1. Using --task translate with an ai prompt produced bad results, plenty of poorly translated items, empty items, and hallucinated items.
2. Using --task translate without an ai prompt produced bad results, plenty of poorly translated items, empty items, and hallucinated items.
3. Using --task transcribe then sending to ollama models for translation produced awful results, was always injecting commentary into the translations despite an hour of trying to get it not to.
4. Using --task transcribe then sending to apple intelligence produced good results, but would require sending it in chunks.

The issue I'm facing now is that --task transcribe --model large-v3 without --language still sometimes translates, which causes Apple Intelligence translations to have issues.

whisperkit-cli transcribe --verbose --task transcribe --report --skip-special-tokens --chunking-strategy vad --model large-v3 

Being used on a mp4 of youtube video id Cea8piIE1Jw resulted in a translation to english at 45 minutes thorugh, with the remaining content being in english.

505
00:44:55,179 --> 00:44:58,859
 Pour le reste, j'aurai tout le temps pour me faire accepter par sa famille.

506
00:44:58,859 --> 00:45:04,300
 So they can accept us and everything, right?

507
00:45:04,300 --> 00:45:09,000
 3am, on the Rio Grande.

508
00:45:09,000 --> 00:45:13,000
 The lights in front of us are in Brunsville, Texas.

... everything is now in english, despite the audio remaining in a non-english language

for youtube video id of IBXydJeKhJ0 we get:

106
00:08:57,280 --> 00:08:59,559
 - Vous gagnez plus d'argent avec la pirogue?

107
00:08:59,559 --> 00:09:04,400
 - Oui, ici, au moins, je vois la couleur de l'argent, plus que dans le bois, en tout cas.

108
00:09:04,400 --> 00:09:11,200
 C'est une grosse perte pour moi. Je viens de perdre tout l'argent que j'avais, 7 euros.

109
00:09:18,119 --> 00:09:20,559
 - Did you eat today? - No.

110
00:09:20,720 --> 00:09:22,799
 We didn't eat anything.

111
00:09:22,960 --> 00:09:27,000
 And the money I had to do the shopping fell into the water.

112
00:09:27,799 --> 00:09:35,799
 Manuel Fernandez, son fils et le reste de sa famille devront attendre le ferry du lendemain pour gagner un peu d'argent et faire leur course.

113
00:09:38,000 --> 00:09:41,000
 Sur le bateau, JC, lui, continue de faire des affaires.

114
00:09:41,700 --> 00:09:43,000
 - Vas-y, donne moi une douzaine.

115
00:09:43,000 --> 00:09:44,159
 - 60 centimes.

the same segment more or less with the first translate prompt:

37
00:06:55,700 --> 00:07:21,200


38
00:07:21,200 --> 00:07:45,300


39
00:07:45,299 --> 00:08:01,399


40
00:08:01,399 --> 00:08:26,699


41
00:08:26,700 --> 00:08:52,000


42
00:08:52,000 --> 00:09:11,200


43
00:09:11,200 --> 00:09:27,000


44
00:09:27,000 --> 00:09:43,000


45
00:09:43,000 --> 00:09:45,000
 60 cents.

with the other translate prompt:

35
00:07:19,700 --> 00:07:21,200
 1,500 tons.

36
00:07:21,200 --> 00:07:45,300


37
00:07:45,299 --> 00:08:01,399


38
00:08:01,399 --> 00:08:26,699


39
00:08:26,700 --> 00:08:52,000


40
00:08:52,000 --> 00:09:11,200


41
00:09:11,200 --> 00:09:27,000


42
00:09:27,000 --> 00:09:43,000


43
00:09:43,000 --> 00:10:07,799


44
00:10:07,799 --> 00:10:28,799


45
00:10:28,799 --> 00:10:58,599


46
00:10:58,599 --> 00:11:01,599
 Subtitles by the Amara.org community

with no translate prompt:

106
00:08:57,280 --> 00:08:59,559
 - Vous gagnez plus d'argent avec la pirogue?

107
00:08:59,559 --> 00:09:04,400
 - Oui, ici, au moins, je vois la couleur de l'argent, plus que dans le bois, en tout cas.

108
00:09:04,400 --> 00:09:11,200
 C'est une grosse perte pour moi. Je viens de perdre tout l'argent que j'avais, 7 euros.

109
00:09:18,119 --> 00:09:20,559
 - Did you eat today? - No.

110
00:09:20,720 --> 00:09:22,799
 We didn't eat anything.

111
00:09:22,960 --> 00:09:27,000
 And the money I had to do the shopping fell into the water.

112
00:09:27,799 --> 00:09:35,799
 Manuel Fernandez, son fils et le reste de sa famille devront attendre le ferry du lendemain pour gagner un peu d'argent et faire leur course.

113
00:09:38,000 --> 00:09:41,000
 Sur le bateau, JC, lui, continue de faire des affaires.

114
00:09:41,700 --> 00:09:43,000
 - Vas-y, donne moi une douzaine.

115
00:09:43,000 --> 00:09:44,159
 - 60 centimes.

[balupton]

The issue I'm facing now is that --task transcribe --model large-v3 without --language still sometimes translates, which causes Apple Intelligence translations to have issues.

Okay, I spent the day on this, and have a good solution that works 90% of the time - just fails sometimes for Indonesian translations due toe Apple Intelligence combining subtitle entries.

Will open-source once it's ready.