Validation loss become higher after 20 hours training time

[misbullah]

Hi,
Yesterday, I tried to run speech.yml from examples.
In the beginning, the validation loss is about 200.xxx, but after training for 20 hours, the validation become higher. Now, it is about 630.xxx.

Is the any problem with this training process?

Because, I checked the graph on deepgram blog: http://blog.deepgram.com/how-to-train-baidus-deepspeech-model-with-kur/, the validation loss become smaller for more iteration.

Thanks.

[scottstephenson]

Does looking at #6 help? Seeing your loss plot would help too.

[misbullah]

Hi @scottstephenson,

Yes, I checked it already. Simple question, how to create loss plot from deepgram training process?
Is there any documentation for it?

I checked that the kur use tensorflow for backend. Does it also support wrap-ctc like implemented in the following git:
https://github.com/mozilla/DeepSpeech

Thanks.

[scottstephenson]

Have a look at the tutorial: https://kur.deepgram.com/tutorial.html

The tensorflow backend uses the Tensorflow CTC implementation, PyTorch and Theano use warp CTC.