generate without punctuation

[zephyrtronium]

It would be nice to be able to generate e.g. bocchi: ryo with the colon when given only bocchi without a colon as prompt. This was one of the goals of #45. It doesn't work now, probably due to the extra space added at the end of the last token by brain.Tokens.

[zephyrtronium]

This may also be related to the phenomenon where Robot happens to pick an empty continuation for a prompt and thereby speaks nothing.

[seth-rah]

I'm not sure, but would this also impact capitalization?

If someone uses Bocchi instead of bocchi would that make a difference?

[zephyrtronium]

Robot already ignores capitalization in the prompt, as well as during continuation generally. Probably worth clarifying that the goal here is not to have Robot ignore punctuation for all continuation, only to allow selecting a punctuation token as the first one following the prompt. Currently that doesn't work because Robot affixes a space to the last token in the prompt, which is done primarily because the same tokenization function is used for both learning and speaking and doing that made the former simpler. But since the tokens in bocchi: ryo are bocchi : ryo , it doesn't match when we tokenize a prompt of bocchi into bocchi .