- Fix attention batching issue (thanks @leonardledy!).
- Add support for XPU devices (thanks @rwood-97!).
- Check model checkpoint hashes (thanks @isaaccorley!).
- Remove unused modulation heads in the decoder (thanks @leonardledy!)
- Make activation checkpointing configurable.
- Remove
bf16_modedue to severe numerical issues and bring backautocast. - Fix dropout bug.