Skip to content

Llama4 FP8 Training Debug - fairscale #1183

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 15 commits into
base: main
Choose a base branch
from
Open

Llama4 FP8 Training Debug - fairscale #1183

wants to merge 15 commits into from

Conversation

@jiecaoyu
Copy link

@jiecaoyu jiecaoyu commented May 7, 2024

What does this PR do?

Fixes # (issue).

Before submitting

  • Did you have fun?
    • Make sure you had fun coding 🙃
  • Did you read the contributor guideline?
  • Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
    • N/A
  • Did you make sure to update the docs?
    • N/A
  • Did you write any new necessary tests?
    • N/A
  • Did you update the changelog? (if needed)
    • N/A

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

{{Naman Goyal}} and others added 15 commits April 11, 2022 11:09
fix for high gpu reserved memory
ac5e712
@stephenroller
Get rid of warning
1915531
@ngoyal2707
bf16 changes and attribute for cpu activations
ba38cf3
changes for pp
8e5c416
@vedanuj
changes for fp8
e910320
@jianyuh
Fix fsdp+pp+te WPS decreasing issue (#1139)
0db6e62 
* Fix fsdp+pp+te WPS decreasing issue

* Address comment; remove unused stuff

* split into wps fix P841842878 only and main_grad fix
@awgu
Removed extra cat before reduce-scatter (#1141)
71aeffe
@awgu
Cleared backward hooks to avoid accumulating over iterations (#1143)
17ecf4a
@artkorenev
fix no shard case (#1150)
a8189f0 
Co-authored-by: Artem Korenev <[email protected]>
@ngoyal2707 @vedanuj @jiecaoyu
[not to be merged yet] added temp changes for fp32 main grad, might n…
3b7cc24 
…ot work for TE (#1151)

* added temp changes for fp32 main grad, might not work for TE

* post rebase

* changes to keep reduced grad in fp32 (#1152)

* fix .grad=None issue when param is not sharded (#1153)

* fixed broken clipping (#1154)

Co-authored-by: Naman Goyal <[email protected]>

---------

Co-authored-by: Naman Goyal <[email protected]>
Co-authored-by: Vedanuj Goswami <[email protected]>
Co-authored-by: Jiecao Yu <[email protected]>
Added reshard hook for frozen params in backward
a4f02ef
@jiecaoyu
Avoid calling _free_fp16_param_shard() too early with PR 1159
f2bb56f
@whbldhwj
Added requires_grad check for params_with_grad method (#1171)
1307b1d 
Co-authored-by: Jie Wang <[email protected]>
@awgu
Changed to only run reshard hook if all gradients computed (#1166)
5faca97 
* Changed to only run reshard hook if all gradients computed

* Fix decreasing it/s with multi-grad hook
@whbldhwj
Add cast input argument (#1175)
7bcbc80 
Co-authored-by: Jie Wang <[email protected]>
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

10 participants

@jiecaoyu @facebook-github-bot @stephenroller @ngoyal2707 @vedanuj @jianyuh @awgu @artkorenev @whbldhwj