Skip to content

Training capabilities for Stream #92

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
JeremyMorlier wants to merge 17 commits into
base: master
Choose a base branch
from
Open

Training capabilities for Stream #92

JeremyMorlier wants to merge 17 commits into from

Conversation

@JeremyMorlier
Copy link

@JeremyMorlier JeremyMorlier commented Oct 8, 2025

Added and Modified features to make stream capable of evaluating training workloads:

New Workloads :

  • Resnet18 with full fine-tuning, Cross Entropy Loss (not considered because of Rtree creation fails over 1D accumulation #78) and ADAM optimizer
  • Resnet18 with LoRa, Cross Entropy Loss and ADAM optimizer
  • Decoder-like LLM with full fine-tuning, Cross Entropy Loss and ADAM optimizer

Nodes:

  • Gemm: added transpose attribute (as done in ZigZag)
  • FusedMatMul
  • ConvTranspose
  • ReduceSum
  • SoftmaxCrossEntropyGrad

Dependency Propagation:

  • Squeeze
  • Unsqueeze
  • Pad
  • Gather: Generalized to search the indices among the ONNX graph initializers and to more axis
  • Reshape: Generalized to adding/removing more than one axis
  • Concat: Generalized for both non constant inputs
  • Split : Search the split index in the initializers and inputs

Tiled Workload Generation

  • Replaced the compute_shortest_path with custom BFS as it selected paths that contain computation nodes

@JeremyMorlier
Added Nodes for Backward Computation
6fb635f 
- ConvTranspose
- InPlaceAccumulator
- SoftmaxCrossEntropy
- SoftmaxCrossEntropyGrad
- ReduceSum
Modifications:
- Changed Reshape for Convolution to apply to TransposedConvolution
@JeremyMorlier
Added New Nodes Parser:
392acec 
- AveragePoolGrad: similar to a ConvTranspose Parser
- ReluGrad: modified SIMDParser
- ReduceSum: Reimplementation using directly the equation (could be used to implement reduce1DParser)
- Pad Parser

Fix GemmTranspose Operator
Modified Gather (and Pad) Parser to catch initializer and not only constants inputs

Parallelization strategies for the new operators
@JeremyMorlier
Replaced NertworkX shortest path that could fail in case of taking th…
786fdd4 
…e wrong path by a BFS that return path without computation node
@JeremyMorlier
Added 0 predecessors caser (may need some changes)
5fcdfcc
@JeremyMorlier
small utility to get the value of an ONNX intializer
33fcf2e
@JeremyMorlier
Updated Asymmetric SIMD to support more dimensions
372fece
@JeremyMorlier
removed logging override of root level
98aa1e7
@JeremyMorlier
Merge1
d98d8d4
@JeremyMorlier
Fix all issues linked to merge (cleanup and correctness still required)
491a4ee
@JeremyMorlier
merge 6971524
0ff9cc9
@JeremyMorlier
update
a3edd4d
@JeremyMorlier
Updated dependencies
762482d 
Generalization of Gather and Reshape relevant_axes
@JeremyMorlier
fixed padding operation the did not work if the node was used multipl…
6a1d0e3 
…e times in tiled workload computation
@JeremyMorlier
Removed Operators
e2c7891 
-  ConvGrad: not directly usable in Stream
-  SoftmaxCrossEntropyLoss : breaks Stream KULeuven-MICAS#78
Added Operators
-  FusedMatMul
-  Squeeze
-  Unsqueeze
Modified Operators :
-  Concat Operator : Allowed for both non constants inputs (0 are concat into the tensor)
Ruff checked all files
@JeremyMorlier
added training workloads
b31385e
@JeremyMorlier
Adapted for Stream main
be19b84
@JeremyMorlier
typo
faa2059
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@JeremyMorlier