-
Notifications
You must be signed in to change notification settings - Fork 91
feat: add wrappers for ATB and ACLNN fused operators. #474
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
feat: add wrappers for ATB and ACLNN fused operators. #474
Conversation
| #elif defined(USE_CUDA) | ||
| cuda::act_and_mul(params.output, params.input, params.act_mode); | ||
| #else | ||
| LOG(FATAL) << "active not implemented"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remove torch::Tensor active_tensor(ActivationParams& params) and add params.output = npu::active(params.input, params.act_mode) here for npu device.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
auto output = torch::empty(
{batch_size,
intermediate_size_ / parallel_args_.tp_group_->world_size()},
gate_up.options());
This is a good modification. However, as described, the current code's output still allocates space preemptively. For NPU operators, they typically allocate their own space and return the result. This unavoidable difference still forces the external calling code to use an #if block to skip space allocation specifically for the NPU case.
To standardize the external calling code, I personally recommend aligning with the NPU's behavior: allocate the space within the operator wrapper/layer and then return it. This approach allows for a unified code structure for all external calls.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so don't add active_tensor and fused_layernorm_tensor these two func in ops_api.h, because no other platform will use such api.
put they in npu_ops_api.h and call them directly in npu layer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Regarding the code snippet above: if we implement the changes as suggested, we would need to introduce #if directives here to skip memory allocation, since the NPU operator handles this internally.
Could we instead consider moving the memory allocation logic for MLU and CUDA into their respective kernel wrappers? This would make the behavior more similar to PyTorch and allow us to unify the calling code here.
(PS: I haven't modified the CUDA or MLU code yet.)
| #endif | ||
| } | ||
|
|
||
| torch::Tensor fused_layernorm_tensor(FusedLayerNormParams& params) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same as above
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Similar to the previous comment.
xllm/core/kernels/param.h
Outdated
| // Must be less than or equal to rope_seqlen if not using discrete | ||
| // position_ids. | ||
| int64_t max_query_len; | ||
| torch::Tensor positions; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
std::optional<torch::Tensor> position_ids already exists.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
During the implementation, I noticed that position_ids are set to empty during the prefill stage, so I initially added position. However, I see that the latest CUDA code addresses the same issue using a different approach. To ensure consistency, I plan to align my implementation with the CUDA method.
7485463 to
9711f41
Compare
No description provided.