Skip to content

Conversation

@Angazenn
Copy link
Collaborator

@Angazenn Angazenn commented Nov 10, 2025

What this PR does / why we need it?

This is cherry-pick from #4097 .
Currently, we set seq_lens in dummy attn_metadata to be max_model_len to get max workspace for attention during capturing.
However, setting it consistently to be max_model_len causing dummy_run to execute a long attention when running actual inference. For example, if there is a single req with seqs_lens as [8] but max_model_len is 131072, the whole process will be slow down by dummy_run as it execute a fake long-seq attention. Therefore, we instead set it to max_query_len, which is also consistent with vLLM gpu implementation.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request aims to optimize dummy attention runs by reducing seq_lens to 1 when not capturing a graph. The change correctly uses the force_attention flag to distinguish between capturing and other dummy runs. However, the accompanying comment is confusing and contradicts the code's logic, which could lead to future maintenance issues. I've suggested an updated comment to accurately reflect the implementation.

@github-actions
Copy link

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

  • A PR should do only one thing, smaller PRs enable faster reviews.
  • Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
  • Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

@Angazenn Angazenn changed the title [cherry-pick][v0.11.0-dev][bugfix] Reduce seq_lens in dummy attn_metadata to be 1 when not capturing [cherry-pick][v0.11.0-dev][bugfix] Change seq_lens in dummy attn_metadata to max_query_len Nov 11, 2025
Signed-off-by: Angazenn <[email protected]>
Signed-off-by: Angazenn <[email protected]>
@yiz-liu yiz-liu merged commit 28a1529 into vllm-project:v0.11.0-dev Nov 12, 2025
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants