Skip to content

[Feat] Support non-gated activations in NVFP4 modelopt path - #29004

Merged
mgoin merged 8 commits into
vllm-project:mainfrom
omera-nv:feat/nvfp4_non_gated_activations
Nov 30, 2025
Merged

mgoin merged 8 commits into
vllm-project:mainfrom
omera-nv:feat/nvfp4_non_gated_activations

Conversation

@omera-nv

@omera-nv omera-nv commented Nov 19, 2025 •

Copy link
Copy Markdown
Contributor

Purpose

For upcoming Nvidia releases, we require support for running non-gated activations for NVFP4 modelopt checkpoints.
This PR enables this by:

  1. Modifying the sizes of the modelopt nvfp4 MoE weights according to the value of self.moe.is_act_and_mul
  2. Calling the flashinfer cutlass fused moe kernel, which supports this feature. Since this is only supported in this path, running such models requires providing the following environment variables: VLLM_USE_FLASHINFER_MOE_FP4=1 VLLM_FLASHINFER_MOE_BACKEND=throughput
  3. Fixing an issue with scale padding that is more common in non-gated models: If the intermediate size is not a multiple of 128, the swizzling logic pads the weight scales. However, for the flashinfer kernel to work the sizes of the weights and scales must match. Therefore, we added a logic to pad the weights of the up and down projections as well to match the padded size of the weight scales. This is more common in non-gated models since if the model is gated, we only require its intermediate size to be a multiple of 64, since there are 2 weights packed into w13_weight_scale in such a case.

Test Plan

We've fixed and expanded the flashinfer fused moe test to also test a non-gated activation (relu2).

Test Result

Old tests were fixed and new tests are passing internally.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for non-gated activations in the NVFP4 modelopt path, which is a valuable feature enhancement. The changes are comprehensive, covering modifications to weight and scale sizing based on whether the activation is gated, and introducing padding logic for weights to match padded scales, which is particularly relevant for non-gated models. The use of the flashinfer cutlass fused MoE kernel is correctly enabled for this feature. The tests have been appropriately expanded to cover the new non-gated activation path, ensuring the changes are well-verified. I have one point of feedback regarding a potentially misleading assertion message.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines 1117 to 1121
self._load_combined_w13_weight_scale(
shard_dim=shard_dim,
loaded_weight=loaded_weight,
param=param,
param=param[expert_id],
tp_rank=self.tp_rank,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Pass whole w13 scale tensor to combined loader

The new call to _load_combined_w13_weight_scale passes param[expert_id] instead of the full param. The helper expects the complete (num_experts, …) tensor so it can compute the shard size along the TP-sharded dimension and copy the sliced view of loaded_weight into the correct place. Slicing per expert turns the argument into a 2‑D tensor, so shard_dim now refers to the wrong axis and param.copy_(loaded_weight) will see mismatched shapes. When loading ModelOpt NVFP4 checkpoints where w1 and w3 scales are combined, this will either raise during copy or populate only one expert’s scales, leaving the rest uninitialized. The previous behavior of passing the whole tensor avoids this problem.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@omera-nv could you please check this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

param is correct when the experts are all in a single weight, param[expert_id] is for when experts are kept in separate weights. Seems like expert_data is always correct so I'll use that instead.

@@ -122,6 +122,7 @@ def workspace_shapes(
- Note: in order for activation chunking to work, the first dimension
of each tuple must be the number of tokens.
"""

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: try to avoid whitespace-only changes in otherwise untouched files

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Comment on lines 1117 to 1121
self._load_combined_w13_weight_scale(
shard_dim=shard_dim,
loaded_weight=loaded_weight,
param=param,
param=param[expert_id],
tp_rank=self.tp_rank,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@omera-nv could you please check this?

@omera-nv
omera-nv force-pushed the feat/nvfp4_non_gated_activations branch from e7ea6e5 to d5cd9a2 Compare November 19, 2025 16:15
@omera-nv
omera-nv requested a review from tlrmchlsmth November 19, 2025 16:24
@mergify

mergify Bot commented Nov 21, 2025

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @omera-nv.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Nov 21, 2025
@omera-nv
omera-nv force-pushed the feat/nvfp4_non_gated_activations branch from 373b773 to b41312e Compare November 24, 2025 12:34
@mergify mergify Bot removed the needs-rebase label Nov 24, 2025
@omera-nv
omera-nv force-pushed the feat/nvfp4_non_gated_activations branch from b41312e to dead075 Compare November 24, 2025 12:55

@tlrmchlsmth tlrmchlsmth left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me now, thanks!

@github-project-automation github-project-automation Bot moved this to In review in NVIDIA Nov 25, 2025
@tlrmchlsmth

Copy link
Copy Markdown
Member

Please fix the merge conflicts, and I'll mark the PR ready and turn on automerge

@mergify

mergify Bot commented Nov 25, 2025

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @omera-nv.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Nov 25, 2025
@omera-nv
omera-nv force-pushed the feat/nvfp4_non_gated_activations branch from dead075 to 1aa3008 Compare November 26, 2025 11:03
@mergify mergify Bot removed the needs-rebase label Nov 26, 2025
@mgoin mgoin added quantization ready ONLY add when PR is ready to merge/full CI is needed labels Nov 26, 2025
Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
omera-nv and others added 5 commits November 30, 2025 10:35
Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
@Naveassaf
Naveassaf force-pushed the feat/nvfp4_non_gated_activations branch from 1aa3008 to 31ecb63 Compare November 30, 2025 08:35
@mgoin
mgoin merged commit 39d2810 into vllm-project:main Nov 30, 2025
51 checks passed
@github-project-automation github-project-automation Bot moved this from In review to Done in NVIDIA Nov 30, 2025
amd-hhashemi pushed a commit to amd-hhashemi/vllm that referenced this pull request Dec 2, 2025
mystous pushed a commit to mystous/vllm_hybrid that referenced this pull request May 10, 2026
my-other-github-account pushed a commit to my-other-github-account/vllm that referenced this pull request May 15, 2026
0826joyce pushed a commit to 0826joyce/vllm-serving-optimization that referenced this pull request May 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

nvidia ready ONLY add when PR is ready to merge/full CI is needed

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants