Repository navigation
[Feat] Support non-gated activations in NVFP4 modelopt path - #29004
Conversation
There was a problem hiding this comment.
Code Review
This pull request adds support for non-gated activations in the NVFP4 modelopt path, which is a valuable feature enhancement. The changes are comprehensive, covering modifications to weight and scale sizing based on whether the activation is gated, and introducing padding logic for weights to match padded scales, which is particularly relevant for non-gated models. The use of the flashinfer cutlass fused MoE kernel is correctly enabled for this feature. The tests have been appropriately expanded to cover the new non-gated activation path, ensuring the changes are well-verified. I have one point of feedback regarding a potentially misleading assertion message.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| self._load_combined_w13_weight_scale( | ||
| shard_dim=shard_dim, | ||
| loaded_weight=loaded_weight, | ||
| param=param, | ||
| param=param[expert_id], | ||
| tp_rank=self.tp_rank, |
There was a problem hiding this comment.
Pass whole w13 scale tensor to combined loader
The new call to _load_combined_w13_weight_scale passes param[expert_id] instead of the full param. The helper expects the complete (num_experts, …) tensor so it can compute the shard size along the TP-sharded dimension and copy the sliced view of loaded_weight into the correct place. Slicing per expert turns the argument into a 2‑D tensor, so shard_dim now refers to the wrong axis and param.copy_(loaded_weight) will see mismatched shapes. When loading ModelOpt NVFP4 checkpoints where w1 and w3 scales are combined, this will either raise during copy or populate only one expert’s scales, leaving the rest uninitialized. The previous behavior of passing the whole tensor avoids this problem.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
param is correct when the experts are all in a single weight, param[expert_id] is for when experts are kept in separate weights. Seems like expert_data is always correct so I'll use that instead.
| @@ -122,6 +122,7 @@ def workspace_shapes( | |||
| - Note: in order for activation chunking to work, the first dimension | |||
| of each tuple must be the number of tokens. | |||
| """ | |||
|
|
|||
There was a problem hiding this comment.
nit: try to avoid whitespace-only changes in otherwise untouched files
| self._load_combined_w13_weight_scale( | ||
| shard_dim=shard_dim, | ||
| loaded_weight=loaded_weight, | ||
| param=param, | ||
| param=param[expert_id], | ||
| tp_rank=self.tp_rank, |
e7ea6e5 to
d5cd9a2
Compare
|
This pull request has merge conflicts that must be resolved before it can be |
373b773 to
b41312e
Compare
b41312e to
dead075
Compare
tlrmchlsmth
left a comment
There was a problem hiding this comment.
Looks good to me now, thanks!
|
Please fix the merge conflicts, and I'll mark the PR ready and turn on automerge |
|
This pull request has merge conflicts that must be resolved before it can be |
dead075 to
1aa3008
Compare
Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com> Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
1aa3008 to
31ecb63
Compare
…ject#29004) Signed-off-by: Hashem Hashemi <hashem.hashemi@amd.com>
Purpose
For upcoming Nvidia releases, we require support for running non-gated activations for NVFP4 modelopt checkpoints.
This PR enables this by:
self.moe.is_act_and_mulVLLM_USE_FLASHINFER_MOE_FP4=1 VLLM_FLASHINFER_MOE_BACKEND=throughputw13_weight_scalein such a case.Test Plan
We've fixed and expanded the flashinfer fused moe test to also test a non-gated activation (relu2).
Test Result
Old tests were fixed and new tests are passing internally.