You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
from_pretrained no longer uses mmap for CPU weights in transformers 5.x causing full materialization #44262
I see that the mode loading was reworked. It seems like on the old transformers version model weights are loaded with memory mapping, while on the new transformers version all the weights are materialized.
I wonder how to achieve the old behavior with the new transformers?
Is this potentially the same issue causing 4-bit quantization spikes?
I'm seeing what looks like the same materialization behavior when attempting to load Gemma 3 12B with 4-bit quantization in Transformers v5.1.0. Even with device_map="auto", the model appears to materialize at full precision before the quantizer can act, leading to immediate OOM.
I’ve documented the logs and a detailed breakdown of the failure here: [Link to forum post]
What you are seeing is merely the fact that safetensors mmaps the tensors (even though we used to, and still do, explicitly slice the safetensors slice objects to materialize them with t = t[...]).
However, almost all operation that need to access them and/or copy them will then put them into memory. For example, if you try to use dtype=torch.float16 in your example, it will force a copy with the new dtype, and the memory will become the full model. Now, since MoE models have automatic weight conversions to a new format since v5, we need to copy the weights at load time no matter what, so they end up materializing the memory immediately.
Note that you are NOT saving any memory, i.e. that memory WILL be materialized when running a forward anyway.
Consider the following example with Llama, which does not need any weight conversion:
Memory: 0.08 GiB
Total model size: 129.46 GiB
Memory before forward: 0.75 GiB
Memory after forward: 128.91 GiB
Now, if you change dtype="auto" -> dtype=torch.float16 (not native dtype, so requires a copy), it prints:
Memory: 129.51 GiB
Total model size: 129.46 GiB
Memory before forward: 130.19 GiB
Memory after forward: 131.04 GiB
And you can notice that everything was materialized since the beginning.
Now, for Qwen it's a bit more particular, as it's a MoE. So before v5 and automatic weight conversion, if you don't change the dtype it would be mmaped, and after forward you would only materialize the experts you used during that forward, resulting in less memory than the full model size. However, as you keep calling forward, you will eventually hit all experts, and then memory consumption is the same.
Now, after all of those explanations, Qwen memory consumption in v5 is still too high conmpared to shat it should be if loading the model immediately with e.g. a non-native dtype - I will investigate to see if some memory is wrongly not released as it should!
@Cyrilvallez Thank you for the explanation! In my particular case I perform some kind of model analysis and don't actually call a model's forward that's why for me this difference is crucial.
I guess in theory it should be possible to postpone merging of 2D MoE weights into a 3D tensor until the actual inference. And from the blogpost https://huggingface.co/blog/moe-transformers I was getting an idea that that could be the case based on the quote "For example, MergeModulelist waits until all experts for a layer are loaded." under the "Lazy Materialization of Tensors" section. But perhaps it means something else. Anyway thanks to your explanation I now understand that there is no such logic.
For anyone having the same issues, I was able to transition to loading the model as:
Activity
Rocketknight1 commentedon Feb 25, 2026
cc @Cyrilvallez I think!
NielBuys commentedon Feb 25, 2026
Hi everyone
Is this potentially the same issue causing 4-bit quantization spikes?
I'm seeing what looks like the same materialization behavior when attempting to load Gemma 3 12B with 4-bit quantization in Transformers v5.1.0. Even with device_map="auto", the model appears to materialize at full precision before the quantizer can act, leading to immediate OOM.
I’ve documented the logs and a detailed breakdown of the failure here: [Link to forum post]
NielBuys commentedon Feb 25, 2026
Maybe another issue with the same problem #43749
Cyrilvallez commentedon Mar 4, 2026
What you are seeing is merely the fact that
safetensorsmmaps the tensors (even though we used to, and still do, explicitly slice the safetensors slice objects to materialize them witht = t[...]).However, almost all operation that need to access them and/or copy them will then put them into memory. For example, if you try to use
dtype=torch.float16in your example, it will force a copy with the new dtype, and the memory will become the full model. Now, since MoE models have automatic weight conversions to a new format since v5, we need to copy the weights at load time no matter what, so they end up materializing the memory immediately.Note that you are NOT saving any memory, i.e. that memory WILL be materialized when running a
forwardanyway.Consider the following example with Llama, which does not need any weight conversion:
It prints:
Now, if you change
dtype="auto"->dtype=torch.float16(not native dtype, so requires a copy), it prints:And you can notice that everything was materialized since the beginning.
Now, for Qwen it's a bit more particular, as it's a MoE. So before v5 and automatic weight conversion, if you don't change the dtype it would be mmaped, and after forward you would only materialize the experts you used during that forward, resulting in less memory than the full model size. However, as you keep calling
forward, you will eventually hit all experts, and then memory consumption is the same.Now, after all of those explanations, Qwen memory consumption in v5 is still too high conmpared to shat it should be if loading the model immediately with e.g. a non-native dtype - I will investigate to see if some memory is wrongly not released as it should!
nikitas-cerebras commentedon Mar 5, 2026
@Cyrilvallez Thank you for the explanation! In my particular case I perform some kind of model analysis and don't actually call a model's forward that's why for me this difference is crucial.
I guess in theory it should be possible to postpone merging of 2D MoE weights into a 3D tensor until the actual inference. And from the blogpost https://huggingface.co/blog/moe-transformers I was getting an idea that that could be the case based on the quote "For example, MergeModulelist waits until all experts for a layer are loaded." under the "Lazy Materialization of Tensors" section. But perhaps it means something else. Anyway thanks to your explanation I now understand that there is no such logic.
For anyone having the same issues, I was able to transition to loading the model as: