Skip to content

from_pretrained no longer uses mmap for CPU weights in transformers 5.x causing full materialization #44262

Closed

Description

@nikitas-cerebras

I have the following piece of code:

import psutil
from transformers import AutoModel


rss_before = psutil.Process().memory_info().rss
model = AutoModel.from_pretrained(
    "Qwen/Qwen3-Coder-30B-A3B-Instruct",
    dtype="auto",
    device_map="auto",
    max_memory={"cpu": "1024GB", 0: 0},
)
rss_after = psutil.Process().memory_info().rss
print(f"Memory: {(rss_after - rss_before) >> 20} MiB")

When running it on transformers 4.57, I get:

Loading checkpoint shards: 100%|██████████| 16/16 [00:00<00:00, 17.03it/s]
Memory: 1321 MiB

When running it on transformers 5.2, I get:

Loading weights: 100%|██████████| 530/530 [00:13<00:00, 39.48it/s, Materializing param=norm.weight]
Memory: 110705 MiB

I see that the mode loading was reworked. It seems like on the old transformers version model weights are loaded with memory mapping, while on the new transformers version all the weights are materialized.

I wonder how to achieve the old behavior with the new transformers?

Activity

  1. Rocketknight1 commented on Feb 25, 2026

    @Rocketknight1
    Member

    cc @Cyrilvallez I think!

  2. NielBuys commented on Feb 25, 2026

    @NielBuys

    Hi everyone

    Is this potentially the same issue causing 4-bit quantization spikes?

    I'm seeing what looks like the same materialization behavior when attempting to load Gemma 3 12B with 4-bit quantization in Transformers v5.1.0. Even with device_map="auto", the model appears to materialize at full precision before the quantizer can act, leading to immediate OOM.

    I’ve documented the logs and a detailed breakdown of the failure here: [Link to forum post]

  3. NielBuys commented on Feb 25, 2026

    @NielBuys

    Maybe another issue with the same problem #43749

  4. Cyrilvallez commented on Mar 4, 2026

    @Cyrilvallez
    Member

    What you are seeing is merely the fact that safetensors mmaps the tensors (even though we used to, and still do, explicitly slice the safetensors slice objects to materialize them with t = t[...]).
    However, almost all operation that need to access them and/or copy them will then put them into memory. For example, if you try to use dtype=torch.float16 in your example, it will force a copy with the new dtype, and the memory will become the full model. Now, since MoE models have automatic weight conversions to a new format since v5, we need to copy the weights at load time no matter what, so they end up materializing the memory immediately.
    Note that you are NOT saving any memory, i.e. that memory WILL be materialized when running a forward anyway.

    Consider the following example with Llama, which does not need any weight conversion:

    import psutil
    import torch
    from transformers import AutoModel
    
    model_id = "meta-llama/Llama-3.3-70B-Instruct"
    # model_id = "Qwen/Qwen3-Coder-30B-A3B-Instruct"
    
    
    rss_before = psutil.Process().memory_info().rss / 1024**3
    model = AutoModel.from_pretrained(
        model_id,
        dtype="auto",
        # dtype=torch.float16,
    )
    rss_after = psutil.Process().memory_info().rss / 1024**3
    print(f"Memory: {(rss_after - rss_before):.2f} GiB")
    
    tot = 0
    for k,v in model.state_dict().items():
        tot += v.element_size() * v.nelement()
    print(f"Total model size: {tot / 1024**3:.2f} GiB")
    
    
    rss_before = psutil.Process().memory_info().rss / 1024**3
    print(f"Memory before forward: {(rss_before):.2f} GiB")
    foo = torch.randint(100, 200, (1, 20))
    _ = model(foo)
    rss_after = psutil.Process().memory_info().rss / 1024**3
    print(f"Memory after forward: {(rss_after):.2f} GiB")

    It prints:

    Memory: 0.08 GiB
    Total model size: 129.46 GiB
    Memory before forward: 0.75 GiB
    Memory after forward: 128.91 GiB

    Now, if you change dtype="auto" -> dtype=torch.float16 (not native dtype, so requires a copy), it prints:

    Memory: 129.51 GiB
    Total model size: 129.46 GiB
    Memory before forward: 130.19 GiB
    Memory after forward: 131.04 GiB

    And you can notice that everything was materialized since the beginning.

    Now, for Qwen it's a bit more particular, as it's a MoE. So before v5 and automatic weight conversion, if you don't change the dtype it would be mmaped, and after forward you would only materialize the experts you used during that forward, resulting in less memory than the full model size. However, as you keep calling forward, you will eventually hit all experts, and then memory consumption is the same.

    Now, after all of those explanations, Qwen memory consumption in v5 is still too high conmpared to shat it should be if loading the model immediately with e.g. a non-native dtype - I will investigate to see if some memory is wrongly not released as it should!

  5. nikitas-cerebras commented on Mar 5, 2026

    @nikitas-cerebras
    Author

    @Cyrilvallez Thank you for the explanation! In my particular case I perform some kind of model analysis and don't actually call a model's forward that's why for me this difference is crucial.

    I guess in theory it should be possible to postpone merging of 2D MoE weights into a 3D tensor until the actual inference. And from the blogpost https://huggingface.co/blog/moe-transformers I was getting an idea that that could be the case based on the quote "For example, MergeModulelist waits until all experts for a layer are loaded." under the "Lazy Materialization of Tensors" section. But perhaps it means something else. Anyway thanks to your explanation I now understand that there is no such logic.

    For anyone having the same issues, I was able to transition to loading the model as:

    config = AutoConfig.from_pretrained("Qwen/Qwen3-Coder-30B-A3B-Instruct")
    with init_empty_weights(include_buffers=False):
        model = AutoModelForCausalLM.from_config(config)
    )
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions