Skip to content

[Security] A PEFT adapter package decides which base model FastChat downloads and serves #3942

Open

Description

@3em0

Summary

FastChat activates its PEFT adapter path for any --model-path that contains an adapter_config.json (fastchat/model/model_adapter.py:633-637), which is exactly how docs/model_support.md:98-101 tells users to serve a LoRA adapter. PeftModelAdapter.load_model() then reads base_model_name_or_path out of that file and uses it as the instruction for what to load next — the only check on the value is a substring test for the literal peft:

config = PeftConfig.from_pretrained(model_path)
base_model_path = config.base_model_name_or_path
if "peft" in base_model_path:
    raise ValueError(
        f"PeftModelAdapter cannot load a base model with 'peft' in the name: {config.base_model_name_or_path}"
    )

The value is then handed to the base-model adapter, which resolves it as a Hugging Face repository id and downloads it (fastchat/model/model_adapter.py:681-685 → BaseModelAdapter.load_model() at :105-132):

# In the normal case, load up the base model weights again.
base_adapter = get_model_adapter(base_model_path)
base_model, tokenizer = base_adapter.load_model(
    base_model_path, from_pretrained_kwargs
)
model = PeftModel.from_pretrained(base_model, model_path)
tokenizer = AutoTokenizer.from_pretrained(
    model_path,
    use_fast=self.use_fast_tokenizer,
    revision=revision,
    trust_remote_code=True,
)
...
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    low_cpu_mem_usage=True,
    trust_remote_code=True,
    **from_pretrained_kwargs,
)

Nothing constrains the origin of that second repository, and the revision is whatever the user passed as --revision (main by default). A LoRA adapter package is the normal distribution unit for adapters, so whoever publishes one decides which repository the victim's machine downloads the model from — and the model FastChat then serves is the attacker's, not the one the user believes they are running. The base model also supplies the tokenizer, so the prompt/stop-token handling comes from the same repository.

Affected versions and environment

  • lm-sys/FastChat (PyPI fschat): last release v0.2.36 (b21d0f780ca4472a13714262a0790f2ee1ade659, 2024-02-11) carries the same code at model_adapter.py:618-619, 642, 655; current main at 587d5cfa1609a43d192cedb8441cac3c17db105d (2025-06-02) is what was reproduced here. fastchat/model/model_adapter.py is byte-identical between that checkout and today's main (SHA-256 364bada5bd04f915ce94ccbca59cbfd95446231e91688a6a2259d4c9fd4b2a43), so there is no fix to test against at report time.
  • Both entry points that call the loader are affected: python -m fastchat.serve.cli (→ fastchat/serve/inference.py:361) and python -m fastchat.serve.model_worker (→ fastchat/serve/model_worker.py:77).
  • Documented activation of the path: docs/model_support.md:98-101 ("Any Peft adapter … To activate, must have peft in the model path"), together with PEFT_SHARE_BASE_WEIGHTS; the same pointer is read a second time in PeftModelAdapter.get_default_conv_template() (model_adapter.py:688-699).
  • Reproduced on Windows 11 x64 / Python 3.12.10 / transformers 4.42.4 / peft 0.11.1 / accelerate 0.31.0 / torch 2.12.0+cpu. The run is offline: HF_ENDPOINT points at a localhost mock of the Hub HTTP API, so every fetch the product performs is observable, and the real huggingface_hub download code path is used unmodified.

Steps to reproduce

  1. Build two data-only second-hop repositories that are identical except for two tensor values: mbe2e-control/fastchat-base and mbe2e-attacker/fastchat-base. Each contains only config.json, generation_config.json, model.safetensors, tokenizer.json, tokenizer_config.json, special_tokens_map.json — no .py, no auto_map, no pickle. The differences are model.embed_tokens.weight[0,0] (0.25 vs 0.75, a measurement oracle) and one row of lm_head.weight that makes the model's greedy first token spell out its own repository id (lm_head.weight 64/7296 elements, model.embed_tokens.weight 1/7296 elements).

  2. Produce one PEFT adapter with PEFT's own save_pretrained() (rank 1, target_modules=["q_proj"], all-zero LoRA tensors) and copy it twice. The copies are byte-identical except one line of adapter_config.json; adapter_model.safetensors has the same SHA-256 in both (10b409b9f4d76f7011cb22740abeb1fe7f6511feb49aca713bf74f4a82d1aeb0):

    • work/adapter_attack_peft/adapter_config.json: "base_model_name_or_path": "mbe2e-attacker/fastchat-base"
    • work/adapter_benign_peft/adapter_config.json: "base_model_name_or_path": "mbe2e-control/fastchat-base"
  3. Run the documented command on the malicious adapter and type one line ("Hello") at its Human: prompt:

    python -m fastchat.serve.cli --model-path work\adapter_attack_peft --device cpu --temperature 0 --max-new-tokens 4
    
  4. The CLI exits cleanly (code 0) and its answer comes out of the repository named in the adapter package — each second-hop repository was built to answer with its own id:

    Human: Assistant: mbe2e-attacker/fastchat-base own mbe2e-attacker/fastchat-base own
    
  5. Every request the run makes goes to that repository, and only to it:

    HEAD /mbe2e-attacker/fastchat-base/resolve/main/tokenizer_config.json
    GET  /mbe2e-attacker/fastchat-base/resolve/main/tokenizer_config.json
    HEAD /mbe2e-attacker/fastchat-base/resolve/main/tokenizer.json
    GET  /mbe2e-attacker/fastchat-base/resolve/main/tokenizer.json
    HEAD /mbe2e-attacker/fastchat-base/resolve/main/special_tokens_map.json
    GET  /mbe2e-attacker/fastchat-base/resolve/main/special_tokens_map.json
    HEAD /mbe2e-attacker/fastchat-base/resolve/main/config.json
    GET  /mbe2e-attacker/fastchat-base/resolve/main/config.json
    HEAD /mbe2e-attacker/fastchat-base/resolve/main/model.safetensors
    GET  /mbe2e-attacker/fastchat-base/resolve/main/model.safetensors
    HEAD /mbe2e-attacker/fastchat-base/resolve/main/generation_config.json
    GET  /mbe2e-attacker/fastchat-base/resolve/main/generation_config.json
    
  6. Second, independent reading of the same call: driving fastchat.model.model_adapter.load_model() (the function chat_loop uses) directly shows the loaded model is a PeftModelForCausalLM whose config._name_or_path is mbe2e-attacker/fastchat-base and whose base embed_tokens.weight[0,0] is 0.75 — the attacker repository's value. The benign adapter returns 0.25 / mbe2e-control/fastchat-base.

  7. Negative control — the benign adapter, byte-identical apart from that one JSON line, contacts only mbe2e-control/fastchat-base, answers with the control repository's id, and never contacts the attacker repository.

Expected behavior

Which base weights a locally obtained adapter is merged into is the user's trust decision. A string read out of the adapter package should not silently choose a remote origin, and when the base model is not the one the user named, the product should not proceed silently.

Actual behavior

base_model_name_or_path is used as-is to select, download and load the second-hop repository. The substring test (if "peft" in base_model_path) only rejects base paths that literally contain peft; it says nothing about origin, revision, owner or integrity. The user sees no message naming the repository that was downloaded, and the CLI then serves the attacker's tokenizer and weights as if they were the adapter's intended base.

Impact

  • Origin selection by model data: a crafted adapter package makes the victim's machine contact and download a model from a repository the attacker chose, at its current default revision (no pin unless the user passes --revision).
  • Silent model substitution: the model that is loaded and served is the attacker's, while the adapter the user chose shows up as the source of the deployment. Prompts, tokenization and outputs are all handled by the substituted model.
  • Data-only is enough: the two repositories used here contain no program code, so the effect does not depend on remote code. Note, however, that the loader passes trust_remote_code=True (model_adapter.py:112, 122) for whatever repository the adapter names, so a second-hop repository that also ships an auto_map/.py entry would have that code loaded as well — that variant was not exercised in this report (the repositories used here declare no auto_map, and no .py was fetched).
  • Availability: because the attacker also chooses the model's size and shape, an oversized or malformed second-hop repository can exhaust memory or abort the run (not demonstrated here).

Suggested fix

  1. Do not treat base_model_name_or_path as an instruction to fetch an arbitrary origin. Require the base model to come from the user (CLI option / config), or validate it against an allow-list, and fail closed otherwise. At most, warn when the adapter's value differs from the user's choice.
  2. When a non-default repository must be accepted, print the resolved repository and revision, require explicit confirmation, and pin a revision instead of defaulting to main.
  3. Record the resolved base repository and revision wherever the served model is reported (model worker log / API model card).
  4. Reconsider trust_remote_code=True for a repository that was selected by a downloaded adapter package.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions