Repository navigation
Description
Summary
FastChat activates its PEFT adapter path for any --model-path that contains an adapter_config.json (fastchat/model/model_adapter.py:633-637), which is exactly how docs/model_support.md:98-101 tells users to serve a LoRA adapter. PeftModelAdapter.load_model() then reads base_model_name_or_path out of that file and uses it as the instruction for what to load next — the only check on the value is a substring test for the literal peft:
config = PeftConfig.from_pretrained(model_path)
base_model_path = config.base_model_name_or_path
if "peft" in base_model_path:
raise ValueError(
f"PeftModelAdapter cannot load a base model with 'peft' in the name: {config.base_model_name_or_path}"
)The value is then handed to the base-model adapter, which resolves it as a Hugging Face repository id and downloads it (fastchat/model/model_adapter.py:681-685 → BaseModelAdapter.load_model() at :105-132):
# In the normal case, load up the base model weights again.
base_adapter = get_model_adapter(base_model_path)
base_model, tokenizer = base_adapter.load_model(
base_model_path, from_pretrained_kwargs
)
model = PeftModel.from_pretrained(base_model, model_path)tokenizer = AutoTokenizer.from_pretrained(
model_path,
use_fast=self.use_fast_tokenizer,
revision=revision,
trust_remote_code=True,
)
...
model = AutoModelForCausalLM.from_pretrained(
model_path,
low_cpu_mem_usage=True,
trust_remote_code=True,
**from_pretrained_kwargs,
)Nothing constrains the origin of that second repository, and the revision is whatever the user passed as --revision (main by default). A LoRA adapter package is the normal distribution unit for adapters, so whoever publishes one decides which repository the victim's machine downloads the model from — and the model FastChat then serves is the attacker's, not the one the user believes they are running. The base model also supplies the tokenizer, so the prompt/stop-token handling comes from the same repository.
Affected versions and environment
lm-sys/FastChat(PyPIfschat): last releasev0.2.36(b21d0f780ca4472a13714262a0790f2ee1ade659, 2024-02-11) carries the same code atmodel_adapter.py:618-619, 642, 655; currentmainat587d5cfa1609a43d192cedb8441cac3c17db105d(2025-06-02) is what was reproduced here.fastchat/model/model_adapter.pyis byte-identical between that checkout and today'smain(SHA-256364bada5bd04f915ce94ccbca59cbfd95446231e91688a6a2259d4c9fd4b2a43), so there is no fix to test against at report time.- Both entry points that call the loader are affected:
python -m fastchat.serve.cli(→fastchat/serve/inference.py:361) andpython -m fastchat.serve.model_worker(→fastchat/serve/model_worker.py:77). - Documented activation of the path:
docs/model_support.md:98-101("Any Peft adapter … To activate, must havepeftin the model path"), together withPEFT_SHARE_BASE_WEIGHTS; the same pointer is read a second time inPeftModelAdapter.get_default_conv_template()(model_adapter.py:688-699). - Reproduced on Windows 11 x64 / Python 3.12.10 / transformers 4.42.4 / peft 0.11.1 / accelerate 0.31.0 / torch 2.12.0+cpu. The run is offline:
HF_ENDPOINTpoints at a localhost mock of the Hub HTTP API, so every fetch the product performs is observable, and the realhuggingface_hubdownload code path is used unmodified.
Steps to reproduce
-
Build two data-only second-hop repositories that are identical except for two tensor values:
mbe2e-control/fastchat-baseandmbe2e-attacker/fastchat-base. Each contains onlyconfig.json,generation_config.json,model.safetensors,tokenizer.json,tokenizer_config.json,special_tokens_map.json— no.py, noauto_map, no pickle. The differences aremodel.embed_tokens.weight[0,0](0.25 vs 0.75, a measurement oracle) and one row oflm_head.weightthat makes the model's greedy first token spell out its own repository id (lm_head.weight64/7296 elements,model.embed_tokens.weight1/7296 elements). -
Produce one PEFT adapter with PEFT's own
save_pretrained()(rank 1,target_modules=["q_proj"], all-zero LoRA tensors) and copy it twice. The copies are byte-identical except one line ofadapter_config.json;adapter_model.safetensorshas the same SHA-256 in both (10b409b9f4d76f7011cb22740abeb1fe7f6511feb49aca713bf74f4a82d1aeb0):work/adapter_attack_peft/adapter_config.json:"base_model_name_or_path": "mbe2e-attacker/fastchat-base"work/adapter_benign_peft/adapter_config.json:"base_model_name_or_path": "mbe2e-control/fastchat-base"
-
Run the documented command on the malicious adapter and type one line ("Hello") at its
Human:prompt:python -m fastchat.serve.cli --model-path work\adapter_attack_peft --device cpu --temperature 0 --max-new-tokens 4 -
The CLI exits cleanly (code 0) and its answer comes out of the repository named in the adapter package — each second-hop repository was built to answer with its own id:
Human: Assistant: mbe2e-attacker/fastchat-base own mbe2e-attacker/fastchat-base own -
Every request the run makes goes to that repository, and only to it:
HEAD /mbe2e-attacker/fastchat-base/resolve/main/tokenizer_config.json GET /mbe2e-attacker/fastchat-base/resolve/main/tokenizer_config.json HEAD /mbe2e-attacker/fastchat-base/resolve/main/tokenizer.json GET /mbe2e-attacker/fastchat-base/resolve/main/tokenizer.json HEAD /mbe2e-attacker/fastchat-base/resolve/main/special_tokens_map.json GET /mbe2e-attacker/fastchat-base/resolve/main/special_tokens_map.json HEAD /mbe2e-attacker/fastchat-base/resolve/main/config.json GET /mbe2e-attacker/fastchat-base/resolve/main/config.json HEAD /mbe2e-attacker/fastchat-base/resolve/main/model.safetensors GET /mbe2e-attacker/fastchat-base/resolve/main/model.safetensors HEAD /mbe2e-attacker/fastchat-base/resolve/main/generation_config.json GET /mbe2e-attacker/fastchat-base/resolve/main/generation_config.json -
Second, independent reading of the same call: driving
fastchat.model.model_adapter.load_model()(the functionchat_loopuses) directly shows the loaded model is aPeftModelForCausalLMwhoseconfig._name_or_pathismbe2e-attacker/fastchat-baseand whose baseembed_tokens.weight[0,0]is 0.75 — the attacker repository's value. The benign adapter returns 0.25 /mbe2e-control/fastchat-base. -
Negative control — the benign adapter, byte-identical apart from that one JSON line, contacts only
mbe2e-control/fastchat-base, answers with the control repository's id, and never contacts the attacker repository.
Expected behavior
Which base weights a locally obtained adapter is merged into is the user's trust decision. A string read out of the adapter package should not silently choose a remote origin, and when the base model is not the one the user named, the product should not proceed silently.
Actual behavior
base_model_name_or_path is used as-is to select, download and load the second-hop repository. The substring test (if "peft" in base_model_path) only rejects base paths that literally contain peft; it says nothing about origin, revision, owner or integrity. The user sees no message naming the repository that was downloaded, and the CLI then serves the attacker's tokenizer and weights as if they were the adapter's intended base.
Impact
- Origin selection by model data: a crafted adapter package makes the victim's machine contact and download a model from a repository the attacker chose, at its current default revision (no pin unless the user passes
--revision). - Silent model substitution: the model that is loaded and served is the attacker's, while the adapter the user chose shows up as the source of the deployment. Prompts, tokenization and outputs are all handled by the substituted model.
- Data-only is enough: the two repositories used here contain no program code, so the effect does not depend on remote code. Note, however, that the loader passes
trust_remote_code=True(model_adapter.py:112, 122) for whatever repository the adapter names, so a second-hop repository that also ships anauto_map/.pyentry would have that code loaded as well — that variant was not exercised in this report (the repositories used here declare noauto_map, and no.pywas fetched). - Availability: because the attacker also chooses the model's size and shape, an oversized or malformed second-hop repository can exhaust memory or abort the run (not demonstrated here).
Suggested fix
- Do not treat
base_model_name_or_pathas an instruction to fetch an arbitrary origin. Require the base model to come from the user (CLI option / config), or validate it against an allow-list, and fail closed otherwise. At most, warn when the adapter's value differs from the user's choice. - When a non-default repository must be accepted, print the resolved repository and revision, require explicit confirmation, and pin a revision instead of defaulting to
main. - Record the resolved base repository and revision wherever the served model is reported (model worker log / API model card).
- Reconsider
trust_remote_code=Truefor a repository that was selected by a downloaded adapter package.
Activity