Skip to content

[BugFix][PD]: make example proxy usable with P2pNcclConnector - #26628

Merged
KuntaiDu merged 4 commits into
vllm-project:mainfrom
pandalee99:pr/dis_1
Nov 20, 2025
Merged

KuntaiDu merged 4 commits into
vllm-project:mainfrom
pandalee99:pr/dis_1

Conversation

@pandalee99

@pandalee99 pandalee99 commented Oct 11, 2025 •

Copy link
Copy Markdown
Contributor

Purpose

IN https://docs.vllm.ai/en/latest/examples/online_serving/disaggregated_prefill.html

  • When running the disaggregated prefill example with P2pNcclConnector (producer/consumer), the default static proxy in examples does not provide the backend KV routing information in a form that the connector can parse. This leads to runtime errors during KV save/load in the model execution path (for example ValueError or EngineCore failures when connector.parse_request_id cannot extract host:port).
  • In typical reproduction, observed failures include:
    • KeyError: 'http_port' (missing required kv_connector_extra_config fields)
    • TypeError (port type mismatches when concatenating strings/ints)
    • zmq.error.ZMQError: Address already in use (if multiple instances bind the same kv/zmq port)
    • ValueError: Request id ... does not contain hostname and port (parse_request_id expects host:port)
    • EngineCore exceptions when connector.save_kv_layer fails to parse request id

such as:

(EngineCore_DP0 pid=588697) INFO 10-09 00:54:01 [parallel_state.py:1165] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=588697) INFO 10-09 00:54:01 [factory.py:51] Creating v1 connector with name: P2pNcclConnector and engine_id: 10200040-2773-407d-b1cf-70a391b3a026
(EngineCore_DP0 pid=588697) WARNING 10-09 00:54:01 [base.py:84] Initializing KVConnectorBase_V1. This API is experimental and subject to change in the future as we iterate the design.
(EngineCore_DP0 pid=588697) INFO 10-09 00:54:01 [__init__.py:1433] Found nccl from library libnccl.so.2
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718] EngineCore failed to start.
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718] Traceback (most recent call last):
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 709, in run_engine_core
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 505, in __init__
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]     super().__init__(vllm_config, executor_class, log_stats,
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 82, in __init__
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 54, in __init__
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]     self._init_executor()
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 48, in _init_executor
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]     self.collective_rpc("init_device")
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]     answer = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3060, in run_method
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]     return func(*args, **kwargs)
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 611, in init_device
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]     ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 193, in init_device
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]     init_worker_distributed_environment(self.vllm_config, self.rank,
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 697, in init_worker_distributed_environment
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]     ensure_kv_transfer_initialized(vllm_config)
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_transfer_state.py", line 63, in ensure_kv_transfer_initialized
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]     _KV_CONNECTOR_AGENT = KVConnectorFactory.create_connector(
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_connector/factory.py", line 61, in create_connector
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]     return connector_cls(config, role)
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_connector.py", line 82, in __init__
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]     self.p2p_nccl_engine = P2pNcclEngine(
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]                            ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_engine.py", line 95, in __init__
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]     f"{self.config.kv_connector_extra_config['http_port']}")
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718]        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
(EngineCore_DP0 pid=588697) ERROR 10-09 00:54:01 [core.py:718] KeyError: 'http_port'
(EngineCore_DP0 pid=588697) Process EngineCore_DP0:
(EngineCore_DP0 pid=588697) Traceback (most recent call last):
(EngineCore_DP0 pid=588697)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=588697)     self.run()
(EngineCore_DP0 pid=588697)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=588697)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=588697)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 722, in run_engine_core
(EngineCore_DP0 pid=588697)     raise e
(EngineCore_DP0 pid=588697)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 709, in run_engine_core
(EngineCore_DP0 pid=588697)     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=588697)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588697)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 505, in __init__
(EngineCore_DP0 pid=588697)     super().__init__(vllm_config, executor_class, log_stats,
(EngineCore_DP0 pid=588697)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 82, in __init__
(EngineCore_DP0 pid=588697)     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=588697)                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588697)   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 54, in __init__
(EngineCore_DP0 pid=588697)     self._init_executor()
(EngineCore_DP0 pid=588697)   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 48, in _init_executor
(EngineCore_DP0 pid=588697)     self.collective_rpc("init_device")
(EngineCore_DP0 pid=588697)   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
(EngineCore_DP0 pid=588697)     answer = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_DP0 pid=588697)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588697)   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3060, in run_method
(EngineCore_DP0 pid=588697)     return func(*args, **kwargs)
(EngineCore_DP0 pid=588697)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588697)   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 611, in init_device
(EngineCore_DP0 pid=588697)     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=588697)     ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588697)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 193, in init_device
(EngineCore_DP0 pid=588697)     init_worker_distributed_environment(self.vllm_config, self.rank,
(EngineCore_DP0 pid=588697)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 697, in init_worker_distributed_environment
(EngineCore_DP0 pid=588697)     ensure_kv_transfer_initialized(vllm_config)
(EngineCore_DP0 pid=588697)   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_transfer_state.py", line 63, in ensure_kv_transfer_initialized
(EngineCore_DP0 pid=588697)     _KV_CONNECTOR_AGENT = KVConnectorFactory.create_connector(
(EngineCore_DP0 pid=588697)                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588697)   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_connector/factory.py", line 61, in create_connector
(EngineCore_DP0 pid=588697)     return connector_cls(config, role)
(EngineCore_DP0 pid=588697)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588697)   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_connector.py", line 82, in __init__
(EngineCore_DP0 pid=588697)     self.p2p_nccl_engine = P2pNcclEngine(
(EngineCore_DP0 pid=588697)                            ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588697)   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_engine.py", line 95, in __init__
(EngineCore_DP0 pid=588697)     f"{self.config.kv_connector_extra_config['http_port']}")
(EngineCore_DP0 pid=588697)        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
(EngineCore_DP0 pid=588697) KeyError: 'http_port'
(EngineCore_DP0 pid=588697) Exception ignored in: <function ExecutorBase.__del__ at 0x7fcabf6f9ee0>
(EngineCore_DP0 pid=588697) Traceback (most recent call last):
(EngineCore_DP0 pid=588697)   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 237, in __del__
(EngineCore_DP0 pid=588697)     self.shutdown()
(EngineCore_DP0 pid=588697)   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 76, in shutdown
(EngineCore_DP0 pid=588697)     worker.shutdown()
(EngineCore_DP0 pid=588697)   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 528, in shutdown
(EngineCore_DP0 pid=588697)     self.worker.shutdown()
(EngineCore_DP0 pid=588697)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 675, in shutdown
(EngineCore_DP0 pid=588697)     self.model_runner.ensure_kv_transfer_shutdown()
(EngineCore_DP0 pid=588697)     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588697) AttributeError: 'NoneType' object has no attribute 'ensure_kv_transfer_shutdown'
[W1009 00:54:01.704390813 ProcessGroupNCCL.cpp:981] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator())
[rank0]:[W1009 00:54:01.716788129 ProcessGroupGloo.cpp:514] Warning: Unable to resolve hostname to a (local) address. Using the loopback address as fallback. Manually set the network interface to bind to with GLOO_SOCKET_IFNAME. (function operator())
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
(EngineCore_DP0 pid=588703) INFO 10-09 00:54:01 [parallel_state.py:1165] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=588703) INFO 10-09 00:54:01 [factory.py:51] Creating v1 connector with name: P2pNcclConnector and engine_id: 836fb35a-7d2b-49ef-a596-36c50ed16ef0
(EngineCore_DP0 pid=588703) WARNING 10-09 00:54:01 [base.py:84] Initializing KVConnectorBase_V1. This API is experimental and subject to change in the future as we iterate the design.
(EngineCore_DP0 pid=588703) INFO 10-09 00:54:01 [__init__.py:1433] Found nccl from library libnccl.so.2
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718] EngineCore failed to start.
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718] Traceback (most recent call last):
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 709, in run_engine_core
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 505, in __init__
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]     super().__init__(vllm_config, executor_class, log_stats,
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 82, in __init__
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 54, in __init__
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]     self._init_executor()
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 48, in _init_executor
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]     self.collective_rpc("init_device")
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]     answer = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3060, in run_method
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]     return func(*args, **kwargs)
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 611, in init_device
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]     ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 193, in init_device
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]     init_worker_distributed_environment(self.vllm_config, self.rank,
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 697, in init_worker_distributed_environment
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]     ensure_kv_transfer_initialized(vllm_config)
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_transfer_state.py", line 63, in ensure_kv_transfer_initialized
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]     _KV_CONNECTOR_AGENT = KVConnectorFactory.create_connector(
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_connector/factory.py", line 61, in create_connector
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]     return connector_cls(config, role)
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_connector.py", line 82, in __init__
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]     self.p2p_nccl_engine = P2pNcclEngine(
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]                            ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_engine.py", line 95, in __init__
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]     f"{self.config.kv_connector_extra_config['http_port']}")
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718]        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
(EngineCore_DP0 pid=588703) ERROR 10-09 00:54:01 [core.py:718] KeyError: 'http_port'
(EngineCore_DP0 pid=588703) Process EngineCore_DP0:
(EngineCore_DP0 pid=588703) Traceback (most recent call last):
(EngineCore_DP0 pid=588703)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=588703)     self.run()
(EngineCore_DP0 pid=588703)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=588703)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=588703)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 722, in run_engine_core
(EngineCore_DP0 pid=588703)     raise e
(EngineCore_DP0 pid=588703)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 709, in run_engine_core
(EngineCore_DP0 pid=588703)     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=588703)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588703)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 505, in __init__
(EngineCore_DP0 pid=588703)     super().__init__(vllm_config, executor_class, log_stats,
(EngineCore_DP0 pid=588703)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 82, in __init__
(EngineCore_DP0 pid=588703)     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=588703)                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588703)   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 54, in __init__
(EngineCore_DP0 pid=588703)     self._init_executor()
(EngineCore_DP0 pid=588703)   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 48, in _init_executor
(EngineCore_DP0 pid=588703)     self.collective_rpc("init_device")
(EngineCore_DP0 pid=588703)   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
(EngineCore_DP0 pid=588703)     answer = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_DP0 pid=588703)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588703)   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3060, in run_method
(EngineCore_DP0 pid=588703)     return func(*args, **kwargs)
(EngineCore_DP0 pid=588703)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588703)   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 611, in init_device
(EngineCore_DP0 pid=588703)     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=588703)     ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588703)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 193, in init_device
(EngineCore_DP0 pid=588703)     init_worker_distributed_environment(self.vllm_config, self.rank,
(EngineCore_DP0 pid=588703)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 697, in init_worker_distributed_environment
(EngineCore_DP0 pid=588703)     ensure_kv_transfer_initialized(vllm_config)
(EngineCore_DP0 pid=588703)   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_transfer_state.py", line 63, in ensure_kv_transfer_initialized
(EngineCore_DP0 pid=588703)     _KV_CONNECTOR_AGENT = KVConnectorFactory.create_connector(
(EngineCore_DP0 pid=588703)                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588703)   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_connector/factory.py", line 61, in create_connector
(EngineCore_DP0 pid=588703)     return connector_cls(config, role)
(EngineCore_DP0 pid=588703)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588703)   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_connector.py", line 82, in __init__
(EngineCore_DP0 pid=588703)     self.p2p_nccl_engine = P2pNcclEngine(
(EngineCore_DP0 pid=588703)                            ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588703)   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_connector/v1/p2p/p2p_nccl_engine.py", line 95, in __init__
(EngineCore_DP0 pid=588703)     f"{self.config.kv_connector_extra_config['http_port']}")
(EngineCore_DP0 pid=588703)        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
(EngineCore_DP0 pid=588703) KeyError: 'http_port'
(EngineCore_DP0 pid=588703) Exception ignored in: <function ExecutorBase.__del__ at 0x7fbd212b9ee0>
(EngineCore_DP0 pid=588703) Traceback (most recent call last):
(EngineCore_DP0 pid=588703)   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 237, in __del__
(EngineCore_DP0 pid=588703)     self.shutdown()
(EngineCore_DP0 pid=588703)   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 76, in shutdown
(EngineCore_DP0 pid=588703)     worker.shutdown()
(EngineCore_DP0 pid=588703)   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 528, in shutdown
(EngineCore_DP0 pid=588703)     self.worker.shutdown()
(EngineCore_DP0 pid=588703)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 675, in shutdown
(EngineCore_DP0 pid=588703)     self.model_runner.ensure_kv_transfer_shutdown()
(EngineCore_DP0 pid=588703)     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=588703) AttributeError: 'NoneType' object has no attribute 'ensure_kv_transfer_shutdown'
[rank0]:[W1009 00:54:02.251169466 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
[rank0]:[W1009 00:54:02.274233780 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=588092) Traceback (most recent call last):
(APIServer pid=588092)   File "/usr/local/bin/vllm", line 10, in <module>
(APIServer pid=588092)     sys.exit(main())
(APIServer pid=588092)              ^^^^^^
(APIServer pid=588092)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 54, in main
(APIServer pid=588092)     args.dispatch_function(args)
(APIServer pid=588092)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 50, in cmd
(APIServer pid=588092)     uvloop.run(run_server(args))
(APIServer pid=588092)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 109, in run
(APIServer pid=588092)     return __asyncio.run(
(APIServer pid=588092)            ^^^^^^^^^^^^^^
(APIServer pid=588092)   File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=588092)     return runner.run(main)
(APIServer pid=588092)            ^^^^^^^^^^^^^^^^
(APIServer pid=588092)   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=588092)     return self._loop.run_until_complete(task)
(APIServer pid=588092)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=588092)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=588092)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 61, in wrapper
(APIServer pid=588092)     return await main
(APIServer pid=588092)            ^^^^^^^^^^
(APIServer pid=588092)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1941, in run_server
(APIServer pid=588092)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=588092)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1961, in run_server_worker
(APIServer pid=588092)     async with build_async_engine_client(
(APIServer pid=588092)                ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=588092)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=588092)     return await anext(self.gen)
(APIServer pid=588092)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=588092)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 179, in build_async_engine_client
(APIServer pid=588092)     async with build_async_engine_client_from_engine_args(
(APIServer pid=588092)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=588092)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=588092)     return await anext(self.gen)
(APIServer pid=588092)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=588092)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 221, in build_async_engine_client_from_engine_args
(APIServer pid=588092)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=588092)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=588092)   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 1589, in inner
(APIServer pid=588092)     return fn(*args, **kwargs)
(APIServer pid=588092)            ^^^^^^^^^^^^^^^^^^^
(APIServer pid=588092)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 212, in from_vllm_config
(APIServer pid=588092)     return cls(
(APIServer pid=588092)            ^^^^
(APIServer pid=588092)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 136, in __init__
(APIServer pid=588092)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=588092)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=588092)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 102, in make_async_mp_client
(APIServer pid=588092)     return AsyncMPClient(*client_args)
(APIServer pid=588092)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=588092)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 769, in __init__
(APIServer pid=588092)     super().__init__(
(APIServer pid=588092)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 448, in __init__
(APIServer pid=588092)     with launch_core_engines(vllm_config, executor_class,
(APIServer pid=588092)          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=588092)   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=588092)     next(self.gen)
(APIServer pid=588092)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 729, in launch_core_engines
(APIServer pid=588092)     wait_for_engine_startup(
(APIServer pid=588092)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 782, in wait_for_engine_startup
(APIServer pid=588092)     raise RuntimeError("Engine core initialization failed. "
(APIServer pid=588092) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(APIServer pid=588093) Traceback (most recent call last):
(APIServer pid=588093)   File "/usr/local/bin/vllm", line 10, in <module>
(APIServer pid=588093)     sys.exit(main())
(APIServer pid=588093)              ^^^^^^
(APIServer pid=588093)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 54, in main
(APIServer pid=588093)     args.dispatch_function(args)
(APIServer pid=588093)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 50, in cmd
(APIServer pid=588093)     uvloop.run(run_server(args))
(APIServer pid=588093)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 109, in run
(APIServer pid=588093)     return __asyncio.run(
(APIServer pid=588093)            ^^^^^^^^^^^^^^
(APIServer pid=588093)   File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=588093)     return runner.run(main)
(APIServer pid=588093)            ^^^^^^^^^^^^^^^^
(APIServer pid=588093)   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=588093)     return self._loop.run_until_complete(task)
(APIServer pid=588093)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=588093)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=588093)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 61, in wrapper
(APIServer pid=588093)     return await main
(APIServer pid=588093)            ^^^^^^^^^^
(APIServer pid=588093)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1941, in run_server
(APIServer pid=588093)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=588093)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1961, in run_server_worker
(APIServer pid=588093)     async with build_async_engine_client(
(APIServer pid=588093)                ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=588093)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=588093)     return await anext(self.gen)
(APIServer pid=588093)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=588093)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 179, in build_async_engine_client
(APIServer pid=588093)     async with build_async_engine_client_from_engine_args(
(APIServer pid=588093)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=588093)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=588093)     return await anext(self.gen)
(APIServer pid=588093)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=588093)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 221, in build_async_engine_client_from_engine_args
(APIServer pid=588093)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=588093)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=588093)   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 1589, in inner
(APIServer pid=588093)     return fn(*args, **kwargs)
(APIServer pid=588093)            ^^^^^^^^^^^^^^^^^^^
(APIServer pid=588093)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 212, in from_vllm_config
(APIServer pid=588093)     return cls(
(APIServer pid=588093)            ^^^^
(APIServer pid=588093)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 136, in __init__
(APIServer pid=588093)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=588093)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=588093)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 102, in make_async_mp_client
(APIServer pid=588093)     return AsyncMPClient(*client_args)
(APIServer pid=588093)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=588093)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 769, in __init__
(APIServer pid=588093)     super().__init__(
(APIServer pid=588093)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 448, in __init__
(APIServer pid=588093)     with launch_core_engines(vllm_config, executor_class,
(APIServer pid=588093)          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=588093)   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=588093)     next(self.gen)
(APIServer pid=588093)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 729, in launch_core_engines
(APIServer pid=588093)     wait_for_engine_startup(
(APIServer pid=588093)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 782, in wait_for_engine_startup
(APIServer pid=588093)     raise RuntimeError("Engine core initialization failed. "
(APIServer pid=588093) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}


Root cause

  • The P2pNcclConnector routing relies on being able to parse host:port (or equivalent routing info) from the request id. The example proxy only provided target info via non-standard headers or via the HTTP URL, but it did not embed the ZMQ/KV addresses into the request id in a format the connector expects. As a result, backends fail to route KV messages and exceptions are raised.
  • Additional contributors: missing or type-mismatched kv config fields in examples (causing KeyError/TypeError), and example scripts binding conflicting kv ports.

Fix

  1. The example proxy now constructs and injects a request_id that contains KV host:port entries for both prefill and decode, e.g.:
    • ___prefill_addr_<prefill_kv_addr>___decode_addr_<decode_kv_addr>_<uuid>
    • The proxy sends this request_id to backends via X-Request-Id. This format is compatible with P2pNcclConnector.parse_request_id.
  2. The proxy runs a minimal prefill→decode flow:
    • prefill: send a request with max_tokens=1 (or max_completion_tokens=1) and wait for it to finish, with the injected request_id.
    • decode: send the original request to the decode backend with the same request_id and stream the response back to the client.
  3. The proxy also sets X-KV-Target to the decode host:port as a compatibility header.

notes

  • Example proxy (benchmarks / examples path):
    • Build request_id containing PREFILL_KV_ADDR and DECODE_KV_ADDR.
    • Add X-Request-Id header to both prefill and decode requests.
    • prefill uses max_tokens=1 and awaits completion; decode uses the original payload and streams response back.
    • Keep X-KV-Target for compatibility.
  • No changes to core vLLM runtime behavior are made in this PR; only the example proxy and documentation are updated.

Test Plan

examples/online_serving/disaggregated_prefill.sh

Test Result

+ echo ''

+ echo 'Output of first request: {"id":"cmpl-___prefill_addr_localhost:14579___decode_addr_localhost:14580_d106fa79d2454c209214e119c307eddc","object":"text_completion","created":1760177542,"model":"/vllm-workspace/tmp2/Qwen3-8B","choices":[{"index":0,"text":" city in the state of California, located on the","logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null,"prompt_logprobs":null,"prompt_token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":4,"total_tokens":14,"completion_tokens":10,"prompt_tokens_details":null},"kv_transfer_params":null}'
Output of first request: {"id":"cmpl-___prefill_addr_localhost:14579___decode_addr_localhost:14580_d106fa79d2454c209214e119c307eddc","object":"text_completion","created":1760177542,"model":"/vllm-workspace/tmp2/Qwen3-8B","choices":[{"index":0,"text":" city in the state of California, located on the","logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null,"prompt_logprobs":null,"prompt_token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":4,"total_tokens":14,"completion_tokens":10,"prompt_tokens_details":null},"kv_transfer_params":null}
+ echo 'Output of second request: {"id":"cmpl-___prefill_addr_localhost:14579___decode_addr_localhost:14580_7e3eaab9969047fe96be5ecd6dbc382c","object":"text_completion","created":1760177542,"model":"/vllm-workspace/tmp2/Qwen3-8B","choices":[{"index":0,"text":" city in Santa Clara County, California, in the","logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null,"prompt_logprobs":null,"prompt_token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":4,"total_tokens":14,"completion_tokens":10,"prompt_tokens_details":null},"kv_transfer_params":null}'
Output of second request: {"id":"cmpl-___prefill_addr_localhost:14579___decode_addr_localhost:14580_7e3eaab9969047fe96be5ecd6dbc382c","object":"text_completion","created":1760177542,"model":"/vllm-workspace/tmp2/Qwen3-8B","choices":[{"index":0,"text":" city in Santa Clara County, California, in the","logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null,"prompt_logprobs":null,"prompt_token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":4,"total_tokens":14,"completion_tokens":10,"prompt_tokens_details":null},"kv_transfer_params":null}
+ echo '🎉🎉 Successfully finished 2 test requests! 🎉🎉'
🎉🎉 Successfully finished 2 test requests! 🎉🎉
+ echo ''

(APIServer pid=662000) INFO:     Shutting down
(APIServer pid=662000) INFO:     Waiting for application shutdown.
(APIServer pid=662000) INFO:     Application shutdown complete.
(APIServer pid=662001) INFO:     Shutting down
(APIServer pid=662001) INFO 10-11 03:12:31 [loggers.py:123] Engine 000: Avg prompt throughput: 0.7 tokens/s, Avg generation throughput: 1.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=662001) INFO:     Waiting for application shutdown.
(APIServer pid=662001) INFO:     Application shutdown complete.


Essential Elements of an Effective PR Description Checklist

relate :https://docs.vllm.ai/en/latest/examples/online_serving/disaggregated_prefill.html

  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Signed-off-by: PAN <1162953505@qq.com>
Signed-off-by: PAN <1162953505@qq.com>
@github-actions

Copy link
Copy Markdown

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

@mergify

mergify Bot commented Oct 11, 2025

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--26628.org.readthedocs.build/en/26628/

@mergify mergify Bot added documentation Improvements or additions to documentation performance Performance-related issues kv-connector labels Oct 11, 2025
Signed-off-by: PAN <1162953505@qq.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request fixes runtime errors in the disaggregated prefill example when using P2pNcclConnector. The changes correctly format the request_id to include KV routing information, which is essential for the connector. The example proxy server logic is updated to perform the prefill and decode steps, and the example shell script is updated with the necessary configurations.

My review focuses on the proxy server implementation. I've identified a critical issue regarding incorrect and inefficient aiohttp session management that will cause runtime errors and performance degradation. I've provided suggestions to fix this by using a single, shared ClientSession for the application's lifetime. The rest of the changes look good and effectively address the bug described.

@chaunceyjiang chaunceyjiang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

examples/online_serving/disaggregated_serving_p2p_nccl_xpyd/disagg_proxy_p2p_nccl_xpyd.py

As far as I know, P2pNcclConnector should use this proxy.

@pandalee99

Copy link
Copy Markdown
Contributor Author

examples/online_serving/disaggregated_serving_p2p_nccl_xpyd/disagg_proxy_p2p_nccl_xpyd.py

As far as I know, P2pNcclConnector should use this proxy.

https://docs.vllm.ai/en/latest/examples/online_serving/disaggregated_prefill.html

But in this example, P2pNcclConnector will use
python3 .../../benchmarks/disagg_benchmarks/disagg_prefill_proxy_server.py

# prefilling instance, which is the KV producer
CUDA_VISIBLE_DEVICES=0 vllm serve $MODEL_NAME \
    --port 8100 \
    --max-model-len 100 \
    --gpu-memory-utilization 0.8 \
    --trust-remote-code \
    --kv-transfer-config \
    '{"kv_connector":"P2pNcclConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2}' &

# decoding instance, which is the KV consumer
CUDA_VISIBLE_DEVICES=1 vllm serve $MODEL_NAME \
    --port 8200 \
    --max-model-len 100 \
    --gpu-memory-utilization 0.8 \
    --trust-remote-code \
    --kv-transfer-config \
    '{"kv_connector":"P2pNcclConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2}' &

# wait until prefill and decode instances are ready
wait_for_server 8100
wait_for_server 8200

python3 ../../benchmarks/disagg_benchmarks/disagg_prefill_proxy_server.py &
sleep 1

@pandalee99

Copy link
Copy Markdown
Contributor Author

@chaunceyjiang Hey, my friend. Actually, you just need to run the original script to identify this issue, and running the current script will fix this error. Should we further merge this PR, or do you have other plans?

@youkaichao

Copy link
Copy Markdown
Member

cc @KuntaiDu

@KuntaiDu

KuntaiDu commented Nov 4, 2025

Copy link
Copy Markdown
Contributor

Thanks @youkaichao for reminder.

We should probably update the example so that it uses
examples/online_serving/disaggregated_serving_p2p_nccl_xpyd/disagg_proxy_p2p_nccl_xpyd.py. @Abatom (the original author of this feature) what's your suggestion on this?

The code change of current PR is a bit large, but if @Abatom is too busy to reply we can merge this PR to fix examples first.

@Abatom

Abatom commented Nov 4, 2025

Copy link
Copy Markdown
Contributor

@KuntaiDu I’ll take a look later—thanks!

@Abatom

Abatom commented Nov 8, 2025

Copy link
Copy Markdown
Contributor

@KuntaiDu @pandalee99 LGTM!

@KuntaiDu

Copy link
Copy Markdown
Contributor

@Abatom thank you for reviewing! Could you please approve the PR on your side? I can then approve on my side and merge.

@Abatom

Abatom commented Nov 19, 2025

Copy link
Copy Markdown
Contributor

@KuntaiDu Approve!

@KuntaiDu KuntaiDu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@KuntaiDu
KuntaiDu enabled auto-merge (squash) November 20, 2025 16:12
@github-actions github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 20, 2025
@KuntaiDu
KuntaiDu merged commit e5bfcb6 into vllm-project:main Nov 20, 2025
12 checks passed
RunkaiTao pushed a commit to RunkaiTao/vllm that referenced this pull request Nov 24, 2025
…roject#26628)

Signed-off-by: PAN <1162953505@qq.com>
Signed-off-by: Runkai Tao <rt572@physics.rutgers.edu>
devpatelio pushed a commit to SumanthRH/vllm that referenced this pull request Nov 29, 2025
kitaekatt pushed a commit to kitaekatt/vllm that referenced this pull request Dec 1, 2025
mystous pushed a commit to mystous/vllm_hybrid that referenced this pull request May 10, 2026
my-other-github-account pushed a commit to my-other-github-account/vllm that referenced this pull request May 15, 2026
0826joyce pushed a commit to 0826joyce/vllm-serving-optimization that referenced this pull request May 19, 2026
plasticchris pushed a commit to plasticchris/vllm that referenced this pull request Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation kv-connector performance Performance-related issues ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants