Skip to content

First-class the use case of multiple pods per node in a specific ComputeDomain #309

Open

Description

@jgehrcke

Currently, it is not immediately obvious for how to create a single-node ComputeDomain and launch multiple pods in it.

That's a valid use case for multiple reasons. It is also technically sound: a shared "node-local" IMEX channel works, "same-node MNNVL" is a viable scenario.

We currently support this use case by manually managing the underlying ResourceClaim. A working example is here: https://github.com/jgehrcke/jpsnips-nv/tree/1a462d07b5ba22e78ae32156bf6c730bd94a133c/dra/single-node

(for output, see below)

In the future, this scenario should be easier to implement and not be conceptually different from a ComputeDomain spread across multiple nodes. Ideally, we can follow through with the philosophy of "ComputeDomain follows workload placement" also when the workload is comprised of multiple pods on the same node.

Output:

$ bash 1-node-cd.sh 
+ kubectl get resourceclaim
NAME                                    STATE                AGE
cd1node-compute-domain-shared-channel   allocated,reserved   7m42s
repro2-compute-domain-shared-channel    pending              2d1h
+ kubectl get computedomains.resource.nvidia.com
NAME                     AGE
cd1node-compute-domain   7m42s
+ kubectl delete -f 1-node-cd.yaml
job.batch "cd1node" deleted
+ kubectl delete resourceclaim/cd1node-compute-domain-shared-channel
resourceclaim.resource.k8s.io "cd1node-compute-domain-shared-channel" deleted
+ kubectl delete computedomains.resource.nvidia.com cd1node-compute-domain
computedomain.resource.nvidia.com "cd1node-compute-domain" deleted
+ set +x
computedomain.resource.nvidia.com/cd1node-compute-domain created
CDUID: 10dfde78-ffb2-434c-aea8-79fa6461c5a4
resourceclaim.resource.k8s.io/cd1node-compute-domain-shared-channel created
job.batch/cd1node created
+ sleep 5
+ kubectl wait --for=condition=Ready pods -l batch.kubernetes.io/job-completion-index=0,job-name=cd1node
pod/cd1node-0-7slzl condition met
+ kubectl wait --for=condition=Ready pods -l batch.kubernetes.io/job-completion-index=1,job-name=cd1node --timeout=40s
pod/cd1node-1-2w87z condition met
+ set +x


pods on nodes:
NAME              READY   STATUS    RESTARTS   AGE   IP              NODE                  NOMINATED NODE   READINESS GATES
cd1node-0-7slzl   1/1     Running   0          30s   192.168.81.15   gb-nvl-043-bianca-7   <none>           <none>
cd1node-1-2w87z   1/1     Running   0          30s   192.168.81.6    gb-nvl-043-bianca-7   <none>           <none>
DAEMON_POD: cd1node-compute-domain-bz99b-dnzc9


IMEX daemon status:
READY
Connectivity Table Legend:
I - Invalid - Node wasn't reachable, no connection status available
N - Never Connected
R - Recovering - Connection was lost, but clean up has not yet been triggered.
D - Disconnected - Connection was lost, and clean up has been triggreed.
A - Authenticating - If GSSAPI enabled, client has initiated mutual authentication.
!V! - Version mismatch, communication disabled.
!M! - Node map mismatch, communication disabled.
C - Connected - Ready for operation

3/28/2025 19:45:16.602
Nodes:
Node #0   - 10.115.131.12   - READY                - Version: 570.124.06

 Nodes From\To  0  
       0        C  
Domain State: UP
READY stopAtReady: 0
keepGoing: 1
Finishing subscription
READY


leader log tail:
[pod/cd1node-0-7slzl/cd1node] total 0
[pod/cd1node-0-7slzl/cd1node] drwxr-xr-x 2 root root     60 Mar 28 19:45 .
[pod/cd1node-0-7slzl/cd1node] drwxr-xr-x 6 root root    480 Mar 28 19:45 ..
[pod/cd1node-0-7slzl/cd1node] crw-rw-rw- 1 root root 234, 0 Mar 28 19:45 channel0


follower log tail:
[pod/cd1node-1-2w87z/cd1node] total 0
[pod/cd1node-1-2w87z/cd1node] drwxr-xr-x 2 root root     60 Mar 28 19:45 .
[pod/cd1node-1-2w87z/cd1node] drwxr-xr-x 6 root root    480 Mar 28 19:45 ..
[pod/cd1node-1-2w87z/cd1node] crw-rw-rw- 1 root root 234, 0 Mar 28 19:45 channel0


IMEX daemon log:
[Mar 28 2025 19:45:04] [INFO] [tid 39] nvidia-imex persistence file /var/run/nvidia-imex/persist.dat does not exist.  Assuming no previous importers.
[Mar 28 2025 19:45:04] [INFO] [tid 39] NvGpu Library version matched with GPU Driver version
[Mar 28 2025 19:45:04] [INFO] [tid 90] Started processing of incoming messages.
[Mar 28 2025 19:45:04] [INFO] [tid 91] Started processing of incoming messages.
[Mar 28 2025 19:45:04] [INFO] [tid 92] Started processing of incoming messages.
[Mar 28 2025 19:45:04] [INFO] [tid 93] Started processing of incoming messages.
[Mar 28 2025 19:45:04] [INFO] [tid 39] Creating gRPC channels to all peers (nPeers = 1).
[Mar 28 2025 19:45:04] [INFO] [tid 39] IMEX_WAIT_FOR_QUORUM != FULL, continuing initialization without waiting for connections to all nodes.
[Mar 28 2025 19:45:04] [INFO] [tid 94] Connection established to node 0 with ip address 10.115.131.12. Number of times connected: 1
[Mar 28 2025 19:45:04] [INFO] [tid 39] GPU event successfully subscribed

Activity

  1. guptaNswati commented on Apr 23, 2025

    @guptaNswati
    Contributor

    So basically have the 2 pods reference to the same resoureclaim, which does not happen in the automatic flow. How do we hack creating multiple channels on a single node? This has been a frequent ask from internal users.

  2. guptaNswati commented on Apr 23, 2025

    @guptaNswati
    Contributor

    BTW this is a good single node IMEX channel test cudaMallocAsync -t ipc_mempools_basic

  3. added
    featureissue/PR that proposes a new feature or functionality
    on May 8, 2025
  4. modified the milestones: v25.12.0, unscheduled on Aug 13, 2025
  5. modified the milestones: v25.8.1, v25.12.0 on Oct 8, 2025
  6. modified the milestones: v25.12.0, v26.4.0 on Dec 11, 2025
  7. modified the milestones: v26.4.0, Backlog on Mar 5, 2026
  8. k8s-triage-robot commented on Jun 3, 2026

    @k8s-triage-robot

    The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

    This bot triages un-triaged issues according to the following rules:

    • After 90d of inactivity, lifecycle/stale is applied
    • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
    • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

    You can:

    • Mark this issue as fresh with /remove-lifecycle stale
    • Close this issue with /close
    • Offer to help out with Issue Triage

    Please send feedback to sig-contributor-experience at kubernetes/community.

    /lifecycle stale

  9. added
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Jun 3, 2026
  10. jgehrcke commented on Jun 26, 2026

    @jgehrcke
    ContributorAuthor

    Still relevant. Quoting myself from #934 to maybe explain the value a little better:

    we want a clean (downside-free) way to associate more than one pod per ComputeDomain on the same node; this greatly simplifies building distributed applications. We track that in #309. That allows for each pod to grab just a single GPU; i.e. application code does not need to manage multiple local GPUs -- that removes one hierarchy level from the overall architecture (significant reduction in complexity).

  11. jgehrcke commented on Jun 26, 2026

    @jgehrcke
    ContributorAuthor

    /remove-lifecycle stale

  12. removed
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Jun 26, 2026
  13. github-actions commented on Sep 25, 2026

    @github-actions

    This issue has been automatically marked as stale because it has not had activity in the last 90 days. It will be closed in 30 days if no further activity occurs. To keep this issue open, add a comment or remove the lifecycle/stale label. To exempt it permanently, apply the lifecycle/frozen label.

  14. added
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Sep 25, 2026
  15. jgehrcke commented on Oct 19, 2026

    @jgehrcke
    ContributorAuthor

    /remove-lifecycle stale

  16. removed
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

featureissue/PR that proposes a new feature or functionality

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions