Skip to content

Wait for resources to become available between actor pool stages - #2357

Merged
ayushdg merged 8 commits into
NVIDIA-NeMo:mainfrom
ayushdg:actorpool-stage-wait
Sep 3, 2026
Merged

ayushdg merged 8 commits into
NVIDIA-NeMo:mainfrom
ayushdg:actorpool-stage-wait

Conversation

@ayushdg

@ayushdg ayushdg commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

Description

Ray actor pool executor goes stage by stage kills all actors and the end of the stage and restarts new set of actors for the next stage.

The executor gauges how many actors it can start by looking at cluster.available_resources.
But given that ray.kill is async and it may take time for the available resources to update, the default sleep of 0.2s may not be enough for all machines to free up existing processes.
This may lead to fewer resources being spun up for future stages or no resources available at all leading to errors.

The new proposed solution adds the following:

  1. interval -> Poll resources every interval seconds
  2. timeout -> If we do not get all intended resources by timeout s the stage starts with whatever is available.

Here's the new process:

  1. Calculate the available resources since pipeline execution - Update with a higher value if more become available later.
  2. Calculate the number of actors a stage could have with these max seen resources.
    a. If a stage now has enough resources to get these actors, start the stage.
    b. If a stage doesn't have enough resources to get these actors, sleep for interval seconds and recheck.
    c. If a stage doesn't have enough resources to get these actors even after timeout raise an error

Usage

# Add snippet demonstrating usage

Checklist

  • I am familiar with the Contributing Guide.
  • New or Existing tests cover these changes.
  • The documentation is up to date with these changes.

Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com>
Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com>
Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 31, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@ayushdg

ayushdg commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 0f04448

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

This branch was previously deployed

3 inactive deployments
public — 761d750e Deployed Sep 3, 2026 by copy-pr-bot[bot] via release / finalize / notify #962
nemo-ci — 761d750e Deployed Sep 3, 2026 by copy-pr-bot[bot] via L0_Unit_Test_GPU-audio #5406
test — 761d750e Deployed Sep 3, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #5406
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants