Skip to content

Adds support for custom ProgressReporter to Ray integration - #3269

Merged
kellyguo11 merged 15 commits into
isaac-sim:mainfrom
ozhanozen:feat/ray-progress-reporter
Nov 25, 2025
Merged

kellyguo11 merged 15 commits into
isaac-sim:mainfrom
ozhanozen:feat/ray-progress-reporter

Conversation

@ozhanozen

Copy link
Copy Markdown
Contributor

Description

This PR adds support for providing a custom ProgressReporter while doing hyperparameter tuning with Ray Integration.

Without the PR, the Ray integration defaults to the standard CLIReporter, which often displays metrics that aren’t particularly relevant or at the desired frequency. Similar to how we allow users to specify a --cfg_class (e.g., CartpoleTheiaJobCfg), this PR lets them optionally provide a custom ProgressReporter class. If such is not provided, it falls back to the default.

Moreover, I have added an example inside vision_cartpole_cfg.py (i.e., CustomCartpoleProgressReporter).

One point to highlight is that the new "context-aware progress reporting" conflicts with custom ProgressReporter, so if a custom ProgressReporter is provided, the PR disables the context-aware progress reporting.

Fixes #3268.

Type of change

  • New feature (non-breaking change which adds functionality)

Checklist

  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the changelog and the corresponding version in the extension's config/extension.toml file
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

@ozhanozen
ozhanozen requested a review from ooctipus as a code owner August 25, 2025 15:56
@ozhanozen

Copy link
Copy Markdown
Contributor Author

@garylvov, would love to get your feedback if you have time!

@garylvov

garylvov commented Aug 26, 2025 •

Copy link
Copy Markdown
Collaborator

Hi thanks for the PR! (EDIT: didn't mean to mention the issue got this confused with early stopping)

@garylvov garylvov left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions Bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team labels Sep 16, 2025

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Greptile Overview

Greptile Summary

This PR adds optional custom ProgressReporter support to Isaac Lab's Ray hyperparameter tuning integration. Users can now specify a custom ProgressReporter class via the --progress_reporter CLI argument in tuner.py, similar to how they specify --cfg_class for job configurations. The PR includes CustomCartpoleProgressReporter in vision_cartpole_cfg.py as a reference implementation showing how to filter metrics and adjust reporting frequency. When a custom reporter is provided, the implementation disables Ray's context-aware progress reporting (via RAY_AIR_NEW_OUTPUT=0 environment variable) to avoid conflicts. The change integrates with the existing dynamic module loading pattern already used for job configurations, maintaining consistency with the codebase's architecture. If no custom reporter is specified, the system falls back to Ray's default CLIReporter.

Important Files Changed

Filename Score Overview
scripts/reinforcement_learning/ray/tuner.py 3/5 Adds --progress_reporter CLI argument, dynamic loading logic, validation checks, and environment variable handling for custom reporters
scripts/reinforcement_learning/ray/hyperparameter_tuning/vision_cartpole_cfg.py 5/5 Adds example CustomCartpoleProgressReporter class demonstrating custom metric selection and reporting frequency
docs/source/features/ray.rst 5/5 Updates documentation line emphasis range from 18-54 to 18-59 to reflect tuner.py changes

Confidence score: 3/5

  • This PR introduces new optional functionality with moderate complexity and has potential edge cases in the validation logic
  • Score reflects incomplete documentation (PR checklist shows documentation changes not completed), missing error handling when specified progress_reporter class doesn't exist in module (lines 496-502), and potential isinstance check issues at line 498 that may fail if users pass pre-instantiated reporter objects instead of classes
  • Pay close attention to scripts/reinforcement_learning/ray/tuner.py lines 496-502 (missing existence validation) and lines 498-500 (isinstance logic may not handle all reporter input types correctly)

Sequence Diagram

sequenceDiagram
    participant User
    participant tuner.py
    participant argparse
    participant importlib
    participant Module
    participant invoke_tuning_run
    participant Ray
    participant ProgressReporter

    User->>tuner.py: Execute with --progress_reporter flag
    tuner.py->>argparse: Parse command-line arguments
    argparse-->>tuner.py: Return parsed args (including progress_reporter name)
    tuner.py->>importlib: Load cfg_file module
    importlib->>Module: spec_from_file_location()
    Module-->>tuner.py: Return module with config classes
    tuner.py->>Module: Get cfg_class instance
    Module-->>tuner.py: Return config instance
    alt Custom ProgressReporter provided
        tuner.py->>Module: Get progress_reporter class by name
        Module-->>tuner.py: Return ProgressReporter class
        tuner.py->>ProgressReporter: Instantiate custom reporter
        ProgressReporter-->>tuner.py: Return reporter instance
        tuner.py->>tuner.py: Set RAY_AIR_NEW_OUTPUT=0
    else No custom reporter
        tuner.py->>tuner.py: progress_reporter = None
    end
    tuner.py->>invoke_tuning_run: Call with cfg, args, progress_reporter
    invoke_tuning_run->>Ray: Initialize Ray with RunConfig
    invoke_tuning_run->>Ray: Configure Tuner with progress_reporter
    Ray->>ProgressReporter: Report progress during tuning
    Ray->>invoke_tuning_run: Execute tuning jobs
    invoke_tuning_run-->>User: Complete tuning run
Loading

3 files reviewed, 3 comments

Edit Code Review Agent Settings | Greptile

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Greptile Overview

Greptile Summary

This PR adds support for custom ProgressReporter classes to the Ray hyperparameter tuning integration, allowing users to customize metrics display and reporting frequency beyond the default CLIReporter.

Key changes:

  • Added --progress_reporter CLI argument to tuner.py to accept custom reporter class names
  • Modified invoke_tuning_run() to accept and pass through a custom ProgressReporter instance
  • Disables Ray's context-aware progress reporting when a custom reporter is provided
  • Includes validation to prevent users from setting metric/mode directly in the reporter class
  • Added CustomCartpoleProgressReporter example in vision_cartpole_cfg.py

Issues found:

  • Logic error in instance validation (line 498) - will incorrectly reject valid ProgressReporter instances
  • Missing error handling (line 496) - silently ignores invalid reporter names instead of raising an error

Confidence Score: 3/5

  • This PR has logic errors in validation code that will cause runtime failures
  • Score reflects two critical logic errors: (1) the isinstance check will reject valid ProgressReporter instances passed by users, causing unexpected TypeErrors, and (2) missing error handling allows invalid reporter names to silently fail. Both issues will lead to confusing runtime behavior. The feature itself is well-designed and the example implementation is correct, but the validation logic needs fixes before merge.
  • scripts/reinforcement_learning/ray/tuner.py requires fixes to instance validation and error handling logic

Important Files Changed

File Analysis

Filename Score Overview
scripts/reinforcement_learning/ray/tuner.py 3/5 Adds custom ProgressReporter support with logic issues in validation and error handling
scripts/reinforcement_learning/ray/hyperparameter_tuning/vision_cartpole_cfg.py 5/5 Added example CustomCartpoleProgressReporter class with proper configuration
docs/source/features/ray.rst 5/5 Updated documentation highlighting for new code section

Sequence Diagram

sequenceDiagram
    participant User
    participant tuner.py
    participant Module
    participant Ray
    
    User->>tuner.py: Run with --progress_reporter CustomCartpoleProgressReporter
    tuner.py->>Module: Load cfg_file and import module
    tuner.py->>Module: hasattr(module, "CustomCartpoleProgressReporter")
    Module-->>tuner.py: True
    tuner.py->>Module: getattr(module, "CustomCartpoleProgressReporter")
    Module-->>tuner.py: CustomCartpoleProgressReporter class
    tuner.py->>tuner.py: Validate: isinstance(type) and issubclass(ProgressReporter)
    tuner.py->>tuner.py: Instantiate: progress_reporter = CustomCartpoleProgressReporter()
    tuner.py->>tuner.py: invoke_tuning_run(cfg, args, progress_reporter)
    tuner.py->>tuner.py: Check: getattr(_metric/_mode)
    tuner.py->>tuner.py: Set RAY_AIR_NEW_OUTPUT=0
    tuner.py->>Ray: Create RunConfig with progress_reporter
    Ray->>Ray: Execute tuning with custom reporter
    Ray-->>User: Display custom metrics
Loading

3 files reviewed, 2 comments

Edit Code Review Agent Settings | Greptile

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Greptile Overview

Greptile Summary

This PR adds support for custom ProgressReporter classes in Ray hyperparameter tuning workflows. Users can now pass --progress_reporter CustomReporterClassName to specify a custom reporter defined in their config file, allowing better control over which metrics are displayed and how frequently.

Key changes:

  • Added --progress_reporter CLI argument to tuner.py
  • Modified invoke_tuning_run() to accept optional progress_reporter parameter
  • When custom reporter is provided, disables Ray's context-aware progress reporting (RAY_AIR_NEW_OUTPUT=0) to avoid conflicts
  • Added validation to prevent users from setting _metric or _mode directly in reporter class (enforces using CLI args instead)
  • Included CustomCartpoleProgressReporter example in vision_cartpole_cfg.py showing custom metric columns and frequency

Issues found:

  • Error handling for missing reporter class is incomplete - silently falls back to default instead of raising error
  • Instance vs class detection logic doesn't handle pre-instantiated reporters

Confidence Score: 3/5

  • Safe to merge after fixing error handling for missing reporter classes
  • Feature adds useful functionality and is well-designed, but has two logic bugs in error handling: (1) silently falls back to default when reporter class doesn't exist instead of raising error, and (2) doesn't handle pre-instantiated reporter instances. Both issues could cause user confusion but won't break existing functionality.
  • Pay attention to scripts/reinforcement_learning/ray/tuner.py lines 496-501 for error handling improvements

Important Files Changed

File Analysis

Filename Score Overview
scripts/reinforcement_learning/ray/tuner.py 3/5 Adds custom ProgressReporter support with conditional validation and initialization logic; error handling for missing reporter needs improvement
scripts/reinforcement_learning/ray/hyperparameter_tuning/vision_cartpole_cfg.py 5/5 Adds example CustomCartpoleProgressReporter implementation; clean and straightforward

Sequence Diagram

sequenceDiagram
    participant User
    participant tuner.py
    participant ConfigModule
    participant Ray
    
    User->>tuner.py: Execute with --progress_reporter arg
    tuner.py->>ConfigModule: Load module from --cfg_file
    tuner.py->>ConfigModule: Instantiate --cfg_class
    
    alt progress_reporter provided
        tuner.py->>ConfigModule: getattr(module, progress_reporter)
        alt is class
            tuner.py->>tuner.py: Instantiate progress_reporter()
        else is instance
            Note over tuner.py: ERROR: Not handled properly
        end
        tuner.py->>tuner.py: Set RAY_AIR_NEW_OUTPUT=0
        tuner.py->>tuner.py: Validate _metric/_mode not set
    end
    
    tuner.py->>Ray: invoke_tuning_run(cfg, args, progress_reporter)
    Ray->>Ray: Create RunConfig with progress_reporter
    Ray->>Ray: Execute tuning with custom reporter
    Ray-->>User: Display custom metrics
Loading

3 files reviewed, 2 comments

Edit Code Review Agent Settings | Greptile

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Greptile Overview

Greptile Summary

This PR adds support for custom ProgressReporter classes in Ray hyperparameter tuning. Users can now specify a custom reporter via the --progress_reporter CLI argument that references a class defined in their config file.

Key changes:

  • Added --progress_reporter argument to tuner.py that accepts a class name from the config module
  • Modified invoke_tuning_run() to accept an optional progress_reporter parameter and pass it to RunConfig
  • Disables Ray's context-aware progress reporting (RAY_AIR_NEW_OUTPUT=0) when a custom reporter is provided
  • Validates that custom reporters don't set _metric or _mode attributes (should come from CLI args instead)
  • Includes example implementation CustomCartpoleProgressReporter in vision_cartpole_cfg.py

Issues found:

  • Error handling has gaps: silently falls back to default when --progress_reporter is specified but class doesn't exist in module
  • Type checking logic doesn't handle case where user passes an already-instantiated reporter object
  • Validation relies on checking private _metric and _mode attributes which may not be consistent across all ProgressReporter subclasses

Confidence Score: 3/5

  • This PR is functional but has error handling issues that could confuse users
  • The core feature works correctly and the example implementation is clean. However, there are two logic errors in error handling: (1) silently falling back to default when the specified class doesn't exist makes debugging difficult, and (2) not handling instance vs class distinction could cause TypeErrors. Additionally, relying on private attributes for validation reduces robustness. These issues won't cause crashes in the happy path but will create confusing behavior when users make mistakes.
  • Pay close attention to scripts/reinforcement_learning/ray/tuner.py lines 496-502 for error handling improvements

Important Files Changed

File Analysis

Filename Score Overview
scripts/reinforcement_learning/ray/tuner.py 3/5 adds custom ProgressReporter support with argument parsing and instantiation logic; has issues with error handling when class not found and instance vs class detection
scripts/reinforcement_learning/ray/hyperparameter_tuning/vision_cartpole_cfg.py 5/5 adds example CustomCartpoleProgressReporter class extending CLIReporter with custom metrics; implementation is clean and straightforward

Sequence Diagram

sequenceDiagram
    participant User
    participant CLI as tuner.py
    participant Module as Config Module
    participant Ray as Ray Tune
    
    User->>CLI: Run with --progress_reporter flag
    CLI->>CLI: Parse arguments
    CLI->>Module: Import config file
    CLI->>Module: Load config class
    
    alt progress_reporter specified
        CLI->>Module: getattr(module, progress_reporter)
        alt class exists
            Module-->>CLI: Return ProgressReporter class
            CLI->>CLI: Check if subclass of tune.ProgressReporter
            CLI->>CLI: Instantiate class
            CLI->>CLI: Disable RAY_AIR_NEW_OUTPUT
            CLI->>CLI: Validate _metric and _mode not set
        else class not found
            CLI->>CLI: Silently use default (None)
        end
    end
    
    CLI->>Ray: invoke_tuning_run(cfg, args, progress_reporter)
    Ray->>Ray: Create RunConfig with progress_reporter
    Ray->>Ray: Execute hyperparameter tuning
    Ray-->>User: Display progress with custom reporter
Loading

3 files reviewed, 3 comments

Edit Code Review Agent Settings | Greptile

@greptile-apps

greptile-apps Bot commented Nov 11, 2025

Copy link
Copy Markdown
Contributor

Greptile Overview

Greptile Summary

This PR adds support for custom ProgressReporter classes in Ray hyperparameter tuning, allowing users to customize metrics display and reporting frequency. The implementation:

  • Adds optional --progress_reporter CLI argument that accepts a class name from the config file
  • Disables Ray's context-aware progress reporting when custom reporter is used (via RAY_AIR_NEW_OUTPUT=0)
  • Validates that custom reporters don't set _metric or _mode attributes directly
  • Provides example implementation (CustomCartpoleProgressReporter) showing typical usage

Issues found:

  • Line 544: If user specifies a non-existent progress reporter class, code silently falls back to default instead of raising an error
  • Line 546-549: Doesn't handle case where user passes an already-instantiated reporter object (will incorrectly raise TypeError)

Confidence Score: 3/5

  • This PR has logical errors in error handling that will cause confusing behavior for users
  • Score reflects two critical logic bugs: (1) silently ignoring invalid reporter class names instead of raising errors, leading to confusing silent fallback behavior, and (2) incorrectly rejecting already-instantiated reporter objects. These will cause user-facing issues but don't affect core functionality when valid class names are provided.
  • Pay close attention to scripts/reinforcement_learning/ray/tuner.py lines 544-549 for the error handling logic

Important Files Changed

File Analysis

Filename Score Overview
scripts/reinforcement_learning/ray/tuner.py 3/5 Adds custom ProgressReporter support with validation logic, but has error handling gaps when class not found and doesn't handle instance inputs
scripts/reinforcement_learning/ray/hyperparameter_tuning/vision_cartpole_cfg.py 5/5 Adds example CustomCartpoleProgressReporter class demonstrating the feature - implementation looks correct

Sequence Diagram

sequenceDiagram
    participant User
    participant CLI as tuner.py CLI
    participant Module as Config Module
    participant Tuner as invoke_tuning_run
    participant Ray as Ray Tune

    User->>CLI: Run with --progress_reporter CustomReporter
    CLI->>Module: Load config file
    CLI->>Module: getattr(module, "CustomReporter")
    alt Class not found
        Module-->>CLI: hasattr returns False
        CLI->>CLI: progress_reporter = None (silent fallback)
    else Class found
        Module-->>CLI: Return CustomReporter class
        CLI->>CLI: Validate is subclass of ProgressReporter
        CLI->>CLI: Instantiate reporter
    end
    CLI->>Tuner: invoke_tuning_run(cfg, args, progress_reporter)
    alt Custom reporter provided
        Tuner->>Tuner: Set RAY_AIR_NEW_OUTPUT=0
        Tuner->>Tuner: Validate no _metric/_mode attributes
    end
    Tuner->>Ray: Create RunConfig with progress_reporter
    Ray->>Ray: Start tuning with custom reporter
Loading

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 files reviewed, 2 comments

Edit Code Review Agent Settings | Greptile

@garylvov

garylvov commented Nov 25, 2025 •

Copy link
Copy Markdown
Collaborator

@kellyguo11 pinging on small diff PR that only touches Ray stuff that's been open for some while, can we please merge? Thank you!!!

@kellyguo11
kellyguo11 merged commit 4049aa8 into isaac-sim:main Nov 25, 2025
Edify0991 pushed a commit to Edify0991/IsaacLab that referenced this pull request Jan 14, 2026
…sim#3269)

# Description

This PR adds support for providing a custom `ProgressReporter` while
doing hyperparameter tuning with Ray Integration.

Without the PR, the Ray integration defaults to the standard
`CLIReporter`, which often displays metrics that aren’t particularly
relevant or at the desired frequency. Similar to how we allow users to
specify a --cfg_class (e.g., `CartpoleTheiaJobCfg`), this PR lets them
optionally provide a custom `ProgressReporter` class. If such is not
provided, it falls back to the default.

Moreover, I have added an example inside `vision_cartpole_cfg.py` (i.e.,
`CustomCartpoleProgressReporter`).

One point to highlight is that the new "[context-aware progress
reporting](ray-project/ray#36949)" conflicts
with custom `ProgressReporter`, so if a custom `ProgressReporter` is
provided, the PR disables the context-aware progress reporting.

Fixes isaac-sim#3268.

## Type of change

- New feature (non-breaking change which adds functionality)

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

---------

Co-authored-by: garylvov <67614381+garylvov@users.noreply.github.com>
nitesh-subedi pushed a commit to nitesh-subedi/IsaacLab that referenced this pull request Feb 5, 2026
…sim#3269)

# Description

This PR adds support for providing a custom `ProgressReporter` while
doing hyperparameter tuning with Ray Integration.

Without the PR, the Ray integration defaults to the standard
`CLIReporter`, which often displays metrics that aren’t particularly
relevant or at the desired frequency. Similar to how we allow users to
specify a --cfg_class (e.g., `CartpoleTheiaJobCfg`), this PR lets them
optionally provide a custom `ProgressReporter` class. If such is not
provided, it falls back to the default.

Moreover, I have added an example inside `vision_cartpole_cfg.py` (i.e.,
`CustomCartpoleProgressReporter`).

One point to highlight is that the new "[context-aware progress
reporting](ray-project/ray#36949)" conflicts
with custom `ProgressReporter`, so if a custom `ProgressReporter` is
provided, the PR disables the context-aware progress reporting.

Fixes isaac-sim#3268.

## Type of change

- New feature (non-breaking change which adds functionality)

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

---------

Co-authored-by: garylvov <67614381+garylvov@users.noreply.github.com>
csj-275 pushed a commit to csj-275/IsaacLab that referenced this pull request Jul 8, 2026
…sim#3269)

# Description

This PR adds support for providing a custom `ProgressReporter` while
doing hyperparameter tuning with Ray Integration.

Without the PR, the Ray integration defaults to the standard
`CLIReporter`, which often displays metrics that aren’t particularly
relevant or at the desired frequency. Similar to how we allow users to
specify a --cfg_class (e.g., `CartpoleTheiaJobCfg`), this PR lets them
optionally provide a custom `ProgressReporter` class. If such is not
provided, it falls back to the default.

Moreover, I have added an example inside `vision_cartpole_cfg.py` (i.e.,
`CustomCartpoleProgressReporter`).

One point to highlight is that the new "[context-aware progress
reporting](ray-project/ray#36949)" conflicts
with custom `ProgressReporter`, so if a custom `ProgressReporter` is
provided, the PR disables the context-aware progress reporting.

Fixes isaac-sim#3268.

## Type of change

- New feature (non-breaking change which adds functionality)

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

---------

Co-authored-by: garylvov <67614381+garylvov@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Proposal] Custom ProgressReporter for Ray integration

4 participants