Repository navigation
Adds support for custom ProgressReporter to Ray integration - #3269
Conversation
|
@garylvov, would love to get your feedback if you have time! |
|
Hi thanks for the PR! (EDIT: didn't mean to mention the issue got this confused with early stopping) |
There was a problem hiding this comment.
Greptile Overview
Greptile Summary
This PR adds optional custom ProgressReporter support to Isaac Lab's Ray hyperparameter tuning integration. Users can now specify a custom ProgressReporter class via the --progress_reporter CLI argument in tuner.py, similar to how they specify --cfg_class for job configurations. The PR includes CustomCartpoleProgressReporter in vision_cartpole_cfg.py as a reference implementation showing how to filter metrics and adjust reporting frequency. When a custom reporter is provided, the implementation disables Ray's context-aware progress reporting (via RAY_AIR_NEW_OUTPUT=0 environment variable) to avoid conflicts. The change integrates with the existing dynamic module loading pattern already used for job configurations, maintaining consistency with the codebase's architecture. If no custom reporter is specified, the system falls back to Ray's default CLIReporter.
Important Files Changed
| Filename | Score | Overview |
|---|---|---|
| scripts/reinforcement_learning/ray/tuner.py | 3/5 | Adds --progress_reporter CLI argument, dynamic loading logic, validation checks, and environment variable handling for custom reporters |
| scripts/reinforcement_learning/ray/hyperparameter_tuning/vision_cartpole_cfg.py | 5/5 | Adds example CustomCartpoleProgressReporter class demonstrating custom metric selection and reporting frequency |
| docs/source/features/ray.rst | 5/5 | Updates documentation line emphasis range from 18-54 to 18-59 to reflect tuner.py changes |
Confidence score: 3/5
- This PR introduces new optional functionality with moderate complexity and has potential edge cases in the validation logic
- Score reflects incomplete documentation (PR checklist shows documentation changes not completed), missing error handling when specified progress_reporter class doesn't exist in module (lines 496-502), and potential isinstance check issues at line 498 that may fail if users pass pre-instantiated reporter objects instead of classes
- Pay close attention to scripts/reinforcement_learning/ray/tuner.py lines 496-502 (missing existence validation) and lines 498-500 (isinstance logic may not handle all reporter input types correctly)
Sequence Diagram
sequenceDiagram
participant User
participant tuner.py
participant argparse
participant importlib
participant Module
participant invoke_tuning_run
participant Ray
participant ProgressReporter
User->>tuner.py: Execute with --progress_reporter flag
tuner.py->>argparse: Parse command-line arguments
argparse-->>tuner.py: Return parsed args (including progress_reporter name)
tuner.py->>importlib: Load cfg_file module
importlib->>Module: spec_from_file_location()
Module-->>tuner.py: Return module with config classes
tuner.py->>Module: Get cfg_class instance
Module-->>tuner.py: Return config instance
alt Custom ProgressReporter provided
tuner.py->>Module: Get progress_reporter class by name
Module-->>tuner.py: Return ProgressReporter class
tuner.py->>ProgressReporter: Instantiate custom reporter
ProgressReporter-->>tuner.py: Return reporter instance
tuner.py->>tuner.py: Set RAY_AIR_NEW_OUTPUT=0
else No custom reporter
tuner.py->>tuner.py: progress_reporter = None
end
tuner.py->>invoke_tuning_run: Call with cfg, args, progress_reporter
invoke_tuning_run->>Ray: Initialize Ray with RunConfig
invoke_tuning_run->>Ray: Configure Tuner with progress_reporter
Ray->>ProgressReporter: Report progress during tuning
Ray->>invoke_tuning_run: Execute tuning jobs
invoke_tuning_run-->>User: Complete tuning run
3 files reviewed, 3 comments
There was a problem hiding this comment.
Greptile Overview
Greptile Summary
This PR adds support for custom ProgressReporter classes to the Ray hyperparameter tuning integration, allowing users to customize metrics display and reporting frequency beyond the default CLIReporter.
Key changes:
- Added
--progress_reporterCLI argument totuner.pyto accept custom reporter class names - Modified
invoke_tuning_run()to accept and pass through a customProgressReporterinstance - Disables Ray's context-aware progress reporting when a custom reporter is provided
- Includes validation to prevent users from setting metric/mode directly in the reporter class
- Added
CustomCartpoleProgressReporterexample invision_cartpole_cfg.py
Issues found:
- Logic error in instance validation (line 498) - will incorrectly reject valid
ProgressReporterinstances - Missing error handling (line 496) - silently ignores invalid reporter names instead of raising an error
Confidence Score: 3/5
- This PR has logic errors in validation code that will cause runtime failures
- Score reflects two critical logic errors: (1) the isinstance check will reject valid ProgressReporter instances passed by users, causing unexpected TypeErrors, and (2) missing error handling allows invalid reporter names to silently fail. Both issues will lead to confusing runtime behavior. The feature itself is well-designed and the example implementation is correct, but the validation logic needs fixes before merge.
- scripts/reinforcement_learning/ray/tuner.py requires fixes to instance validation and error handling logic
Important Files Changed
File Analysis
| Filename | Score | Overview |
|---|---|---|
| scripts/reinforcement_learning/ray/tuner.py | 3/5 | Adds custom ProgressReporter support with logic issues in validation and error handling |
| scripts/reinforcement_learning/ray/hyperparameter_tuning/vision_cartpole_cfg.py | 5/5 | Added example CustomCartpoleProgressReporter class with proper configuration |
| docs/source/features/ray.rst | 5/5 | Updated documentation highlighting for new code section |
Sequence Diagram
sequenceDiagram
participant User
participant tuner.py
participant Module
participant Ray
User->>tuner.py: Run with --progress_reporter CustomCartpoleProgressReporter
tuner.py->>Module: Load cfg_file and import module
tuner.py->>Module: hasattr(module, "CustomCartpoleProgressReporter")
Module-->>tuner.py: True
tuner.py->>Module: getattr(module, "CustomCartpoleProgressReporter")
Module-->>tuner.py: CustomCartpoleProgressReporter class
tuner.py->>tuner.py: Validate: isinstance(type) and issubclass(ProgressReporter)
tuner.py->>tuner.py: Instantiate: progress_reporter = CustomCartpoleProgressReporter()
tuner.py->>tuner.py: invoke_tuning_run(cfg, args, progress_reporter)
tuner.py->>tuner.py: Check: getattr(_metric/_mode)
tuner.py->>tuner.py: Set RAY_AIR_NEW_OUTPUT=0
tuner.py->>Ray: Create RunConfig with progress_reporter
Ray->>Ray: Execute tuning with custom reporter
Ray-->>User: Display custom metrics
3 files reviewed, 2 comments
There was a problem hiding this comment.
Greptile Overview
Greptile Summary
This PR adds support for custom ProgressReporter classes in Ray hyperparameter tuning workflows. Users can now pass --progress_reporter CustomReporterClassName to specify a custom reporter defined in their config file, allowing better control over which metrics are displayed and how frequently.
Key changes:
- Added
--progress_reporterCLI argument totuner.py - Modified
invoke_tuning_run()to accept optionalprogress_reporterparameter - When custom reporter is provided, disables Ray's context-aware progress reporting (
RAY_AIR_NEW_OUTPUT=0) to avoid conflicts - Added validation to prevent users from setting
_metricor_modedirectly in reporter class (enforces using CLI args instead) - Included
CustomCartpoleProgressReporterexample invision_cartpole_cfg.pyshowing custom metric columns and frequency
Issues found:
- Error handling for missing reporter class is incomplete - silently falls back to default instead of raising error
- Instance vs class detection logic doesn't handle pre-instantiated reporters
Confidence Score: 3/5
- Safe to merge after fixing error handling for missing reporter classes
- Feature adds useful functionality and is well-designed, but has two logic bugs in error handling: (1) silently falls back to default when reporter class doesn't exist instead of raising error, and (2) doesn't handle pre-instantiated reporter instances. Both issues could cause user confusion but won't break existing functionality.
- Pay attention to
scripts/reinforcement_learning/ray/tuner.pylines 496-501 for error handling improvements
Important Files Changed
File Analysis
| Filename | Score | Overview |
|---|---|---|
| scripts/reinforcement_learning/ray/tuner.py | 3/5 | Adds custom ProgressReporter support with conditional validation and initialization logic; error handling for missing reporter needs improvement |
| scripts/reinforcement_learning/ray/hyperparameter_tuning/vision_cartpole_cfg.py | 5/5 | Adds example CustomCartpoleProgressReporter implementation; clean and straightforward |
Sequence Diagram
sequenceDiagram
participant User
participant tuner.py
participant ConfigModule
participant Ray
User->>tuner.py: Execute with --progress_reporter arg
tuner.py->>ConfigModule: Load module from --cfg_file
tuner.py->>ConfigModule: Instantiate --cfg_class
alt progress_reporter provided
tuner.py->>ConfigModule: getattr(module, progress_reporter)
alt is class
tuner.py->>tuner.py: Instantiate progress_reporter()
else is instance
Note over tuner.py: ERROR: Not handled properly
end
tuner.py->>tuner.py: Set RAY_AIR_NEW_OUTPUT=0
tuner.py->>tuner.py: Validate _metric/_mode not set
end
tuner.py->>Ray: invoke_tuning_run(cfg, args, progress_reporter)
Ray->>Ray: Create RunConfig with progress_reporter
Ray->>Ray: Execute tuning with custom reporter
Ray-->>User: Display custom metrics
3 files reviewed, 2 comments
There was a problem hiding this comment.
Greptile Overview
Greptile Summary
This PR adds support for custom ProgressReporter classes in Ray hyperparameter tuning. Users can now specify a custom reporter via the --progress_reporter CLI argument that references a class defined in their config file.
Key changes:
- Added
--progress_reporterargument totuner.pythat accepts a class name from the config module - Modified
invoke_tuning_run()to accept an optionalprogress_reporterparameter and pass it toRunConfig - Disables Ray's context-aware progress reporting (
RAY_AIR_NEW_OUTPUT=0) when a custom reporter is provided - Validates that custom reporters don't set
_metricor_modeattributes (should come from CLI args instead) - Includes example implementation
CustomCartpoleProgressReporterinvision_cartpole_cfg.py
Issues found:
- Error handling has gaps: silently falls back to default when
--progress_reporteris specified but class doesn't exist in module - Type checking logic doesn't handle case where user passes an already-instantiated reporter object
- Validation relies on checking private
_metricand_modeattributes which may not be consistent across all ProgressReporter subclasses
Confidence Score: 3/5
- This PR is functional but has error handling issues that could confuse users
- The core feature works correctly and the example implementation is clean. However, there are two logic errors in error handling: (1) silently falling back to default when the specified class doesn't exist makes debugging difficult, and (2) not handling instance vs class distinction could cause TypeErrors. Additionally, relying on private attributes for validation reduces robustness. These issues won't cause crashes in the happy path but will create confusing behavior when users make mistakes.
- Pay close attention to
scripts/reinforcement_learning/ray/tuner.pylines 496-502 for error handling improvements
Important Files Changed
File Analysis
| Filename | Score | Overview |
|---|---|---|
| scripts/reinforcement_learning/ray/tuner.py | 3/5 | adds custom ProgressReporter support with argument parsing and instantiation logic; has issues with error handling when class not found and instance vs class detection |
| scripts/reinforcement_learning/ray/hyperparameter_tuning/vision_cartpole_cfg.py | 5/5 | adds example CustomCartpoleProgressReporter class extending CLIReporter with custom metrics; implementation is clean and straightforward |
Sequence Diagram
sequenceDiagram
participant User
participant CLI as tuner.py
participant Module as Config Module
participant Ray as Ray Tune
User->>CLI: Run with --progress_reporter flag
CLI->>CLI: Parse arguments
CLI->>Module: Import config file
CLI->>Module: Load config class
alt progress_reporter specified
CLI->>Module: getattr(module, progress_reporter)
alt class exists
Module-->>CLI: Return ProgressReporter class
CLI->>CLI: Check if subclass of tune.ProgressReporter
CLI->>CLI: Instantiate class
CLI->>CLI: Disable RAY_AIR_NEW_OUTPUT
CLI->>CLI: Validate _metric and _mode not set
else class not found
CLI->>CLI: Silently use default (None)
end
end
CLI->>Ray: invoke_tuning_run(cfg, args, progress_reporter)
Ray->>Ray: Create RunConfig with progress_reporter
Ray->>Ray: Execute hyperparameter tuning
Ray-->>User: Display progress with custom reporter
3 files reviewed, 3 comments
Greptile OverviewGreptile SummaryThis PR adds support for custom
Issues found:
Confidence Score: 3/5
Important Files ChangedFile Analysis
Sequence DiagramsequenceDiagram
participant User
participant CLI as tuner.py CLI
participant Module as Config Module
participant Tuner as invoke_tuning_run
participant Ray as Ray Tune
User->>CLI: Run with --progress_reporter CustomReporter
CLI->>Module: Load config file
CLI->>Module: getattr(module, "CustomReporter")
alt Class not found
Module-->>CLI: hasattr returns False
CLI->>CLI: progress_reporter = None (silent fallback)
else Class found
Module-->>CLI: Return CustomReporter class
CLI->>CLI: Validate is subclass of ProgressReporter
CLI->>CLI: Instantiate reporter
end
CLI->>Tuner: invoke_tuning_run(cfg, args, progress_reporter)
alt Custom reporter provided
Tuner->>Tuner: Set RAY_AIR_NEW_OUTPUT=0
Tuner->>Tuner: Validate no _metric/_mode attributes
end
Tuner->>Ray: Create RunConfig with progress_reporter
Ray->>Ray: Start tuning with custom reporter
|
|
@kellyguo11 pinging on small diff PR that only touches Ray stuff that's been open for some while, can we please merge? Thank you!!! |
…sim#3269) # Description This PR adds support for providing a custom `ProgressReporter` while doing hyperparameter tuning with Ray Integration. Without the PR, the Ray integration defaults to the standard `CLIReporter`, which often displays metrics that aren’t particularly relevant or at the desired frequency. Similar to how we allow users to specify a --cfg_class (e.g., `CartpoleTheiaJobCfg`), this PR lets them optionally provide a custom `ProgressReporter` class. If such is not provided, it falls back to the default. Moreover, I have added an example inside `vision_cartpole_cfg.py` (i.e., `CustomCartpoleProgressReporter`). One point to highlight is that the new "[context-aware progress reporting](ray-project/ray#36949)" conflicts with custom `ProgressReporter`, so if a custom `ProgressReporter` is provided, the PR disables the context-aware progress reporting. Fixes isaac-sim#3268. ## Type of change - New feature (non-breaking change which adds functionality) ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Co-authored-by: garylvov <67614381+garylvov@users.noreply.github.com>
…sim#3269) # Description This PR adds support for providing a custom `ProgressReporter` while doing hyperparameter tuning with Ray Integration. Without the PR, the Ray integration defaults to the standard `CLIReporter`, which often displays metrics that aren’t particularly relevant or at the desired frequency. Similar to how we allow users to specify a --cfg_class (e.g., `CartpoleTheiaJobCfg`), this PR lets them optionally provide a custom `ProgressReporter` class. If such is not provided, it falls back to the default. Moreover, I have added an example inside `vision_cartpole_cfg.py` (i.e., `CustomCartpoleProgressReporter`). One point to highlight is that the new "[context-aware progress reporting](ray-project/ray#36949)" conflicts with custom `ProgressReporter`, so if a custom `ProgressReporter` is provided, the PR disables the context-aware progress reporting. Fixes isaac-sim#3268. ## Type of change - New feature (non-breaking change which adds functionality) ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Co-authored-by: garylvov <67614381+garylvov@users.noreply.github.com>
…sim#3269) # Description This PR adds support for providing a custom `ProgressReporter` while doing hyperparameter tuning with Ray Integration. Without the PR, the Ray integration defaults to the standard `CLIReporter`, which often displays metrics that aren’t particularly relevant or at the desired frequency. Similar to how we allow users to specify a --cfg_class (e.g., `CartpoleTheiaJobCfg`), this PR lets them optionally provide a custom `ProgressReporter` class. If such is not provided, it falls back to the default. Moreover, I have added an example inside `vision_cartpole_cfg.py` (i.e., `CustomCartpoleProgressReporter`). One point to highlight is that the new "[context-aware progress reporting](ray-project/ray#36949)" conflicts with custom `ProgressReporter`, so if a custom `ProgressReporter` is provided, the PR disables the context-aware progress reporting. Fixes isaac-sim#3268. ## Type of change - New feature (non-breaking change which adds functionality) ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Co-authored-by: garylvov <67614381+garylvov@users.noreply.github.com>
Description
This PR adds support for providing a custom
ProgressReporterwhile doing hyperparameter tuning with Ray Integration.Without the PR, the Ray integration defaults to the standard
CLIReporter, which often displays metrics that aren’t particularly relevant or at the desired frequency. Similar to how we allow users to specify a --cfg_class (e.g.,CartpoleTheiaJobCfg), this PR lets them optionally provide a customProgressReporterclass. If such is not provided, it falls back to the default.Moreover, I have added an example inside
vision_cartpole_cfg.py(i.e.,CustomCartpoleProgressReporter).One point to highlight is that the new "context-aware progress reporting" conflicts with custom
ProgressReporter, so if a customProgressReporteris provided, the PR disables the context-aware progress reporting.Fixes #3268.
Type of change
Checklist
pre-commitchecks with./isaaclab.sh --formatconfig/extension.tomlfileCONTRIBUTORS.mdor my name already exists there