Repository navigation
Releases
v5.0.0
Release Artifacts
The links below identify the Holoscan SDK 5.0 installation channels. Check individual channels for published artifacts.
-
ð Docker container: tag
v5.0.0-cuda13 -
ð Python wheels:
pip install holoscan==5.0.0Python wheel packages
All packages below are version 5.0.0. Native component wheels target CPython 3.12+ (
abi3) on Linux (manylinux_2_39) with CUDA 13. Universal wheels are metapackages that select the corresponding native components.Package Contents holoscan Full SDK metapackage holoscan-cu13 CUDA 13 SDK metapackage holoscan-core Core metapackage holoscan-core-cu13 Core native component (CUDA 13) holoscan-vision Vision metapackage holoscan-vision-cu13 Vision native component (CUDA 13) holoscan-visualization Visualization metapackage holoscan-visualization-cu13 Visualization native component (CUDA 13) holoscan-inference Inference metapackage holoscan-inference-cu13 Inference native component (CUDA 13) For more details, see Python Bindings.
-
ðĶïļ Debian packages
-
ð Documentation
See SDK Installation prerequisites for supported platforms and system requirements.
Release Notes
New Features and Improvements
Important
Holoscan SDK 5.0 is a ground-up rewrite of the Holoscan SDK runtime and API.
Learn more about the Holoscan platform at holoscan.org.
API stability
The Stable source API core covers basic in-process graph/operator authoring, ordinary built-in scalar ports, existing explicitly named and unambiguous Python port declarations/lookup, and structured error handling in C++ and Python. These documented source contracts remain compatible across 5.x updates; deprecation and removal follow the major-release policy described in API stability. The Stable entry-point call contracts do not freeze omitted-contract trigger resolution, clock selection/admission, or automatic queue/resource derivation; these policies remain Beta. Tensor declarations, ownership, output specifications, and pools remain Beta. Plan inspection, resolved-defaults reports, typed records, and report formats remain Prototype. Existing schema version counters and compatibility epochs are retained across GA and follow their own evolution rules. Source API stability does not promise binary compatibility between SDK packages or cross-version interoperability for channels, schemas, or serialized artifacts. Follow the release-specific rebuild and deployment requirements when upgrading.
Programming model
- Callback-scoped scalar borrowing (C++ and Python) -
Input<T>::selected_sample()exposes immutable selected data and metadata without a payload copy, heap allocation, or owning read claim.sensor_io::ImageDescriptorScratchprepares bounded metadata storage during startup for Image transforms; output pixels still use a separate Tensor loan. Hosted Python Operators expose the same frozen scalar selection asOperatorPort.selected_sample(): metadata remains independently inspectable, while.datalazily materializes an owning Python projection. C++ views, references, and the Python selection facade must not escape the compute callback. Owning receive APIs remain available. - Prepared generated-schema transforms (C++) - Generated ordinary and descriptor-Tensor schemas can opt into bounded reusable native metadata with
NATIVE_BOUNDS_FILE.schema::SchemaScratch<T>andschema::DescriptorMetadataScratch<T>retain cold-prepared capacity, whileOutput<T>::emit_ref()synchronously publishes an ordinary generated schema without a by-value native-object copy. Values outside authored native bounds fail withErrorCode::kResourceExhausted; altering prepared owners or retaining a nested owner across reuse fails closed withErrorCode::kPreparedStorageInvalid. The bounds policy is a build and deployment contract rather than wire identity, so every producer and prepared consumer for a schema must use the same bounds file. Owning APIs remain available alongside prepared storage. holoscan::GraphAPI - Graph definition without anApplicationwrapper.compile()produces an immutableExecutionPlan;run_async()returns aRunSessionfor deterministic asynchronous execution and clean shutdown.- Typed Operators and ports - Operators declare strongly typed input and output ports; payloads flow as
Sample<T>with causal metadata. - Reusable Graph embedding - A compiled subgraph can be embedded into a parent graph and reused across multiple compositions.
- CUDA Graph regions - Applications can explicitly fuse an eligible fixed-shape, linear GPU region with
RegionConstraint::cuda_graph(). Runtime captures member CUDA work once after lifecycle startup and replays it on the source's planned stream while retaining ordinary Tensor readiness and publication behavior. The ordinary execution path remains available for comparison. - Automatic graph partitioning and inspection - Graphs without partition declarations use the
holoscan.partition.source_chain/1planner, which forms linear execution chains across the complete DAG, including fan-out, fan-in, diamond, and parallel edge topologies. Different partitions may execute callbacks concurrently. Applications can author exact manual anchors and request an automatic remainder, usepartition(name).include_all()for one complete composed partition, and inspect the accepted cover throughExecutionPlan::partitions(). Green Context bindings require exact named anchors; dedicated contexts are not assigned automatically. - CUDA streams follow execution partitions - By default, each active execution-partition/device pair receives one Runtime-owned CUDA compute stream, allowing independent branches to submit GPU work independently while Tensor readiness and release ordering remain automatic. Applications can select
CudaComputeStreamPolicy::kSharedProcessDeviceinCompileOptionswhen measurements favor one ordinary stream per process group and device. Separate streams permit but do not guarantee concurrent CUDA execution. - Support-thread placement (C++) - Applications can request CPU affinity for the run-session controller and, when present, the process-local Channel pump. The compiled plan records the admitted placement. A pump binding does not create a pump when the selected transport does not need one.
- CUDA Green Context manual placement (developer preview) - Applications can query a qualified CUDA 13 target, bind an exact manually authored execution partition to a driver-structured set of streaming multiprocessors, and inspect the accepted placement and diagnostics in the compiled plan. The compiler and Runtime validate the target, materialize the Green Context streams before operator execution, and preserve Tensor ordering across ordinary and Green Context streams. Automatic graph partitioning does not automatically size or assign dedicated contexts. See the CUDA Green Contexts guide.
- Lifecycle stage graph - Bounded operator lifecycle with deterministic startup, running, and shutdown stages.
- Managed shutdown signals - On Linux, blocking
run(plan)synchronously accepts SIGINT and SIGTERM through an HSDK signal domain by default; the first accepted occurrence requests cooperative stop.RunSignalPolicy::kCallerManagedprovides a blocking opt-out, while earlyManagedShutdownSignalsplusrun_async(plan, signals)provides managed asynchronous execution. Plainrun_async(plan)remains caller-managed. A second accepted occurrence delegates the actual signal under the unchanged prior policy; first-signal shutdown remains cooperative, with no forced cancellation or whole-run completion bound. A clean first accepted SIGINT returns control after cleanup with a nonzero blocking result (currently1). Managed blocking runs require an adoptable signal domain or a single live task during setup; otherwise they fail before session creation. EstablishManagedShutdownSignalsbefore creating participant threads when managed asynchronous execution is needed. An adopted domain has one binding attempt; close it after retirement and create a fresh domain for another managed run. A child created while the domain is active inherits blocked SIGINT/SIGTERM, andexec()preserves that mask. Reset the child mask withsigprocmask()beforeexec(), or usePOSIX_SPAWN_SETSIGMASK;system()andpopen()do not reset it automatically.
Temporal scheduling
- Temporal contracts -
OnEach,LatestWithin, andSkipcombinators control per-input selection policy; each operator declares aContractthat is verified at compile time. - Maximum start delay - C++
Contract::max_start_delay(...)and Py...
v4.6.0
Release Artifacts
- ð Docker container: tag
v4.6.0-cuda13,v4.6.0-cuda12-dgpuandv4.6.0-cuda12-igpu - ð Python wheel:
pip install holoscan==4.6.0 - ðĶïļ Debian packages:
4.6.0.0-1 - ð Documentation
See supported platforms for compatibility.
Release Notes
New Features and Improvements
Source build & Release Artifacts
- Updated the Holoscan Sensor Bridge (HoloLink) dependency to version 2.7.0. This release adds precompiled
hololink_modulebinaries to the Holoscan SDK development container, allowing applications to adapt to different HSB devices and HSB-IP versions without recompiling. For full details, refer to the Holoscan Sensor Bridge release notes and user guide. - Adjusted the bundled
fmtintegration to avoid NVCC diagnostic 2417-D in C++17 source builds.
Bug fixes
| Issue | Description |
|---|---|
| 6385530, 6403160 | Hardened PoseTree UCX deserialization to reject malformed frame names and invalid or truncated snapshots before mutating state. |
| 6385365, 6385375, 6385708, 6385727, 6385828 | Hardened distributed-application and Holoviz codec deserialization against invalid sizes, counts, incomplete payloads, and failed-value access. |
| 6397322 | Rejected impossible AppWorker port-allocation requests before reserving memory and added validation for requested counts and port ranges. |
| 6384975, 6385172, 6385241, 6385242, 6385277 | Hardened Python GXF recording deserialization against truncated headers, inconsistent tensor metadata, invalid payload ranges, and malformed timestamps. |
| HoloInfer | Fixed a multi-context initialization race that could fail with Current Manager not initialized while preserving the existing public lifecycle API. |
| Holoviz | Fixed Vulkan device-creation failure handling to report the returned VkResult instead of continuing with an invalid device and subsequently crashing. |
Known Issues
| Issue | Description |
|---|---|
| 6059987, 6154550 | CUDA 13.2 forward compatibility is not available for some R580.00 drivers (IGX-SW 2.0) and the "cuda-compat-13-2" package is intentionally removed from the Holoscan SDK CUDA 13 build and development containers. Developers using Holoscan SDK with CUDA 13 containers can rely on CUDA 13 minor compatibility to develop and run on CUDA R580 drivers out of the box. The Holoscan SDK development container may launch with an expected 803 warning titled "WARNING: CUDA Minor Version Compatibility mode ENABLED." Some features may not be available under minor compatibility such as NVRTC SASS runtime compilation. Please refer to CUDA Compatibility documentation for more details. |
| The PyTorch distribution embedded in the Holoscan SDK release container may encounter NVRTC PTX failures consistent with 6059987 above on R580 platforms (x86_64, Jetpack 7.1). Please upgrade to at least R595 drivers. | |
| 6127417 | Holoinfer functional tests using the ONNX Runtime backend may fail on multi-GPU systems. |
| 6656473 | On IGX Thor, the Holoscan Sensor Bridge Python adapter test test_python_adapter_drives_hsb_lite_module can fail with KeyError: 'module_name' when the module directory is not specified. Run the test with --module-dir=/opt/nvidia/hololink/lib/hololink/modules as a workaround. |
v4.5.0
Release Artifacts
- ð Docker container: tag
v4.5.0-cuda13,v4.5.0-cuda12-dgpuandv4.5.0-cuda12-igpu - ð Python wheel:
pip install holoscan==4.5.0 - ðĶïļ Debian packages:
4.5.0.0-1 - ð Documentation
See supported platforms for compatibility.
Release Notes
New Features and Improvements
Core
MultiThreadScheduleris now identified as a legacy scheduler retained for backward compatibility.EventBasedScheduleris recommended for new multi-threaded applications because it avoids polling-thread CPU overhead and supports CPU thread pinning and Linux real-time scheduling policies.
Examples/Benchmarks
- The multi-threaded examples now use
EventBasedSchedulerby default while retaining an option to select the legacyMultiThreadSchedulerfor comparison.
Holoinfer module
InferenceOpnow automatically selects optimized TensorRT dispatch paths for eligible single- and multi-model pipelines, including static- and dynamic-shape execution in sequential or parallel configurations. Existing configurations require no new option; unsupported backends and advanced configurations continue to use the standard dispatch path.- Added
input_mapandoutput_mapas the preferred inference configuration names. The existingpre_processor_mapandinference_mapnames remain supported for backward compatibility.
Source build & Release Artifacts
- Updated the Graph Execution Framework (GXF) dependency to version 5.7.1 with minor security fixes.
- Source-build CUDA architecture selection now accepts semicolon-delimited lists in addition to comma- and space-delimited lists.
Documentation
- Added an Archives entry to the documentation version selector that links to previous Holoscan SDK releases.
- Clarified that source-project workflows should invoke the Holoscan CLI through the repository-provided wrapper so project-specific defaults are applied.
Bug fixes
| Issue | Description |
|---|---|
| 6045751 | The "python-api-system.test_multiple_run" test case previously failed inconsistently (25% of runs). Fixed the test procedure by adding an extra baseline snapshot for leak detection. |
| 6100869 | Distributed apps could stochastically fail at startup with UCX ERROR UCX_KEEPALIVE_INTERVAL value must be greater than 0 on UCX 1.20. UCX's internal config-parser state uses a pthread_mutex_t that is not fork-safe; child processes (driver/worker) launched via fork() could inherit a zero-initialized keepalive interval. Fixed by setting UCX_KEEPALIVE_INTERVAL=20s in the container environment. |
| 6336980 | ./holohub setup could fail on JetPack 7.2 and IGX OS 2.0 due to PEP 668 system-Python restrictions or permissions creating /opt/sccache. Fixed in HoloHub by selecting a single user-owned CLI environment, avoiding automatic host system-Python installation, and elevating only individual system operations when required. |
| HolovizOp | Fixed a segfault when use_exclusive_display=true and vsync=true are both enabled. The present mode is now set before swapchain creation to avoid an immediate recreation on the exclusive display-plane surface. |
Known Issues
| Issue | Description |
|---|---|
| 6059987, 6154550 | CUDA 13.2 forward compatibility is not available for some R580.00 drivers (IGX-SW 2.0) and the "cuda-compat-13-2" package is intentionally removed from the Holoscan SDK CUDA 13 build and development containers. Developers using Holoscan SDK with CUDA 13 containers can rely on CUDA 13 minor compatibility to develop and run on CUDA R580 drivers out of the box. The Holoscan SDK development container may launch with an expected 803 warning titled "WARNING: CUDA Minor Version Compatibility mode ENABLED." Some features may not be available under minor compatibility such as NVRTC SASS runtime compilation. Please refer to CUDA Compatibility documentation for more details. |
| The PyTorch distribution embedded in the Holoscan SDK release container may encounter NVRTC PTX failures consistent with 6059987 above on R580 platforms (x86_64, Jetpack 7.1). Please upgrade to at least R595 drivers. | |
| 6127417 | Holoinfer functional tests using the ONNX Runtime backend may fail on multi-GPU systems. |
v4.4.0
Release Artifacts
- ð Docker container: tag
v4.4.0-cuda13,v4.4.0-cuda12-dgpuandv4.4.0-cuda12-igpu - ð Python wheel:
pip install holoscan==4.4.0 - ðĶïļ Debian packages:
4.4.0.0-1 - ð Documentation
See supported platforms for compatibility.
Release Notes
New Features and Improvements
Core
[[nodiscard]]has been added to allPoseTreeinterface methods (e.g.,get,set,remove,has). Callers that previously discarded these return values will now receive a compiler warning or error. See the Breaking Changes section below for migration guidance.
Operators/Resources/Conditions
FormatConverterOpandFormatConverterGpuResidentOpnow accept"uyvy"as anin_dtype, in addition to the existing"yuyv". The new mode handles YUV 4:2:2 packed input with U/Cb-first byte order (e.g. from many CSI/MIPI camera sensors) and converts it to RGB888 via NPP'snppiCbYCr422ToRGB, removing the need to byte-reorder UYVY frames before format conversion.
Holoviz module
- Improved error surfacing during Vulkan instance creation failure. Error messages now identify the specific layers and extensions that could not be loaded, making it easier to diagnose initialization failures on new or unusual hardware configurations.
- Holoviz now emits a warning when an
InputSpecwith a higher bit-depth pixel format (e.g., 10-bit) is rendered to a lower bit-depth swapchain surface (e.g., an 8-bit surface). The warning identifies the format mismatch and the resulting bit-depth quantization.
Source build & Release Artifacts
- Holoscan Sensor Bridge dependency updated to 2.6.0 GA.
- Fixed the RAPIDS CMake download and setup procedure for source builds (resolved failures caused by an upstream API change in the RAPIDS CMake bootstrap URL).
Documentation
- The Holoscan SDK documentation has been migrated to a Fern-based build system, now published at docs.nvidia.com/holoscan/sdk-user-guide. The new system includes an AI-powered "Ask a question" assistant, a "View as Markdown" option on every page, and direct AI coding-assistant integrations ("Open in Claude", "Open in ChatGPT", "Connect to Cursor", "Connect to Claude Code") accessible from the "More actions" menu.
- Added a new Holoscan CLI Guide page documenting the
holoscan-cliPyPI package, an all-in-one command-line tool to discover, build, run, test, package, and lint Holoscan projects from source. - Added a new JetPack Performance Tuning page with guidance on installing the real-time kernel and running with a minimal headless BSP to achieve lower, more predictable latency on Jetson developer kits.
Breaking Changes
- AJA capture setup guide moved to HoloHub. The
aja_setup.mdpage has been removed from the Holoscan SDK User Guide. Refer to HoloHub for AJA device setup documentation. - Emergent Vision Technologies setup guide removed. The
emergent_setup.mdpage has been removed from the Holoscan SDK User Guide. Refer to the Emergent Vision Technologies documentation for camera setup instructions. PoseTreeinterface methods are now[[nodiscard]]. All return values ofPoseTreemethods (e.g.,get,set,remove,has) carry[[nodiscard]]. Code that called these methods without checking the returnedexpected_tvalue will now produce compiler warnings or errors. Update call sites to inspect or explicitly discard the return value (e.g.,std::ignore = pose_tree.set(...)).- PDF documentation no longer published. The offline PDF user guide is no longer generated. Access the Holoscan SDK documentation online at docs.nvidia.com/holoscan/sdk-user-guide.
Bug fixes
| Issue | Description |
|---|
Known Issues
| Issue | Description |
|---|---|
| 6045751 | The "python-api-system.test_multiple_run" test case is known to fail inconsistently (25% of runs) following PyTest refactoring noted above. |
| 6059987, 6154550 | CUDA 13.2 forward compatibility is not available for some R580.00 drivers (IGX-SW 2.0) and the "cuda-compat-13-2" package is intentionally removed from the Holoscan SDK CUDA 13 build and development containers. Developers using Holoscan SDK with CUDA 13 containers can rely on CUDA 13 minor compatibility to develop and run on CUDA R580 drivers out of the box. The Holoscan SDK development container may launch with an expected 803 warning titled "WARNING: CUDA Minor Version Compatibility mode ENABLED." Some features may not be available under minor compatibility such as NVRTC SASS runtime compilation. Please refer to CUDA Compatibility documentation for more details. |
| The PyTorch distribution embedded in the Holoscan SDK release container may encounter NVRTC PTX failures consistent with 6059987 above on R580 platforms (x86_64, Jetpack 7.1). Please upgrade to at least R595 drivers. | |
| 6204001 | The Holoscan Sensor Bridge TAO PeopleNet examples embedded in the Holoscan SDK development container artifact may fail on IGX Thor platforms with the message: "CUDA driver error 801 (CUDA_ERROR_NOT_SUPPORTED)". Root cause is under investigation. As a workaround, please install and use the latest Holoscan Sensor Bridge project outside of the container environment. |
| 6336980 | ./holohub setup fails on JetPack 7.2 (AGX Thor) and IGX OS 2.0 (IGX Thor) platforms. Running with sudo produces a PEP 668 "externally-managed-environment" error; running from a Python virtual environment fails with Permission denied while creating /opt/sccache. Workaround: for the sudo case, pass --break-system-packages via the env override: sudo env HOLOSCAN_CLI_INSTALL_ARGS='--break-system-packages --pre --extra-index-url https://pypi.nvidia.com holoscan-cli>4.2.0' ./holohub setup. For the virtual-environment case, pre-create the directory first: sudo mkdir -p /opt/sccache && ./holohub setup. |
v4.3.0
Release Artifacts
- ð Docker container: tag
v4.3.0-cuda13,v4.3.0-cuda12-dgpuandv4.3.0-cuda12-igpu - ð Python wheel:
pip install holoscan==4.3.0 - ðĶïļ Debian packages:
4.3.0.0-1 - ð Documentation
See supported platforms for compatibility.
Release Notes
New Features and Improvements
Core
Operator::initialize()overrides no longer need to explicitly callOperator::initialize()from the base class. The framework now automatically runs framework-level initialization (GXF codelet registration, port unique-ID assignment, execution-context creation) after the user'sinitialize()override returns. Existing operators that callOperator::initialize()are unaffected â the call is idempotent and remains safe.
Holoviz module
- HolovizOp: emit a startup warning when the selected swapchain surface format is 8-bit, informing users that 10-bit input formats (
A2B10G10R10_UNORM_PACK32,A2R10G10B10_UNORM_PACK32) â if used asInputSpecâ will be quantized to 8-bit, and how to enable 10-bit output.
Source build & Release Artifacts
- Holoscan SDK Conda recipes and packages are built for CUDA 13 only and require:
- x86_64: glibc âĨ 2.34 (Ubuntu 22.04 and newer)
- aarch64: glibc âĨ 2.39 (Ubuntu 24.04 and newer. Supports IGX OS 2.x, Jetpack 7.x)
- Holoscan SDK Conda support for CUDA 12 is removed. Platforms not meeting the GLIBC and CUDA driver baselines described above should upgrade system requirements or continue to use to the v4.2 Conda packages.
Documentation
- Added a new AI Coding-Assistant Skills section to the user guide, describing the Holoscan-team agent skills published in the NVIDIA/skills catalog: how they work, how to install them via the
skillsCLI (npx skills add nvidia/skills), and which Holoscan SDK install paths they currently cover. This release introduces the first six skills âholoscan-setup,holoscan-install-container,holoscan-install-debian,holoscan-install-wheel,holoscan-install-conda, andholoscan-install-sourceâ also bundled underpublic/skills/for reference.
Bug fixes
| Issue | Description |
|---|---|
| 5211869 | Error "Failed to start server on 0.0.0.0:10002" when debugging the Distributed Endoscopy Tool Tracking application in VS Code on IGX Orin dGPU. |
| 5928213 | PresentDoneCondition crash (SIGABRT / std::terminate) on platforms where the VK_KHR_present_wait Vulkan device extension is unavailable (e.g., IGX Thor iGPU/dGPU, Orin iGPU). |
| 6153741 | Segmentation fault on emit of a GXF Entity with preexisting MetadataDictionary component. |
Known Issues
| Issue | Description |
|---|---|
| 6045751 | The "python-api-system.test_multiple_run" test case is known to fail inconsistently (25% of runs) following PyTest refactoring noted above. |
| 6059987, 6154550 | CUDA 13.2 forward compatibility is not available for some R580.00 drivers (IGX-SW 2.0) and the "cuda-compat-13-2" package is intentionally removed from the Holoscan SDK CUDA 13 build and development containers. Developers using Holoscan SDK with CUDA 13 containers can rely on CUDA 13 minor compatibility to develop and run on CUDA R580 drivers out of the box. The Holoscan SDK development container may launch with an expected 803 warning titled "WARNING: CUDA Minor Version Compatibility mode ENABLED." Some features may not be available under minor compatibility such as NVRTC SASS runtime compilation. Please refer to CUDA Compatibility documentation for more details. |
| The PyTorch distribution embedded in the Holoscan SDK release container may encounter NVRTC PTX failures consistent with 6059987 above on R580 platforms (x86_64, Jetpack 7.1). Please upgrade to at least R595 drivers. | |
| 6204001 | The Holoscan Sensor Bridge TAO PeopleNet examples embedded in the Holoscan SDK development container artifact may fail on IGX Thor platforms with the message: "CUDA driver error 801 (CUDA_ERROR_NOT_SUPPORTED)". Root cause is under investigation. As a workaround, please install and use the latest Holoscan Sensor Bridge project outside of the container environment. |
v4.2.0
Release Artifacts
- ð Docker container: tag
v4.2.0-cuda13,v4.2.0-cuda12-dgpuandv4.2.0-cuda12-igpu - ð Python wheel:
pip install holoscan==4.2.0 - ðĶïļ Debian packages:
4.2.0.1-1 - ð Documentation
See supported platforms for compatibility.
Release Notes
New Features and Improvements
- GPU-resident graph execution now supports Directed Acyclic Graph (DAG) of operators.
Examples/Benchmarks
- New benchmark for performance measurement of
SCHED_DEADLINEscheduling of Holoscan SDK operators under the presence of background operators and pinned scheduler dispatcher thread.
Holoinfer module
- Refactored HoloInfer tests to use GTest entries for individual test discovery with CTest.
Utils
- Enabled device-side debug prints for GPU-resident graph execution when
HOLOSCAN_LOG_LEVELis set toDEBUG. - Refactored PyTest coverage for per-file test discovery with CTest.
Source build & Release Artifacts
- Updated Holoscan SDK development stack for Production Branch 6 alignment.
- Updated CCCL from 2.5.0 (CUDA 12) or 3.0.3 (CUDA 13) to 3.2.0 across CUDA major development.
- Updated CLI11 from 2.3.2 to 2.5.0 to resolve a CMake version warning.
- Bumped NVIDIA GXF from 5.5 to 5.6.
- Updated magic_enum from 0.9.3 to 0.9.7, which requires header path updates.
- Updated NCCL from 2.27 to 2.29 for PyTorch compatibility.
- Updated NSight Systems from 2025.3.1 to 2026.2.1.
- Updated ONNX Runtime from 1.22.1 to 1.24.2.
- Updated PyTorch from 2.9.1 to 2.11.0.
- Updated RMM from 25.10.00 to 26.02.00 with API breakage.
- Updated TensorRT (CUDA 13) from 10.13 to 10.16.
- Updated UCX from 1.19.0 to 1.20.0.
- Updated UCXX from 0.46.0 to 0.48.00.
- Updated Holoscan SDK CUDA 13 release container base to Deep Learning Framework TensorRT 26.03 for Production Branch 6 alignment.
- Updated CUDA 13.0 to CUDA 13.2. Requires >=R595 host driver for full support or >=R580 with CUDA minor compatibility. See known issue 6059987 below.
matx-config.cmakeis now exported as part of the Holoscan SDK installation. Downstream users can use MatX targets from the embedded library in the Holoscan SDK installation.find_package(holoscan) target_link_libraries(my_library PUBLIC matx::matx)
Bug fixes
| Issue | Description |
|---|
Known Issues
| Issue | Description |
|---|---|
| 5211869 | On IGX Orin dGPU, error "Failed to start server on 0.0.0.0:10002" is reported when debugging the Distributed Endoscopy Tool Tracking application in VS Code. |
| 5928213 | PresentDoneCondition crashes with SIGABRT (std::terminate) on platforms where the VK_KHR_present_wait Vulkan device extension is unavailable (e.g. IGX Thor iGPU/dGPU, Orin iGPU). Use FirstPixelOutCondition as an alternative. Fix is targeted for v4.3. |
| 6045751 | The "python-api-system.test_multiple_run" test case is known to fail inconsistently (25% of runs) following PyTest refactoring noted above. |
| 6059987, 6154550 | CUDA 13.2 forward compatibility is not available for some R580.00 drivers (IGX-SW 2.0) and the "cuda-compat-13-2" package is intentionally removed from the Holoscan SDK CUDA 13 build and development containers. Developers using Holoscan SDK with CUDA 13 containers can rely on CUDA 13 minor compatibility to develop and run on CUDA R580 drivers out of the box. The Holoscan SDK development container may launch with an expected 803 warning titled "WARNING: CUDA Minor Version Compatibility mode ENABLED." Some features may not be available under minor compatibility such as NVRTC SASS runtime compilation. Please refer to CUDA Compatibility documentation for more details. |
| The PyTorch distribution embedded in the Holoscan SDK release container may encounter NVRTC PTX failures consistent with 6059987 above on R580 platforms (x86_64, Jetpack 7.1). Please upgrade to at least R595 drivers. | |
| 6153741 | The Holoscan Sensor Bridge 2.5.0 or 2.6.0 (EA) examples tao_peoplenet.py and linux_tao_peoplenet.py may segfault after 10 to 30 seconds due to GXF internal metadata handling. To work around the issue, please set is_metadata_enabled=False in the sample application. |
v4.1.0
Release Artifacts
- ð Docker container: tag
v4.1.0-cuda13,v4.1.0-cuda12-dgpuandv4.1.0-cuda12-igpu - ð Python wheel:
pip install holoscan==4.1.0 - ðĶïļ Debian packages:
4.1.0.1-1 - ð Documentation
See supported platforms for compatibility.
Release Notes
New Features and Improvements
Core
- GPU-resident operators now support multiple input and output ports per connection.
- Renamed the existing flow-oriented graph API to use the canonical
FlowGraph/FlowGraphImplC++ names and the Pythonholoscan.flow_graphsmodule, while reserving theGraphname andcore/graphspath for future use. - The
EventBasedSchedulernow exposes advanced performance-tuning parameters from the underlying GXF scheduler that can reduce lock contention and improve scaling with higher worker thread counts. New optimizations include work stealing between worker queues, a worker-side post-check fast path that bypasses the dispatcher for READY/WAIT_TIME transitions, and sharding of internal notification and wait-state queues. In this release the work stealing (enable_queue_stealing) and post-check fast path optimizations (enable_worker_postcheck_fastpath) are not enabled by default to better preserve v4.0 scheduling behavior and get more real-world experience using these options. See the EventBasedScheduler documentation for details on each parameter. - The underlying GXF runtime now automatically uses an entity pool to reuse message entities, eliminating repeated entity creation/destruction overhead in high-throughput pipelines. This is transparent and requires no application changes. To disable pooling (e.g., for debugging), set the environment variable
GXF_ENTITY_POOL_SIZE=0before launching the application. Fragment::add_subgraphnow takes ownership of the subgraph, enabling a factory pattern where subgraphs of runtime-determined type can be created externally and added to a fragment or parent subgraph.- Added
Fragment::subgraphs()andSubgraph::nested_subgraphs()accessors for inspecting the subgraph hierarchy (e.g. for graph visualization).
Operators/Resources/Conditions
- HolovizOp now supports 16-bit signed and unsigned integer RGB, 32-bit floating point RGB, and 16-bit floating point R, RGB and RGBA image formats.
Holoviz module
- Added support for
R16G16B16_UNORM,R16G16B16_SNORM,R16G16B16_SFLOATandR32G32B32_SFLOATimage formats.
Breaking Changes
- Removed the
holoscan.cliPython module stub that was deprecated in v2.9. Users who need CLI
functionality should install the standaloneholoscan-cli
package viapip install holoscan-cli.
Bug fixes
| Issue | Description |
|---|---|
| 5606400, 5929120 | Resolved Python GIL crash (Fatal Python error: take_gil: PyMUTEX_LOCK(gil->mutex) failed or SIGSEGV) that occurred when running the V4L2 camera Python example on AGX Orin iGPU and AGX Thor iGPU. The issue was caused by libv4l2 plugin loading, which corrupted glibc TLS destructor pointers through dlopen/dlclose of NVIDIA V4L2 codec plugins, as well as a stale PyThreadState left in CPython's GILState TSS, leading to a poisoned PyGILState_Ensure() on GXF worker threads. The immediate solution was to avoid using libv4l2 wrappers in V4L2VideoCaptureOp. A permanent fix is being developed on the NVIDIA V4L2 plugin side. |
| 5933258 | Fixed SIGSEGV in __nptl_deallocate_tsd during thread teardown when running the V4L2 camera C++ example on AGX Orin iGPU and AGX Thor iGPU. Root cause: v4l2_open() loaded all NVIDIA V4L2 plugins indiscriminately, including the Thor/OpenRM CUVID plugin (libv4l2_nvcuvidvideocodec.so) whose libEGL/libGLdispatch dependency chain registered a pthread_key_create TLS destructor that became a dangling pointer after dlclose. Fix: replace v4l2_open()/v4l2_close() with raw POSIX open()/close() in V4L2VideoCaptureOp. A permanent fix is being developed on the NVIDIA V4L2 plugin side. |
| 5946872 | Fixed CuPy failing with Permission denied when the container runs as a non-root user other than the default Ubuntu user (UID 1000), e.g. docker run -u 1001:1001. CuPy was writing its kernel cache under $HOME/.cupy/kernel_cache, which may not exist or be writable for arbitrary UIDs. Official Holoscan Docker images now set CUPY_CACHE_DIR=/tmp/.cupy/kernel_cache (world-writable), and the run script exports a writable cache path for dev-container workflows. Custom images that omit that ENV should set CUPY_CACHE_DIR to a writable directory before running Python. |
Known Issues
| Issue | Description |
|---|---|
| 5211869 | On IGX Orin dGPU, error "Failed to start server on 0.0.0.0:10002" is reported when debugging the Distributed Endoscopy Tool Tracking application in VS Code. |
| 5928213 | On IGX Thor dGPU (Blackwell), holoviz_conditions C++ example application launches with no frame detected and reports the error VK_KHR_present_wait is not available. |
v4.0.0
Release Artifacts
- ð Docker container: tag
v4.0.0-cuda13,v4.0.0-cuda12-dgpuandv4.0.0-cuda12-igpu - ð Python wheel:
pip install holoscan==4.0.0 - ðĶïļ Debian packages:
4.0.0.0-1 - ð Documentation
See supported platforms for compatibility.
Release Notes
New Features and Improvements
Core
-
GPU-resident graph (C++): Holoscan SDK introduces a new execution
mode, called GPU-resident graph execution, that keeps the CUDA compute pipeline on
the GPU for the lifetime of the application, reducing CPU scheduling
overheads, CUDA kernel launch cost, CPU-GPU coordination and improving
deterministic, low-latency behavior.- GPU-resident operators and fragments: Operators inheriting from
holoscan::GPUResidentOperatoruse device memory for ports and are captured into CUDA Graphs. All operators in a fragment must be GPU-resident for that fragment to use GPU-resident graph execution; currently only linear chains of operators are supported. - Port declaration: Ports can be declared by memory block size (executor-allocated shared device memory) or by device pointer (operator-managed); see the GPU-resident user guide for connection semantics.
- Data ready handler (optional): A separate GPU-resident fragment can be registered as a data ready handler to run at the start of each iteration and decide when input data is ready, enabling sensor-driven pipelines with GPU-direct technologies (e.g., Holoscan Sensor Bridge) without host CPU involvement.
- Host control API: After
run_async(), useFragment::gpu_resident()to signal data ready, poll result ready, and configure timeout and optional host sync; see the GPU-resident user guide and examples. - GPU-resident graph execution is supported in C++ only; Python support is planned for a future release.
- GPU-resident operators and fragments: Operators inheriting from
-
Data Flow Tracking - programmatic access and probing: Data flow tracking now supports richer observability and targeting of arbitrary operators.
- Programmatic access in operators: Operators can read data flow tracking information during execution via
get_data_flow_tracking_label()(C++/Python). This returns aMessageLabelfor a given input port, with path names, number of paths, and timing information, enabling adaptive behavior, debugging, in-operator and runtime analysis without waiting for post-run results. - Tracking to arbitrary operators: Use
DataFlowTracker::add_probe_operator()to treat any intermediate (non-root, non-leaf) operator as a probe point. The tracker then reports latency from root operators to the probed operator and the number of messages published by that operator. Probe operators are currently validated withDoubleBufferTransmitter/DoubleBufferReceiverand default connection configurations; see the Data Flow Tracking documentation for details and theflow_trackerexample for usage.
- Programmatic access in operators: Operators can read data flow tracking information during execution via
Operators/Resources/Conditions
- New classes related to publish/subscribe (pub/sub) messaging were added (
PubSubContext,PubSubReceiver,PubSubTransmitter) as wrappers around underlying GXF pub/sub interfaces. In this release, Holoscan does not ship a concrete pub/sub backend implementation, so these APIs should be considered experimental scaffolding for future releases. Relatedly,IOSpecnow includestopicandqosmethods to describe intended pub/sub topic and QoS settings, which will become active once a concrete backend is provided.
Examples
- The
flow_trackerexample demonstrates the new data flow tracking capabilities: programmatic access viaget_data_flow_tracking_label()inside an operator and tracking to intermediate operators viaadd_probe_operator(). - Added
gpu_resident_example,gpu_resident_input, andgpu_resident_inference_exampledemonstrating GPU-resident execution: operator chains, device memory ports, (ingpu_resident_input) device-driven data ready control, and (ingpu_resident_inference_example) GPU-resident inference with the TensorRT (TRT) backend. - Added
matx_allocatorexample (examples/matx/matx_allocator/) demonstrating how to use theMatXAllocatoradapter withRMMAllocatorin a Holoscan pipeline with DLPack tensor interop.
Utils
- Added
MatXAllocatorutility class (holoscan/utils/matx_allocator.hpp) â a lightweight adapter that enables any Holoscan SDK allocator (e.g.,RMMAllocator,BlockMemoryPool,StreamOrderedAllocator) to be used with MatX GPU tensor operations. Supports stream-aware async allocation/deallocation when used withCudaAllocator-derived allocators and a CUDA stream.
Source build & Release Artifacts
-
Added Holoscan Sensor Bridge 2.5.0
basic feature functionality in the Holoscan SDK developer container environment. Holoscan Sensor Bridge libraries and
tools are built with RDMA over Converged Ethernet (RoCE) support targeting discrete GPU platforms with
ConnectX Network Interface Card (NIC) such as DGX Spark or IGX platforms. Libraries may be found under
the/opt/nvidia/hololinkpath in the discrete GPU developer containers. Please refer to the Holoscan
Sensor Bridge User Guide for
detailed requirements.- Note that the containerized Holoscan Sensor Bridge libraries do not support integrated GPU networking stacks (Jetson, IGX Orin 500). Please rebuild Holoscan Sensor Bridge from source for iGPU platform support.
- Holoscan Sensor Bridge support for x86_64 configurations is experimental and not formally verified. Holoscan Sensor Bridge binaries have been included in the Holoscan SDK x86_64 development container to support developers with Holoscan Sensor Bridge emulation via software loopback. We do not recommend using real Holoscan Sensor Bridge hardware with x86_64 platforms at this time.
- Please run the container as the root user to enable full Holoscan Sensor Bridge runtime features.
-
Updated Holoscan SDK CMake import rules with improved handling for external dependencies.
Holoscan SDK C++ and Python binary distributions (Debian packages, Python wheels, etc) traditionally
contain runtime libraries and development rules for dependencies such as RMM, SPDLOG, YAML-CPP and more,
to reduce the downstream developer burden. In this version we improve "holoscan-config.cmake" and
associated files in the Holoscan SDK binary distribution to describe embedded dependencies with
dedicated project configuration files such as "rmm-config.cmake", and rely on CMake's "find_package"
mechanism to relax dependency embeddings.-
Downstream projects may override Holoscan SDK embedded CMake transitive dependencies with "find_package":
# Find RMM from a custom installation in your development environment find_package(rmm HINTS path/to/my/rmm) # Holoscan SDK will favor the custom RMM installation over its own embedded distribution find_package(holoscan)
-
Downstream projects may depend directly on the Holoscan SDK embedded dependencies:
# Populate any embedded dependencies not already provided find_package(holoscan) # If not set in advance, will use the RMM installation embedded in the Holoscan SDK distribution find_package(rmm) # result: rmm_DIR path is already set to the Holoscan SDK installation ... target_link_libraries(mylibrary PUBLIC rmm::rmm) # dependency imported targets are available
-
Note that some header-only dependencies (concurrentqueue, MatX) remain in the Holoscan SDK
distribution without explicit CMake imported targets to describe them.
-
-
Removed the legacy Holoscan SDK runtime Dockerfile as not supported. Please refer to
Holoscan CLI and
HoloHub sample apps for guidance and examples for
writing an optimized Dockerfile for your Holoscan-based project. -
The NGC
holoscan_dev_debresource (Debian packages up to HSDK 0.6) has been archived and
removed from NGC. Debian packages are now distributed exclusively via
NVIDIA Holoscan downloads; use the container
image, Python wheel, or Debian package options listed in the Release Artifacts section above.
Breaking Changes
- Removed the
holoscan.cliPython module stub that was deprecated in v2.9. Users who need CLI
functionality should install the standaloneholoscan-cli
package viapip install holoscan-cli.
Bug fixes
| Issue | Description |
|---|---|
| 5601128 | On IGX Orin iGPU, fix segmentation fault by ensuring matx::make_tensor does not fall back from managed memory to host-pinned memory during allocation |
| 5898338 | Fix error where InferenceOp ignores CudaStreamPool passed positionally (only the named Arg works) |
Known Issues
| Issue | Description |
|---|---|
| 5211869 | On IGX Orin dGPU, error "Failed to start server on 0.0.0.0:10002" is reported when debugging the Distributed Endoscopy Tool Tracking application in VS Code. |
| 5606400 | On AGX Thor Jedha, v4l2_camera_usb_webcam Python example application fails with a segmenta... |
v3.11.0
Release Artifacts
- ð Docker container: tag
v3.11.0-cuda13,v3.11.0-cuda12-dgpuandv3.11.0-cuda12-igpu - ð Python wheel:
pip install holoscan==3.11.0 - ðĶïļ Debian packages:
3.11.0.0-1 - ð Documentation
See supported platforms for compatibility.
Release Notes
New Features and Improvements
Core
-
Stream-Aware Deallocation: Added automatic CUDA stream propagation to
TensorandVideoBuffercomponents duringOutputContext::emit()to enable pool-based allocators (BlockMemoryPool,RMMAllocator,StreamOrderedAllocator) to defer memory reuse until GPU operations complete on the associated stream. This prevents potential race conditions where memory could be returned to the pool while GPU kernels are still reading from it when an operator returned from compute while GPU work was still in progress.For sink operators (operators that consume data but don't emit it), a new
set_deallocation_stream(cudaStream_t stream)method has been added toholoscan::Tensor(holoscan.Tensor.set_deallocation_stream(stream)in Python). Sink operators that return asynchronously while GPU work is ongoing should call this method on received tensors to inform the allocator which stream last accessed the tensor's memory. The method returnstrueif the stream was set successfully, orfalsefor tensors not managed by a Holoscan/GXF allocator (e.g., tensors from CuPy or PyTorch via DLPack).Stream propagation can be disabled by setting the environment variable
HOLOSCAN_DISABLE_ENTITY_STREAM_PROPAGATION=1. See the CUDA Stream Handling documentation for more details. -
Subgraph API Enhancements: Several improvements have been made to the
Subgraphclass to simplify authoring and usage:- Config file support: The
Subgraphconstructor now accepts an optionalconfig_fileparameter to specify a YAML configuration file for the subgraph, enablingfrom_config()duringcompose(). - Broadcast input ports: Input interface ports can now be connected to multiple internal operators, enabling a single external port to broadcast data to multiple destinations within the subgraph.
- Simplified
add_interface_port: The internal operator port name is now optional (defaults to the external port name), and port direction (input/output) can be auto-detected from the operator's port definitions. Fragment::add_subgraph(Fragment.add_subgraphin Python): New method to add a subgraph without interface ports that doesn't need to be connected viaadd_flow.Subgraph::operators()(Subgraph.operatorsin Python): New method to retrieve all operators belonging to the subgraph and its nested subgraphs.- Convenience wrappers: Added
add_data_loggerandregister_servicemethods toSubgraphthat delegate to the parentFragment. - Python bindings: The
InterfacePortclass and interface port retrieval methods (interface_ports,get_interface_operator_port) are now exposed in Python.
- Config file support: The
-
The
MetadataDictionaryobject returned by the C++Operator::metadata()method now has adeep_copy()method that creates independent copies of both the dictionary structure and all containedMetadataObjectinstances, preserving each entry'sMetadataPolicy. For Python, the dict-likeMetadataDictionaryobject returned by theOperator.metadataproperty implements__copy__(shallow copy) and__deepcopy__(callsdeep_copy()) dunder methods, socopy.copy()andcopy.deepcopy()from Python's built-incopymodule behave accordingly. The deep copy creates truly independent metadata dictionaries where modifications to one won't affect another. Note that values stored asstd::shared_ptr<T>are not deep-copiedâonly the pointer is copied. -
A unique, descriptive default name is assigned to all C++ Operator, Condition, Resource, Scheduler, and DataLogger classes provided by the SDK. Previously, for C++, a generic name such as "unnamed_condition" or "unnamed_resource" was used if the user did not provide a name string as the first argument to
make_condition,make_resource, etc. The default names in C++ improve consistency with the existing Python API. This change should not require any changes to existing applications but should make logging more informative when default names were used. -
All scheduler classes (
EventBasedScheduler,GreedyScheduler,MultiThreadScheduler) support a newnetwork_connection_timeoutparameter which defaults to 5000 ms. This parameter is used only by distributed applications. During the initial phase when network connections are being established, thisnetwork_connection_timeout(in ms) is used instead ofstop_on_deadlock_timeoutfor deadlock detection. The parameter should be set long enough to allow sufficient time for UCX connections to be established between all fragments without triggering false deadlock detection. After thenetwork_connection_timeoutperiod has expired, the application switches to using the standardstop_on_deadlock_timeoutfor deadlock detection. Single fragment applications ignorenetwork_connection_timeoutand use onlystop_on_deadlock_timeout. -
For distributed applications, if
Application::scheduler(Application.schedulerin Python) is called, it will now automatically set that scheduler for all fragments of the application unless those fragments already have their own scheduler set viaFragment::scheduler. Previously, theApplication::schedulermethod only worked for non-distributed applications and did not have any effect for distributed applications. -
Various inconsistencies in custom scheduler assignment for distributed applications have been resolved. Whenever the user explicitly sets a scheduler via
Fragment::scheduler(Fragment.schedulerin Python), that scheduler type is now always respected and the user-specified parameters are applied (unless overridden by environment variables). As before, when environment variablesHOLOSCAN_STOP_ON_DEADLOCK,HOLOSCAN_STOP_ON_DEADLOCK_TIMEOUT,HOLOSCAN_MAX_DURATION_MSand/orHOLOSCAN_CHECK_RECESSION_PERIOD_MSare specified, the values specified by these environment variables will override any corresponding argument that was set for the scheduler passed toFragment::scheduler. Similarly, ifHOLOSCAN_DISTRIBUTED_APP_SCHEDULERis set and that scheduler type does not match the user-defined one passed toFragment.scheduler, the scheduler specified by the environment variable will be used.
Operators/Resources/Conditions
-
CudaStreamConditionEnhancements: TheCudaStreamConditionhas been reimplemented to address several limitations of the original:- Multi-receiver support: Now supports multi-receiver ports (
IOSpec::kAnySize) where the number of connections is not known at compile time. Specify the base port name (e.g.,"receivers") and all matching ports (receivers:0,receivers:1, etc.) will be automatically discovered. - Queue size > 1: Now supports input ports with queue sizes greater than 1. By default, all messages in the queue are checked for CUDA streams (
check_all_messages=true). Set tofalseto only check the first message per receiver. - Multiple CudaStreamId per entity: Now correctly handles GXF entities containing more than one
CudaStreamIdcomponent. - Multiple port names: The new
receiversparameter accepts astd::vector<std::string>(list[str]in Python) of receiver names to monitor multiple input ports. - Backwards compatibility: The legacy
receiverparameter (single port name) is still supported but deprecated. A warning will be logged recommending migration to thereceiversparameter.
- Multi-receiver support: Now supports multi-receiver ports (
-
A new
notify_scheduler()method has been added to the baseConditionclass (holoscan::Condition::notify_schedulerin C++,holoscan.core.Condition.notify_schedulerin Python). This method allows event-based conditions (those returningkWaitEventfromcheck()) to signal to the scheduler that an asynchronous event has completed and the condition should be re-evaluated. This is used internally byCudaStreamConditionandAsynchronousCondition, and is now available for custom native conditions. -
The
Receiverclass now exposespeek()andsync()methods.peek(index)allows conditions to inspect messages in the queue without consuming them, andsync()moves messages from the back stage to the main stage for double-buffer queues. -
AsyncDataLoggerResource(and derived classes likeAsyncConsoleLogger) now supports a configurablequeue_typeparameter (DataLoggerQueueTypeenum) to select between two queue implementations:LockFree(default): High-throughput lock-free queue with per-producer FIFO ordering. Best for most use cases.Ordered: Mutex-based queue with strict global FIFO ordering across all producers. Use this when strict temporal ordering is required (e.g., compliance logging, debugging, causal analysis).
In YAML configuration, use
queue_type: "lock_free"orqueue_type: "ordered"(case-insensitive).
Holoviz module
- New present modes are now supported:
SHARED_DEMAND_REFRESHandSHARED_CONTINUOUS_REFRESHto support front buffer rendering. These present modes are available in exclusive display (direct rendering) mode only and can be used to implement front buffer rendering.FIFO_LATEST_READYwhich waits for vblank, presents the current image and skips all previou...
v3.10.0.post1
Post 3.10.0 Release Fixes:
- Update GXF downloading URL in the Dockerfile to use the public NVIDIA artifactory URL.