Skip to content

Releases

v5.0.0

Choose a tag to compare

@jcfr jcfr released this 21 Oct 03:30
358c772

Release Artifacts

The links below identify the Holoscan SDK 5.0 installation channels. Check individual channels for published artifacts.

See SDK Installation prerequisites for supported platforms and system requirements.

Release Notes

New Features and Improvements

Important

Holoscan SDK 5.0 is a ground-up rewrite of the Holoscan SDK runtime and API.

Learn more about the Holoscan platform at holoscan.org.

API stability

The Stable source API core covers basic in-process graph/operator authoring, ordinary built-in scalar ports, existing explicitly named and unambiguous Python port declarations/lookup, and structured error handling in C++ and Python. These documented source contracts remain compatible across 5.x updates; deprecation and removal follow the major-release policy described in API stability. The Stable entry-point call contracts do not freeze omitted-contract trigger resolution, clock selection/admission, or automatic queue/resource derivation; these policies remain Beta. Tensor declarations, ownership, output specifications, and pools remain Beta. Plan inspection, resolved-defaults reports, typed records, and report formats remain Prototype. Existing schema version counters and compatibility epochs are retained across GA and follow their own evolution rules. Source API stability does not promise binary compatibility between SDK packages or cross-version interoperability for channels, schemas, or serialized artifacts. Follow the release-specific rebuild and deployment requirements when upgrading.

Programming model

  • Callback-scoped scalar borrowing (C++ and Python) - Input<T>::selected_sample() exposes immutable selected data and metadata without a payload copy, heap allocation, or owning read claim. sensor_io::ImageDescriptorScratch prepares bounded metadata storage during startup for Image transforms; output pixels still use a separate Tensor loan. Hosted Python Operators expose the same frozen scalar selection as OperatorPort.selected_sample(): metadata remains independently inspectable, while .data lazily materializes an owning Python projection. C++ views, references, and the Python selection facade must not escape the compute callback. Owning receive APIs remain available.
  • Prepared generated-schema transforms (C++) - Generated ordinary and descriptor-Tensor schemas can opt into bounded reusable native metadata with NATIVE_BOUNDS_FILE. schema::SchemaScratch<T> and schema::DescriptorMetadataScratch<T> retain cold-prepared capacity, while Output<T>::emit_ref() synchronously publishes an ordinary generated schema without a by-value native-object copy. Values outside authored native bounds fail with ErrorCode::kResourceExhausted; altering prepared owners or retaining a nested owner across reuse fails closed with ErrorCode::kPreparedStorageInvalid. The bounds policy is a build and deployment contract rather than wire identity, so every producer and prepared consumer for a schema must use the same bounds file. Owning APIs remain available alongside prepared storage.
  • holoscan::Graph API - Graph definition without an Application wrapper. compile() produces an immutable ExecutionPlan; run_async() returns a RunSession for deterministic asynchronous execution and clean shutdown.
  • Typed Operators and ports - Operators declare strongly typed input and output ports; payloads flow as Sample<T> with causal metadata.
  • Reusable Graph embedding - A compiled subgraph can be embedded into a parent graph and reused across multiple compositions.
  • CUDA Graph regions - Applications can explicitly fuse an eligible fixed-shape, linear GPU region with RegionConstraint::cuda_graph(). Runtime captures member CUDA work once after lifecycle startup and replays it on the source's planned stream while retaining ordinary Tensor readiness and publication behavior. The ordinary execution path remains available for comparison.
  • Automatic graph partitioning and inspection - Graphs without partition declarations use the holoscan.partition.source_chain/1 planner, which forms linear execution chains across the complete DAG, including fan-out, fan-in, diamond, and parallel edge topologies. Different partitions may execute callbacks concurrently. Applications can author exact manual anchors and request an automatic remainder, use partition(name).include_all() for one complete composed partition, and inspect the accepted cover through ExecutionPlan::partitions(). Green Context bindings require exact named anchors; dedicated contexts are not assigned automatically.
  • CUDA streams follow execution partitions - By default, each active execution-partition/device pair receives one Runtime-owned CUDA compute stream, allowing independent branches to submit GPU work independently while Tensor readiness and release ordering remain automatic. Applications can select CudaComputeStreamPolicy::kSharedProcessDevice in CompileOptions when measurements favor one ordinary stream per process group and device. Separate streams permit but do not guarantee concurrent CUDA execution.
  • Support-thread placement (C++) - Applications can request CPU affinity for the run-session controller and, when present, the process-local Channel pump. The compiled plan records the admitted placement. A pump binding does not create a pump when the selected transport does not need one.
  • CUDA Green Context manual placement (developer preview) - Applications can query a qualified CUDA 13 target, bind an exact manually authored execution partition to a driver-structured set of streaming multiprocessors, and inspect the accepted placement and diagnostics in the compiled plan. The compiler and Runtime validate the target, materialize the Green Context streams before operator execution, and preserve Tensor ordering across ordinary and Green Context streams. Automatic graph partitioning does not automatically size or assign dedicated contexts. See the CUDA Green Contexts guide.
  • Lifecycle stage graph - Bounded operator lifecycle with deterministic startup, running, and shutdown stages.
  • Managed shutdown signals - On Linux, blocking run(plan) synchronously accepts SIGINT and SIGTERM through an HSDK signal domain by default; the first accepted occurrence requests cooperative stop. RunSignalPolicy::kCallerManaged provides a blocking opt-out, while early ManagedShutdownSignals plus run_async(plan, signals) provides managed asynchronous execution. Plain run_async(plan) remains caller-managed. A second accepted occurrence delegates the actual signal under the unchanged prior policy; first-signal shutdown remains cooperative, with no forced cancellation or whole-run completion bound. A clean first accepted SIGINT returns control after cleanup with a nonzero blocking result (currently 1). Managed blocking runs require an adoptable signal domain or a single live task during setup; otherwise they fail before session creation. Establish ManagedShutdownSignals before creating participant threads when managed asynchronous execution is needed. An adopted domain has one binding attempt; close it after retirement and create a fresh domain for another managed run. A child created while the domain is active inherits blocked SIGINT/SIGTERM, and exec() preserves that mask. Reset the child mask with sigprocmask() before exec(), or use POSIX_SPAWN_SETSIGMASK; system() and popen() do not reset it automatically.

Temporal scheduling

  • Temporal contracts - OnEach, LatestWithin, and Skip combinators control per-input selection policy; each operator declares a Contract that is verified at compile time.
  • Maximum start delay - C++ Contract::max_start_delay(...) and Py...
Read more

v4.6.0

Choose a tag to compare

@jcfr jcfr released this 03 Sep 04:10
66a9609

Release Artifacts

See supported platforms for compatibility.

Release Notes

New Features and Improvements

Source build & Release Artifacts
  • Updated the Holoscan Sensor Bridge (HoloLink) dependency to version 2.7.0. This release adds precompiled hololink_module binaries to the Holoscan SDK development container, allowing applications to adapt to different HSB devices and HSB-IP versions without recompiling. For full details, refer to the Holoscan Sensor Bridge release notes and user guide.
  • Adjusted the bundled fmt integration to avoid NVCC diagnostic 2417-D in C++17 source builds.

Bug fixes

Issue Description
6385530, 6403160 Hardened PoseTree UCX deserialization to reject malformed frame names and invalid or truncated snapshots before mutating state.
6385365, 6385375, 6385708, 6385727, 6385828 Hardened distributed-application and Holoviz codec deserialization against invalid sizes, counts, incomplete payloads, and failed-value access.
6397322 Rejected impossible AppWorker port-allocation requests before reserving memory and added validation for requested counts and port ranges.
6384975, 6385172, 6385241, 6385242, 6385277 Hardened Python GXF recording deserialization against truncated headers, inconsistent tensor metadata, invalid payload ranges, and malformed timestamps.
HoloInfer Fixed a multi-context initialization race that could fail with Current Manager not initialized while preserving the existing public lifecycle API.
Holoviz Fixed Vulkan device-creation failure handling to report the returned VkResult instead of continuing with an invalid device and subsequently crashing.

Known Issues

Issue Description
6059987, 6154550 CUDA 13.2 forward compatibility is not available for some R580.00 drivers (IGX-SW 2.0) and the "cuda-compat-13-2" package is intentionally removed from the Holoscan SDK CUDA 13 build and development containers. Developers using Holoscan SDK with CUDA 13 containers can rely on CUDA 13 minor compatibility to develop and run on CUDA R580 drivers out of the box. The Holoscan SDK development container may launch with an expected 803 warning titled "WARNING: CUDA Minor Version Compatibility mode ENABLED." Some features may not be available under minor compatibility such as NVRTC SASS runtime compilation. Please refer to CUDA Compatibility documentation for more details.
The PyTorch distribution embedded in the Holoscan SDK release container may encounter NVRTC PTX failures consistent with 6059987 above on R580 platforms (x86_64, Jetpack 7.1). Please upgrade to at least R595 drivers.
6127417 Holoinfer functional tests using the ONNX Runtime backend may fail on multi-GPU systems.
6656473 On IGX Thor, the Holoscan Sensor Bridge Python adapter test test_python_adapter_drives_hsb_lite_module can fail with KeyError: 'module_name' when the module directory is not specified. Run the test with --module-dir=/opt/nvidia/hololink/lib/hololink/modules as a workaround.

v4.5.0

Choose a tag to compare

@jcfr jcfr released this 31 Jul 13:17
017eb05

Release Artifacts

See supported platforms for compatibility.

Release Notes

New Features and Improvements

Core
  • MultiThreadScheduler is now identified as a legacy scheduler retained for backward compatibility. EventBasedScheduler is recommended for new multi-threaded applications because it avoids polling-thread CPU overhead and supports CPU thread pinning and Linux real-time scheduling policies.
Examples/Benchmarks
  • The multi-threaded examples now use EventBasedScheduler by default while retaining an option to select the legacy MultiThreadScheduler for comparison.
Holoinfer module
  • InferenceOp now automatically selects optimized TensorRT dispatch paths for eligible single- and multi-model pipelines, including static- and dynamic-shape execution in sequential or parallel configurations. Existing configurations require no new option; unsupported backends and advanced configurations continue to use the standard dispatch path.
  • Added input_map and output_map as the preferred inference configuration names. The existing pre_processor_map and inference_map names remain supported for backward compatibility.
Source build & Release Artifacts
  • Updated the Graph Execution Framework (GXF) dependency to version 5.7.1 with minor security fixes.
  • Source-build CUDA architecture selection now accepts semicolon-delimited lists in addition to comma- and space-delimited lists.
Documentation
  • Added an Archives entry to the documentation version selector that links to previous Holoscan SDK releases.
  • Clarified that source-project workflows should invoke the Holoscan CLI through the repository-provided wrapper so project-specific defaults are applied.

Bug fixes

Issue Description
6045751 The "python-api-system.test_multiple_run" test case previously failed inconsistently (25% of runs). Fixed the test procedure by adding an extra baseline snapshot for leak detection.
6100869 Distributed apps could stochastically fail at startup with UCX ERROR UCX_KEEPALIVE_INTERVAL value must be greater than 0 on UCX 1.20. UCX's internal config-parser state uses a pthread_mutex_t that is not fork-safe; child processes (driver/worker) launched via fork() could inherit a zero-initialized keepalive interval. Fixed by setting UCX_KEEPALIVE_INTERVAL=20s in the container environment.
6336980 ./holohub setup could fail on JetPack 7.2 and IGX OS 2.0 due to PEP 668 system-Python restrictions or permissions creating /opt/sccache. Fixed in HoloHub by selecting a single user-owned CLI environment, avoiding automatic host system-Python installation, and elevating only individual system operations when required.
HolovizOp Fixed a segfault when use_exclusive_display=true and vsync=true are both enabled. The present mode is now set before swapchain creation to avoid an immediate recreation on the exclusive display-plane surface.

Known Issues

Issue Description
6059987, 6154550 CUDA 13.2 forward compatibility is not available for some R580.00 drivers (IGX-SW 2.0) and the "cuda-compat-13-2" package is intentionally removed from the Holoscan SDK CUDA 13 build and development containers. Developers using Holoscan SDK with CUDA 13 containers can rely on CUDA 13 minor compatibility to develop and run on CUDA R580 drivers out of the box. The Holoscan SDK development container may launch with an expected 803 warning titled "WARNING: CUDA Minor Version Compatibility mode ENABLED." Some features may not be available under minor compatibility such as NVRTC SASS runtime compilation. Please refer to CUDA Compatibility documentation for more details.
The PyTorch distribution embedded in the Holoscan SDK release container may encounter NVRTC PTX failures consistent with 6059987 above on R580 platforms (x86_64, Jetpack 7.1). Please upgrade to at least R595 drivers.
6127417 Holoinfer functional tests using the ONNX Runtime backend may fail on multi-GPU systems.

v4.4.0

Choose a tag to compare

@jcfr jcfr released this 03 Jul 12:14
e64af8f

Release Artifacts

See supported platforms for compatibility.

Release Notes

New Features and Improvements

Core
  • [[nodiscard]] has been added to all PoseTree interface methods (e.g., get, set, remove, has). Callers that previously discarded these return values will now receive a compiler warning or error. See the Breaking Changes section below for migration guidance.
Operators/Resources/Conditions
  • FormatConverterOp and FormatConverterGpuResidentOp now accept "uyvy" as an in_dtype, in addition to the existing "yuyv". The new mode handles YUV 4:2:2 packed input with U/Cb-first byte order (e.g. from many CSI/MIPI camera sensors) and converts it to RGB888 via NPP's nppiCbYCr422ToRGB, removing the need to byte-reorder UYVY frames before format conversion.
Holoviz module
  • Improved error surfacing during Vulkan instance creation failure. Error messages now identify the specific layers and extensions that could not be loaded, making it easier to diagnose initialization failures on new or unusual hardware configurations.
  • Holoviz now emits a warning when an InputSpec with a higher bit-depth pixel format (e.g., 10-bit) is rendered to a lower bit-depth swapchain surface (e.g., an 8-bit surface). The warning identifies the format mismatch and the resulting bit-depth quantization.
Source build & Release Artifacts
  • Holoscan Sensor Bridge dependency updated to 2.6.0 GA.
  • Fixed the RAPIDS CMake download and setup procedure for source builds (resolved failures caused by an upstream API change in the RAPIDS CMake bootstrap URL).
Documentation
  • The Holoscan SDK documentation has been migrated to a Fern-based build system, now published at docs.nvidia.com/holoscan/sdk-user-guide. The new system includes an AI-powered "Ask a question" assistant, a "View as Markdown" option on every page, and direct AI coding-assistant integrations ("Open in Claude", "Open in ChatGPT", "Connect to Cursor", "Connect to Claude Code") accessible from the "More actions" menu.
  • Added a new Holoscan CLI Guide page documenting the holoscan-cli PyPI package, an all-in-one command-line tool to discover, build, run, test, package, and lint Holoscan projects from source.
  • Added a new JetPack Performance Tuning page with guidance on installing the real-time kernel and running with a minimal headless BSP to achieve lower, more predictable latency on Jetson developer kits.

Breaking Changes

  • AJA capture setup guide moved to HoloHub. The aja_setup.md page has been removed from the Holoscan SDK User Guide. Refer to HoloHub for AJA device setup documentation.
  • Emergent Vision Technologies setup guide removed. The emergent_setup.md page has been removed from the Holoscan SDK User Guide. Refer to the Emergent Vision Technologies documentation for camera setup instructions.
  • PoseTree interface methods are now [[nodiscard]]. All return values of PoseTree methods (e.g., get, set, remove, has) carry [[nodiscard]]. Code that called these methods without checking the returned expected_t value will now produce compiler warnings or errors. Update call sites to inspect or explicitly discard the return value (e.g., std::ignore = pose_tree.set(...)).
  • PDF documentation no longer published. The offline PDF user guide is no longer generated. Access the Holoscan SDK documentation online at docs.nvidia.com/holoscan/sdk-user-guide.

Bug fixes

Issue Description

Known Issues

Issue Description
6045751 The "python-api-system.test_multiple_run" test case is known to fail inconsistently (25% of runs) following PyTest refactoring noted above.
6059987, 6154550 CUDA 13.2 forward compatibility is not available for some R580.00 drivers (IGX-SW 2.0) and the "cuda-compat-13-2" package is intentionally removed from the Holoscan SDK CUDA 13 build and development containers. Developers using Holoscan SDK with CUDA 13 containers can rely on CUDA 13 minor compatibility to develop and run on CUDA R580 drivers out of the box. The Holoscan SDK development container may launch with an expected 803 warning titled "WARNING: CUDA Minor Version Compatibility mode ENABLED." Some features may not be available under minor compatibility such as NVRTC SASS runtime compilation. Please refer to CUDA Compatibility documentation for more details.
The PyTorch distribution embedded in the Holoscan SDK release container may encounter NVRTC PTX failures consistent with 6059987 above on R580 platforms (x86_64, Jetpack 7.1). Please upgrade to at least R595 drivers.
6204001 The Holoscan Sensor Bridge TAO PeopleNet examples embedded in the Holoscan SDK development container artifact may fail on IGX Thor platforms with the message: "CUDA driver error 801 (CUDA_ERROR_NOT_SUPPORTED)". Root cause is under investigation. As a workaround, please install and use the latest Holoscan Sensor Bridge project outside of the container environment.
6336980 ./holohub setup fails on JetPack 7.2 (AGX Thor) and IGX OS 2.0 (IGX Thor) platforms. Running with sudo produces a PEP 668 "externally-managed-environment" error; running from a Python virtual environment fails with Permission denied while creating /opt/sccache. Workaround: for the sudo case, pass --break-system-packages via the env override: sudo env HOLOSCAN_CLI_INSTALL_ARGS='--break-system-packages --pre --extra-index-url https://pypi.nvidia.com holoscan-cli>4.2.0' ./holohub setup. For the virtual-environment case, pre-create the directory first: sudo mkdir -p /opt/sccache && ./holohub setup.

v4.3.0

Choose a tag to compare

@jcfr jcfr released this 31 May 10:50
e3a702d

Release Artifacts

See supported platforms for compatibility.

Release Notes

New Features and Improvements

Core
  • Operator::initialize() overrides no longer need to explicitly call Operator::initialize() from the base class. The framework now automatically runs framework-level initialization (GXF codelet registration, port unique-ID assignment, execution-context creation) after the user's initialize() override returns. Existing operators that call Operator::initialize() are unaffected — the call is idempotent and remains safe.
Holoviz module
  • HolovizOp: emit a startup warning when the selected swapchain surface format is 8-bit, informing users that 10-bit input formats (A2B10G10R10_UNORM_PACK32, A2R10G10B10_UNORM_PACK32) — if used as InputSpec — will be quantized to 8-bit, and how to enable 10-bit output.
Source build & Release Artifacts
  • Holoscan SDK Conda recipes and packages are built for CUDA 13 only and require:
    • x86_64: glibc â‰Ĩ 2.34 (Ubuntu 22.04 and newer)
    • aarch64: glibc â‰Ĩ 2.39 (Ubuntu 24.04 and newer. Supports IGX OS 2.x, Jetpack 7.x)
  • Holoscan SDK Conda support for CUDA 12 is removed. Platforms not meeting the GLIBC and CUDA driver baselines described above should upgrade system requirements or continue to use to the v4.2 Conda packages.
Documentation
  • Added a new AI Coding-Assistant Skills section to the user guide, describing the Holoscan-team agent skills published in the NVIDIA/skills catalog: how they work, how to install them via the skills CLI (npx skills add nvidia/skills), and which Holoscan SDK install paths they currently cover. This release introduces the first six skills — holoscan-setup, holoscan-install-container, holoscan-install-debian, holoscan-install-wheel, holoscan-install-conda, and holoscan-install-source — also bundled under public/skills/ for reference.

Bug fixes

Issue Description
5211869 Error "Failed to start server on 0.0.0.0:10002" when debugging the Distributed Endoscopy Tool Tracking application in VS Code on IGX Orin dGPU.
5928213 PresentDoneCondition crash (SIGABRT / std::terminate) on platforms where the VK_KHR_present_wait Vulkan device extension is unavailable (e.g., IGX Thor iGPU/dGPU, Orin iGPU).
6153741 Segmentation fault on emit of a GXF Entity with preexisting MetadataDictionary component.

Known Issues

Issue Description
6045751 The "python-api-system.test_multiple_run" test case is known to fail inconsistently (25% of runs) following PyTest refactoring noted above.
6059987, 6154550 CUDA 13.2 forward compatibility is not available for some R580.00 drivers (IGX-SW 2.0) and the "cuda-compat-13-2" package is intentionally removed from the Holoscan SDK CUDA 13 build and development containers. Developers using Holoscan SDK with CUDA 13 containers can rely on CUDA 13 minor compatibility to develop and run on CUDA R580 drivers out of the box. The Holoscan SDK development container may launch with an expected 803 warning titled "WARNING: CUDA Minor Version Compatibility mode ENABLED." Some features may not be available under minor compatibility such as NVRTC SASS runtime compilation. Please refer to CUDA Compatibility documentation for more details.
The PyTorch distribution embedded in the Holoscan SDK release container may encounter NVRTC PTX failures consistent with 6059987 above on R580 platforms (x86_64, Jetpack 7.1). Please upgrade to at least R595 drivers.
6204001 The Holoscan Sensor Bridge TAO PeopleNet examples embedded in the Holoscan SDK development container artifact may fail on IGX Thor platforms with the message: "CUDA driver error 801 (CUDA_ERROR_NOT_SUPPORTED)". Root cause is under investigation. As a workaround, please install and use the latest Holoscan Sensor Bridge project outside of the container environment.

v4.2.0

Choose a tag to compare

@jcfr jcfr released this 11 May 21:06
1f6ef48

Release Artifacts

See supported platforms for compatibility.

Release Notes

New Features and Improvements

  • GPU-resident graph execution now supports Directed Acyclic Graph (DAG) of operators.
Examples/Benchmarks
  • New benchmark for performance measurement of SCHED_DEADLINE scheduling of Holoscan SDK operators under the presence of background operators and pinned scheduler dispatcher thread.
Holoinfer module
  • Refactored HoloInfer tests to use GTest entries for individual test discovery with CTest.
Utils
  • Enabled device-side debug prints for GPU-resident graph execution when HOLOSCAN_LOG_LEVEL is set to DEBUG.
  • Refactored PyTest coverage for per-file test discovery with CTest.
Source build & Release Artifacts
  • Updated Holoscan SDK development stack for Production Branch 6 alignment.
    • Updated CCCL from 2.5.0 (CUDA 12) or 3.0.3 (CUDA 13) to 3.2.0 across CUDA major development.
    • Updated CLI11 from 2.3.2 to 2.5.0 to resolve a CMake version warning.
    • Bumped NVIDIA GXF from 5.5 to 5.6.
    • Updated magic_enum from 0.9.3 to 0.9.7, which requires header path updates.
    • Updated NCCL from 2.27 to 2.29 for PyTorch compatibility.
    • Updated NSight Systems from 2025.3.1 to 2026.2.1.
    • Updated ONNX Runtime from 1.22.1 to 1.24.2.
    • Updated PyTorch from 2.9.1 to 2.11.0.
    • Updated RMM from 25.10.00 to 26.02.00 with API breakage.
    • Updated TensorRT (CUDA 13) from 10.13 to 10.16.
    • Updated UCX from 1.19.0 to 1.20.0.
    • Updated UCXX from 0.46.0 to 0.48.00.
  • Updated Holoscan SDK CUDA 13 release container base to Deep Learning Framework TensorRT 26.03 for Production Branch 6 alignment.
    • Updated CUDA 13.0 to CUDA 13.2. Requires >=R595 host driver for full support or >=R580 with CUDA minor compatibility. See known issue 6059987 below.
  • matx-config.cmake is now exported as part of the Holoscan SDK installation. Downstream users can use MatX targets from the embedded library in the Holoscan SDK installation.
    find_package(holoscan)
    target_link_libraries(my_library PUBLIC matx::matx)

Bug fixes

Issue Description

Known Issues

Issue Description
5211869 On IGX Orin dGPU, error "Failed to start server on 0.0.0.0:10002" is reported when debugging the Distributed Endoscopy Tool Tracking application in VS Code.
5928213 PresentDoneCondition crashes with SIGABRT (std::terminate) on platforms where the VK_KHR_present_wait Vulkan device extension is unavailable (e.g. IGX Thor iGPU/dGPU, Orin iGPU). Use FirstPixelOutCondition as an alternative. Fix is targeted for v4.3.
6045751 The "python-api-system.test_multiple_run" test case is known to fail inconsistently (25% of runs) following PyTest refactoring noted above.
6059987, 6154550 CUDA 13.2 forward compatibility is not available for some R580.00 drivers (IGX-SW 2.0) and the "cuda-compat-13-2" package is intentionally removed from the Holoscan SDK CUDA 13 build and development containers. Developers using Holoscan SDK with CUDA 13 containers can rely on CUDA 13 minor compatibility to develop and run on CUDA R580 drivers out of the box. The Holoscan SDK development container may launch with an expected 803 warning titled "WARNING: CUDA Minor Version Compatibility mode ENABLED." Some features may not be available under minor compatibility such as NVRTC SASS runtime compilation. Please refer to CUDA Compatibility documentation for more details.
The PyTorch distribution embedded in the Holoscan SDK release container may encounter NVRTC PTX failures consistent with 6059987 above on R580 platforms (x86_64, Jetpack 7.1). Please upgrade to at least R595 drivers.
6153741 The Holoscan Sensor Bridge 2.5.0 or 2.6.0 (EA) examples tao_peoplenet.py and linux_tao_peoplenet.py may segfault after 10 to 30 seconds due to GXF internal metadata handling. To work around the issue, please set is_metadata_enabled=False in the sample application.

v4.1.0

Choose a tag to compare

@jcfr jcfr released this 01 Apr 20:57
77d6387

Release Artifacts

See supported platforms for compatibility.

Release Notes

New Features and Improvements

Core
  • GPU-resident operators now support multiple input and output ports per connection.
  • Renamed the existing flow-oriented graph API to use the canonical FlowGraph / FlowGraphImpl C++ names and the Python holoscan.flow_graphs module, while reserving the Graph name and core/graphs path for future use.
  • The EventBasedScheduler now exposes advanced performance-tuning parameters from the underlying GXF scheduler that can reduce lock contention and improve scaling with higher worker thread counts. New optimizations include work stealing between worker queues, a worker-side post-check fast path that bypasses the dispatcher for READY/WAIT_TIME transitions, and sharding of internal notification and wait-state queues. In this release the work stealing (enable_queue_stealing) and post-check fast path optimizations (enable_worker_postcheck_fastpath) are not enabled by default to better preserve v4.0 scheduling behavior and get more real-world experience using these options. See the EventBasedScheduler documentation for details on each parameter.
  • The underlying GXF runtime now automatically uses an entity pool to reuse message entities, eliminating repeated entity creation/destruction overhead in high-throughput pipelines. This is transparent and requires no application changes. To disable pooling (e.g., for debugging), set the environment variable GXF_ENTITY_POOL_SIZE=0 before launching the application.
  • Fragment::add_subgraph now takes ownership of the subgraph, enabling a factory pattern where subgraphs of runtime-determined type can be created externally and added to a fragment or parent subgraph.
  • Added Fragment::subgraphs() and Subgraph::nested_subgraphs() accessors for inspecting the subgraph hierarchy (e.g. for graph visualization).
Operators/Resources/Conditions
  • HolovizOp now supports 16-bit signed and unsigned integer RGB, 32-bit floating point RGB, and 16-bit floating point R, RGB and RGBA image formats.
Holoviz module
  • Added support for R16G16B16_UNORM, R16G16B16_SNORM, R16G16B16_SFLOAT and R32G32B32_SFLOAT image formats.

Breaking Changes

  • Removed the holoscan.cli Python module stub that was deprecated in v2.9. Users who need CLI
    functionality should install the standalone holoscan-cli
    package via pip install holoscan-cli.

Bug fixes

Issue Description
5606400, 5929120 Resolved Python GIL crash (Fatal Python error: take_gil: PyMUTEX_LOCK(gil->mutex) failed or SIGSEGV) that occurred when running the V4L2 camera Python example on AGX Orin iGPU and AGX Thor iGPU. The issue was caused by libv4l2 plugin loading, which corrupted glibc TLS destructor pointers through dlopen/dlclose of NVIDIA V4L2 codec plugins, as well as a stale PyThreadState left in CPython's GILState TSS, leading to a poisoned PyGILState_Ensure() on GXF worker threads. The immediate solution was to avoid using libv4l2 wrappers in V4L2VideoCaptureOp. A permanent fix is being developed on the NVIDIA V4L2 plugin side.
5933258 Fixed SIGSEGV in __nptl_deallocate_tsd during thread teardown when running the V4L2 camera C++ example on AGX Orin iGPU and AGX Thor iGPU. Root cause: v4l2_open() loaded all NVIDIA V4L2 plugins indiscriminately, including the Thor/OpenRM CUVID plugin (libv4l2_nvcuvidvideocodec.so) whose libEGL/libGLdispatch dependency chain registered a pthread_key_create TLS destructor that became a dangling pointer after dlclose. Fix: replace v4l2_open()/v4l2_close() with raw POSIX open()/close() in V4L2VideoCaptureOp. A permanent fix is being developed on the NVIDIA V4L2 plugin side.
5946872 Fixed CuPy failing with Permission denied when the container runs as a non-root user other than the default Ubuntu user (UID 1000), e.g. docker run -u 1001:1001. CuPy was writing its kernel cache under $HOME/.cupy/kernel_cache, which may not exist or be writable for arbitrary UIDs. Official Holoscan Docker images now set CUPY_CACHE_DIR=/tmp/.cupy/kernel_cache (world-writable), and the run script exports a writable cache path for dev-container workflows. Custom images that omit that ENV should set CUPY_CACHE_DIR to a writable directory before running Python.

Known Issues

Issue Description
5211869 On IGX Orin dGPU, error "Failed to start server on 0.0.0.0:10002" is reported when debugging the Distributed Endoscopy Tool Tracking application in VS Code.
5928213 On IGX Thor dGPU (Blackwell), holoviz_conditions C++ example application launches with no frame detected and reports the error VK_KHR_present_wait is not available.

v4.0.0

Choose a tag to compare

@jcfr jcfr released this 09 Mar 19:47
874dc9d

Release Artifacts

See supported platforms for compatibility.

Release Notes

New Features and Improvements

Core
  • GPU-resident graph (C++): Holoscan SDK introduces a new execution
    mode, called GPU-resident graph execution, that keeps the CUDA compute pipeline on
    the GPU for the lifetime of the application, reducing CPU scheduling
    overheads, CUDA kernel launch cost, CPU-GPU coordination and improving
    deterministic, low-latency behavior.

    • GPU-resident operators and fragments: Operators inheriting from holoscan::GPUResidentOperator use device memory for ports and are captured into CUDA Graphs. All operators in a fragment must be GPU-resident for that fragment to use GPU-resident graph execution; currently only linear chains of operators are supported.
    • Port declaration: Ports can be declared by memory block size (executor-allocated shared device memory) or by device pointer (operator-managed); see the GPU-resident user guide for connection semantics.
    • Data ready handler (optional): A separate GPU-resident fragment can be registered as a data ready handler to run at the start of each iteration and decide when input data is ready, enabling sensor-driven pipelines with GPU-direct technologies (e.g., Holoscan Sensor Bridge) without host CPU involvement.
    • Host control API: After run_async(), use Fragment::gpu_resident() to signal data ready, poll result ready, and configure timeout and optional host sync; see the GPU-resident user guide and examples.
    • GPU-resident graph execution is supported in C++ only; Python support is planned for a future release.
  • Data Flow Tracking - programmatic access and probing: Data flow tracking now supports richer observability and targeting of arbitrary operators.

    • Programmatic access in operators: Operators can read data flow tracking information during execution via get_data_flow_tracking_label() (C++/Python). This returns a MessageLabel for a given input port, with path names, number of paths, and timing information, enabling adaptive behavior, debugging, in-operator and runtime analysis without waiting for post-run results.
    • Tracking to arbitrary operators: Use DataFlowTracker::add_probe_operator() to treat any intermediate (non-root, non-leaf) operator as a probe point. The tracker then reports latency from root operators to the probed operator and the number of messages published by that operator. Probe operators are currently validated with DoubleBufferTransmitter/DoubleBufferReceiver and default connection configurations; see the Data Flow Tracking documentation for details and the flow_tracker example for usage.
Operators/Resources/Conditions
  • New classes related to publish/subscribe (pub/sub) messaging were added (PubSubContext, PubSubReceiver, PubSubTransmitter) as wrappers around underlying GXF pub/sub interfaces. In this release, Holoscan does not ship a concrete pub/sub backend implementation, so these APIs should be considered experimental scaffolding for future releases. Relatedly, IOSpec now includes topic and qos methods to describe intended pub/sub topic and QoS settings, which will become active once a concrete backend is provided.
Examples
  • The flow_tracker example demonstrates the new data flow tracking capabilities: programmatic access via get_data_flow_tracking_label() inside an operator and tracking to intermediate operators via add_probe_operator().
  • Added gpu_resident_example, gpu_resident_input, and gpu_resident_inference_example demonstrating GPU-resident execution: operator chains, device memory ports, (in gpu_resident_input) device-driven data ready control, and (in gpu_resident_inference_example) GPU-resident inference with the TensorRT (TRT) backend.
  • Added matx_allocator example (examples/matx/matx_allocator/) demonstrating how to use the MatXAllocator adapter with RMMAllocator in a Holoscan pipeline with DLPack tensor interop.
Utils
  • Added MatXAllocator utility class (holoscan/utils/matx_allocator.hpp) — a lightweight adapter that enables any Holoscan SDK allocator (e.g., RMMAllocator, BlockMemoryPool, StreamOrderedAllocator) to be used with MatX GPU tensor operations. Supports stream-aware async allocation/deallocation when used with CudaAllocator-derived allocators and a CUDA stream.
Source build & Release Artifacts
  • Added Holoscan Sensor Bridge 2.5.0
    basic feature functionality in the Holoscan SDK developer container environment. Holoscan Sensor Bridge libraries and
    tools are built with RDMA over Converged Ethernet (RoCE) support targeting discrete GPU platforms with
    ConnectX Network Interface Card (NIC) such as DGX Spark or IGX platforms. Libraries may be found under
    the /opt/nvidia/hololink path in the discrete GPU developer containers. Please refer to the Holoscan
    Sensor Bridge User Guide for
    detailed requirements.

    • Note that the containerized Holoscan Sensor Bridge libraries do not support integrated GPU networking stacks (Jetson, IGX Orin 500). Please rebuild Holoscan Sensor Bridge from source for iGPU platform support.
    • Holoscan Sensor Bridge support for x86_64 configurations is experimental and not formally verified. Holoscan Sensor Bridge binaries have been included in the Holoscan SDK x86_64 development container to support developers with Holoscan Sensor Bridge emulation via software loopback. We do not recommend using real Holoscan Sensor Bridge hardware with x86_64 platforms at this time.
    • Please run the container as the root user to enable full Holoscan Sensor Bridge runtime features.
  • Updated Holoscan SDK CMake import rules with improved handling for external dependencies.
    Holoscan SDK C++ and Python binary distributions (Debian packages, Python wheels, etc) traditionally
    contain runtime libraries and development rules for dependencies such as RMM, SPDLOG, YAML-CPP and more,
    to reduce the downstream developer burden. In this version we improve "holoscan-config.cmake" and
    associated files in the Holoscan SDK binary distribution to describe embedded dependencies with
    dedicated project configuration files such as "rmm-config.cmake", and rely on CMake's "find_package"
    mechanism to relax dependency embeddings.

    • Downstream projects may override Holoscan SDK embedded CMake transitive dependencies with "find_package":

      # Find RMM from a custom installation in your development environment
      find_package(rmm HINTS path/to/my/rmm)
      
      # Holoscan SDK will favor the custom RMM installation over its own embedded distribution
      find_package(holoscan)
    • Downstream projects may depend directly on the Holoscan SDK embedded dependencies:

      # Populate any embedded dependencies not already provided
      find_package(holoscan)
      
      # If not set in advance, will use the RMM installation embedded in the Holoscan SDK distribution
      find_package(rmm)     # result: rmm_DIR path is already set to the Holoscan SDK installation
      ...
      target_link_libraries(mylibrary PUBLIC rmm::rmm)   # dependency imported targets are available
    • Note that some header-only dependencies (concurrentqueue, MatX) remain in the Holoscan SDK
      distribution without explicit CMake imported targets to describe them.

  • Removed the legacy Holoscan SDK runtime Dockerfile as not supported. Please refer to
    Holoscan CLI and
    HoloHub sample apps for guidance and examples for
    writing an optimized Dockerfile for your Holoscan-based project.

  • The NGC holoscan_dev_deb resource (Debian packages up to HSDK 0.6) has been archived and
    removed from NGC. Debian packages are now distributed exclusively via
    NVIDIA Holoscan downloads; use the container
    image, Python wheel, or Debian package options listed in the Release Artifacts section above.

Breaking Changes

  • Removed the holoscan.cli Python module stub that was deprecated in v2.9. Users who need CLI
    functionality should install the standalone holoscan-cli
    package via pip install holoscan-cli.

Bug fixes

Issue Description
5601128 On IGX Orin iGPU, fix segmentation fault by ensuring matx::make_tensor does not fall back from managed memory to host-pinned memory during allocation
5898338 Fix error where InferenceOp ignores CudaStreamPool passed positionally (only the named Arg works)

Known Issues

Issue Description
5211869 On IGX Orin dGPU, error "Failed to start server on 0.0.0.0:10002" is reported when debugging the Distributed Endoscopy Tool Tracking application in VS Code.
5606400 On AGX Thor Jedha, v4l2_camera_usb_webcam Python example application fails with a segmenta...
Read more

v3.11.0

Choose a tag to compare

@jcfr jcfr released this 04 Feb 23:07
c5d3327

Release Artifacts

See supported platforms for compatibility.

Release Notes

New Features and Improvements

Core
  • Stream-Aware Deallocation: Added automatic CUDA stream propagation to Tensor and VideoBuffer components during OutputContext::emit() to enable pool-based allocators (BlockMemoryPool, RMMAllocator, StreamOrderedAllocator) to defer memory reuse until GPU operations complete on the associated stream. This prevents potential race conditions where memory could be returned to the pool while GPU kernels are still reading from it when an operator returned from compute while GPU work was still in progress.

    For sink operators (operators that consume data but don't emit it), a new set_deallocation_stream(cudaStream_t stream) method has been added to holoscan::Tensor (holoscan.Tensor.set_deallocation_stream(stream) in Python). Sink operators that return asynchronously while GPU work is ongoing should call this method on received tensors to inform the allocator which stream last accessed the tensor's memory. The method returns true if the stream was set successfully, or false for tensors not managed by a Holoscan/GXF allocator (e.g., tensors from CuPy or PyTorch via DLPack).

    Stream propagation can be disabled by setting the environment variable HOLOSCAN_DISABLE_ENTITY_STREAM_PROPAGATION=1. See the CUDA Stream Handling documentation for more details.

  • Subgraph API Enhancements: Several improvements have been made to the Subgraph class to simplify authoring and usage:

    • Config file support: The Subgraph constructor now accepts an optional config_file parameter to specify a YAML configuration file for the subgraph, enabling from_config() during compose().
    • Broadcast input ports: Input interface ports can now be connected to multiple internal operators, enabling a single external port to broadcast data to multiple destinations within the subgraph.
    • Simplified add_interface_port: The internal operator port name is now optional (defaults to the external port name), and port direction (input/output) can be auto-detected from the operator's port definitions.
    • Fragment::add_subgraph (Fragment.add_subgraph in Python): New method to add a subgraph without interface ports that doesn't need to be connected via add_flow.
    • Subgraph::operators() (Subgraph.operators in Python): New method to retrieve all operators belonging to the subgraph and its nested subgraphs.
    • Convenience wrappers: Added add_data_logger and register_service methods to Subgraph that delegate to the parent Fragment.
    • Python bindings: The InterfacePort class and interface port retrieval methods (interface_ports, get_interface_operator_port) are now exposed in Python.
  • The MetadataDictionary object returned by the C++ Operator::metadata() method now has a deep_copy() method that creates independent copies of both the dictionary structure and all contained MetadataObject instances, preserving each entry's MetadataPolicy. For Python, the dict-like MetadataDictionary object returned by the Operator.metadata property implements __copy__ (shallow copy) and __deepcopy__ (calls deep_copy()) dunder methods, so copy.copy() and copy.deepcopy() from Python's built-in copy module behave accordingly. The deep copy creates truly independent metadata dictionaries where modifications to one won't affect another. Note that values stored as std::shared_ptr<T> are not deep-copied—only the pointer is copied.

  • A unique, descriptive default name is assigned to all C++ Operator, Condition, Resource, Scheduler, and DataLogger classes provided by the SDK. Previously, for C++, a generic name such as "unnamed_condition" or "unnamed_resource" was used if the user did not provide a name string as the first argument to make_condition, make_resource, etc. The default names in C++ improve consistency with the existing Python API. This change should not require any changes to existing applications but should make logging more informative when default names were used.

  • All scheduler classes (EventBasedScheduler, GreedyScheduler, MultiThreadScheduler) support a new network_connection_timeout parameter which defaults to 5000 ms. This parameter is used only by distributed applications. During the initial phase when network connections are being established, this network_connection_timeout (in ms) is used instead of stop_on_deadlock_timeout for deadlock detection. The parameter should be set long enough to allow sufficient time for UCX connections to be established between all fragments without triggering false deadlock detection. After the network_connection_timeout period has expired, the application switches to using the standard stop_on_deadlock_timeout for deadlock detection. Single fragment applications ignore network_connection_timeout and use only stop_on_deadlock_timeout.

  • For distributed applications, if Application::scheduler (Application.scheduler in Python) is called, it will now automatically set that scheduler for all fragments of the application unless those fragments already have their own scheduler set via Fragment::scheduler. Previously, the Application::scheduler method only worked for non-distributed applications and did not have any effect for distributed applications.

  • Various inconsistencies in custom scheduler assignment for distributed applications have been resolved. Whenever the user explicitly sets a scheduler via Fragment::scheduler (Fragment.scheduler in Python), that scheduler type is now always respected and the user-specified parameters are applied (unless overridden by environment variables). As before, when environment variables HOLOSCAN_STOP_ON_DEADLOCK, HOLOSCAN_STOP_ON_DEADLOCK_TIMEOUT, HOLOSCAN_MAX_DURATION_MS and/or HOLOSCAN_CHECK_RECESSION_PERIOD_MS are specified, the values specified by these environment variables will override any corresponding argument that was set for the scheduler passed to Fragment::scheduler. Similarly, if HOLOSCAN_DISTRIBUTED_APP_SCHEDULER is set and that scheduler type does not match the user-defined one passed to Fragment.scheduler, the scheduler specified by the environment variable will be used.

Operators/Resources/Conditions
  • CudaStreamCondition Enhancements: The CudaStreamCondition has been reimplemented to address several limitations of the original:

    • Multi-receiver support: Now supports multi-receiver ports (IOSpec::kAnySize) where the number of connections is not known at compile time. Specify the base port name (e.g., "receivers") and all matching ports (receivers:0, receivers:1, etc.) will be automatically discovered.
    • Queue size > 1: Now supports input ports with queue sizes greater than 1. By default, all messages in the queue are checked for CUDA streams (check_all_messages=true). Set to false to only check the first message per receiver.
    • Multiple CudaStreamId per entity: Now correctly handles GXF entities containing more than one CudaStreamId component.
    • Multiple port names: The new receivers parameter accepts a std::vector<std::string> (list[str] in Python) of receiver names to monitor multiple input ports.
    • Backwards compatibility: The legacy receiver parameter (single port name) is still supported but deprecated. A warning will be logged recommending migration to the receivers parameter.
  • A new notify_scheduler() method has been added to the base Condition class (holoscan::Condition::notify_scheduler in C++, holoscan.core.Condition.notify_scheduler in Python). This method allows event-based conditions (those returning kWaitEvent from check()) to signal to the scheduler that an asynchronous event has completed and the condition should be re-evaluated. This is used internally by CudaStreamCondition and AsynchronousCondition, and is now available for custom native conditions.

  • The Receiver class now exposes peek() and sync() methods. peek(index) allows conditions to inspect messages in the queue without consuming them, and sync() moves messages from the back stage to the main stage for double-buffer queues.

  • AsyncDataLoggerResource (and derived classes like AsyncConsoleLogger) now supports a configurable queue_type parameter (DataLoggerQueueType enum) to select between two queue implementations:

    • LockFree (default): High-throughput lock-free queue with per-producer FIFO ordering. Best for most use cases.
    • Ordered: Mutex-based queue with strict global FIFO ordering across all producers. Use this when strict temporal ordering is required (e.g., compliance logging, debugging, causal analysis).

    In YAML configuration, use queue_type: "lock_free" or queue_type: "ordered" (case-insensitive).

Holoviz module
  • New present modes are now supported:
    • SHARED_DEMAND_REFRESH and SHARED_CONTINUOUS_REFRESH to support front buffer rendering. These present modes are available in exclusive display (direct rendering) mode only and can be used to implement front buffer rendering.
    • FIFO_LATEST_READY which waits for vblank, presents the current image and skips all previou...
Read more

v3.10.0.post1

Choose a tag to compare

@jcfr jcfr released this 12 Jan 22:26
134bebd

Post 3.10.0 Release Fixes:

  • Update GXF downloading URL in the Dockerfile to use the public NVIDIA artifactory URL.