Repository navigation
Add dynamic CUB dispatch for radix_sort - #4135
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
🟨 CI finished in 1h 03m: Pass: 3%/93 | Total: 10h 32m | Avg: 6m 47s | Max: 1h 01m | Hits: 98%/320
|
| Project | |
|---|---|
| CCCL Infrastructure | |
| libcu++ | |
| +/- | CUB |
| Thrust | |
| CUDA Experimental | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| CCCL Infrastructure | |
| libcu++ | |
| +/- | CUB |
| +/- | Thrust |
| CUDA Experimental | |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
🏃 Runner counts (total jobs: 93)
| # | Runner |
|---|---|
| 66 | linux-amd64-cpu16 |
| 9 | windows-amd64-cpu16 |
| 6 | linux-amd64-gpu-rtxa6000-latest-1 |
| 4 | linux-arm64-cpu16 |
| 3 | linux-amd64-gpu-h100-latest-1 |
| 3 | linux-amd64-gpu-rtx4090-latest-1 |
| 2 | linux-amd64-gpu-rtx2080-latest-1 |
|
/ok to test |
🟩 CI finished in 1h 38m: Pass: 100%/93 | Total: 2d 00h | Avg: 31m 00s | Max: 1h 16m | Hits: 90%/133890
|
| Project | |
|---|---|
| CCCL Infrastructure | |
| libcu++ | |
| +/- | CUB |
| Thrust | |
| CUDA Experimental | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| CCCL Infrastructure | |
| libcu++ | |
| +/- | CUB |
| +/- | Thrust |
| CUDA Experimental | |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
🏃 Runner counts (total jobs: 93)
| # | Runner |
|---|---|
| 66 | linux-amd64-cpu16 |
| 9 | windows-amd64-cpu16 |
| 6 | linux-amd64-gpu-rtxa6000-latest-1 |
| 4 | linux-arm64-cpu16 |
| 3 | linux-amd64-gpu-h100-latest-1 |
| 3 | linux-amd64-gpu-rtx4090-latest-1 |
| 2 | linux-amd64-gpu-rtx2080-latest-1 |
| else if (wrapped_policy.IsOnesweep()) | ||
| { | ||
| // Regular size | ||
| return InvokeManyTiles<ActivePolicyT>(detail::bool_constant_v<ActivePolicyT::ONESWEEP>); | ||
| return InvokeManyTiles(detail::bool_constant_v<true>, wrapped_policy); | ||
| } | ||
| else | ||
| { | ||
| return InvokeManyTiles(detail::bool_constant_v<false>, wrapped_policy); | ||
| } |
There was a problem hiding this comment.
This will now instantiate and compile both branches, leading to a lot more generated code. Previously, only the onesweep branch would be compiled on newer GPU architectures.
There was a problem hiding this comment.
I added a guard to revert to this if we are not compiling c.parallel
| const int RADIX_BITS = policy.RadixBits(policy.Onesweep()); | ||
| const int RADIX_DIGITS = 1 << RADIX_BITS; | ||
| const int ONESWEEP_ITEMS_PER_THREAD = policy.Onesweep().ItemsPerThread(); | ||
| const int ONESWEEP_BLOCK_THREADS = policy.Onesweep().BlockThreads(); | ||
| const int ONESWEEP_TILE_ITEMS = ONESWEEP_ITEMS_PER_THREAD * ONESWEEP_BLOCK_THREADS; |
There was a problem hiding this comment.
I do not understand why these arent constexpr functions. All of that are compile time values
There was a problem hiding this comment.
Unfortunately, they are only constexpr in CUB. When you instantiate and call the dispatcher from cccl/c/parallel/* (from the python C backend), policy is a type where all the functions return runtime values. That is because the host part (i.e. the dispatcher) is precompiled, but the device part (the kernel) is JIT compiled using NVRTC. Therefore, the CUB dispatchers need to work in both worlds, instantiated with concrete types for use in C++, and instantiated with some type-erased placeholders and runtime values for use with JIT compiled kernels from the Python world. It's a bit unusual, I admit.
There was a problem hiding this comment.
That is a nontrivial pessimization. Would it be possible to handle this selectively through a macro?
There should be no ODR / ABI issues here because it that is not part of the function signature but it would still help the compiler a lot on the C++ side
There was a problem hiding this comment.
That is a nontrivial pessimization. Would it be possible to handle this selectively through a macro?
Wow. So simple. I must have spent too much time in WG21 to see the simple solution. @gevtushenko , is there a specific reason CCCL/C cannot customize CUB using a macro in some cases instead?
Alternatively, for the really critical parts where we discard entire code paths based on a constant expression, we could test whether the value given by the policy is really constant (try to evaluate it at compiletime) and use an if constexpr if it is, and otherwise perform a runtime branch, like:
if constexpr (requires { int a[1+wrapped_policy.IsOnesweep()]; }) { // test if use as a constant expression is valid
// compile-time dispatch
} else {
// runtime dispatch
}There was a problem hiding this comment.
I ended up adding using a macro to decide whether to use compile-time or run-time dispatch for onesweep. We can also use this macro for the kernel launch parameters above, but I think we should leave that for a separate PR because all the other c.parallel algorithms required us to make similar changes to CUB, so we should come up with one solution and use it everywhere.
…y info to avoid unnecessary instantiations
|
/ok to test |
🟩 CI finished in 1h 28m: Pass: 100%/97 | Total: 2d 06h | Avg: 33m 36s | Max: 1h 27m | Hits: 89%/134138
|
| Project | |
|---|---|
| CCCL Infrastructure | |
| libcu++ | |
| +/- | CUB |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| CCCL Infrastructure | |
| libcu++ | |
| +/- | CUB |
| +/- | Thrust |
| CUDA Experimental | |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
🏃 Runner counts (total jobs: 97)
| # | Runner |
|---|---|
| 68 | linux-amd64-cpu16 |
| 9 | windows-amd64-cpu16 |
| 6 | linux-arm64-cpu16 |
| 6 | linux-amd64-gpu-rtxa6000-latest-1 |
| 3 | linux-amd64-gpu-h100-latest-1 |
| 3 | linux-amd64-gpu-rtx4090-latest-1 |
| 2 | linux-amd64-gpu-rtx2080-latest-1 |
…el will use indirect args for those
|
/ok to test |
|
/ok to test |
🟨 CI finished in 5h 57m: Pass: 98%/97 | Total: 2d 02h | Avg: 31m 20s | Max: 1h 13m | Hits: 94%/132841
|
| Project | |
|---|---|
| CCCL Infrastructure | |
| libcu++ | |
| +/- | CUB |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| CCCL Infrastructure | |
| libcu++ | |
| +/- | CUB |
| +/- | Thrust |
| CUDA Experimental | |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
🏃 Runner counts (total jobs: 97)
| # | Runner |
|---|---|
| 68 | linux-amd64-cpu16 |
| 9 | windows-amd64-cpu16 |
| 6 | linux-arm64-cpu16 |
| 6 | linux-amd64-gpu-rtxa6000-latest-1 |
| 3 | linux-amd64-gpu-h100-latest-1 |
| 3 | linux-amd64-gpu-rtx4090-latest-1 |
| 2 | linux-amd64-gpu-rtx2080-latest-1 |
🟩 CI finished in 6h 33m: Pass: 100%/97 | Total: 2d 02h | Avg: 31m 18s | Max: 1h 13m | Hits: 94%/134064
|
| Project | |
|---|---|
| CCCL Infrastructure | |
| libcu++ | |
| +/- | CUB |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| CCCL Infrastructure | |
| libcu++ | |
| +/- | CUB |
| +/- | Thrust |
| CUDA Experimental | |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
🏃 Runner counts (total jobs: 97)
| # | Runner |
|---|---|
| 68 | linux-amd64-cpu16 |
| 9 | windows-amd64-cpu16 |
| 6 | linux-arm64-cpu16 |
| 6 | linux-amd64-gpu-rtxa6000-latest-1 |
| 3 | linux-amd64-gpu-h100-latest-1 |
| 3 | linux-amd64-gpu-rtx4090-latest-1 |
| 2 | linux-amd64-gpu-rtx2080-latest-1 |
🟩 CI finished in 1h 42m: Pass: 100%/101 | Total: 21h 26m | Avg: 12m 44s | Max: 1h 34m | Hits: 95%/140414
|
| Project | |
|---|---|
| CCCL Infrastructure | |
| libcu++ | |
| +/- | CUB |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| CCCL Infrastructure | |
| libcu++ | |
| +/- | CUB |
| +/- | Thrust |
| CUDA Experimental | |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
🏃 Runner counts (total jobs: 101)
| # | Runner |
|---|---|
| 72 | linux-amd64-cpu16 |
| 9 | windows-amd64-cpu16 |
| 6 | linux-arm64-cpu16 |
| 6 | linux-amd64-gpu-rtxa6000-latest-1 |
| 3 | linux-amd64-gpu-h100-latest-1 |
| 3 | linux-amd64-gpu-rtx4090-latest-1 |
| 2 | linux-amd64-gpu-rtx2080-latest-1 |
Description
closes #3797
Checklist