Repository navigation
Add pytest-benchmarks for cuda_parallel - #4357
Conversation
leofang
left a comment
There was a problem hiding this comment.
FWIW I always find pytest-benchmark outputs confusing, but I am sure it is just a me problem... 😆
🟩 CI finished in 1h 28m: Pass: 100%/1 | Total: 1h 28m | Avg: 1h 28m | Max: 1h 28m
|
| Project | |
|---|---|
| CCCL Infrastructure | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| +/- | python |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| CCCL Infrastructure | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| +/- | python |
| CCCL C Parallel Library | |
| Catch2Helper |
🏃 Runner counts (total jobs: 1)
| # | Runner |
|---|---|
| 1 | linux-amd64-gpu-rtx2080-latest-1 |
ae5b5a7 to
66bda8f
Compare
🟩 CI finished in 1h 33m: Pass: 100%/1 | Total: 1h 33m | Avg: 1h 33m | Max: 1h 33m
|
| Project | |
|---|---|
| CCCL Infrastructure | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| +/- | python |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| CCCL Infrastructure | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| +/- | python |
| CCCL C Parallel Library | |
| Catch2Helper |
🏃 Runner counts (total jobs: 1)
| # | Runner |
|---|---|
| 1 | linux-amd64-gpu-rtx2080-latest-1 |
|
|
||
|
|
||
| def bench_merge_sort_iterator(benchmark, size): | ||
| benchmark(merge_sort_iterator, size, build_only=False) |
There was a problem hiding this comment.
I am out of my league here but are we sure we want to be maintaining a monolithic file for all the benchmarks? Python benchmarking infra looks simplistic rn as it's maturing but maybe it would be better if we start off by splitting them in separate files. Sth like:
benchmarks
|---- pybench_util.py (for any common utility code)
|---- reduce.py (or reduce_bench.py)
|---- merge_sort.py
If not, maybe just renaming the monolithic file to something more specific atm.
There was a problem hiding this comment.
- I split up
bench_cuda_parallel.pyintobench_reduce.pyandbench_merge_sort.py - I put all common functionality in a
conftest.py
| "pytest-xdist", | ||
| "cupy-cuda12x", | ||
| "typing_extensions", | ||
| "pytest-benchmark", |
There was a problem hiding this comment.
do we want pytest-benchmark in the test dep group? maybe start a different dependency group for benchmarking, again for future scaling reasons?
There was a problem hiding this comment.
I did consider this but it felt like a bit overkill. Also when in doubt, do as RAPIDS does
| ) | ||
|
|
||
|
|
||
| def bench_compile_merge_sort_iterator(benchmark): |
There was a problem hiding this comment.
this is the fourth occurrence of the following structure:
def bench_compile_ALGO_KIND(benchmark):
def setup():
# This function is called once before the benchmark runs
# to set up the environment.
algorithms.ALGO.cache_clear()
benchmark.pedantic(
ALGO_KIND,
kwargs={"size": 10, "build_only": True},
rounds=3,
iterations=1,
setup=setup,
)seems like a common utility that can be placed in some header, like util.py (proposed below) in some form like:
def bench_compile_template(benchmark, ALGO,KIND):
...or
def bench_compile_template(benchmark, ALGO, ALGO_KIND):
...so we can avoid code duplication. Again, if these benchmarks are only intended for these 4 sets of algo_kind and there is not expected scaling, feel free to scrap my remarks.
There was a problem hiding this comment.
great suggestion - I refactored this pattern into a fixture called compile_benchmark and put it in the conftest.py.
oleksandr-pavlyk
left a comment
There was a problem hiding this comment.
Looks great! Thanks @shwina
🟩 CI finished in 1h 31m: Pass: 100%/1 | Total: 1h 31m | Avg: 1h 31m | Max: 1h 31m
|
| Project | |
|---|---|
| CCCL Infrastructure | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| +/- | python |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| CCCL Infrastructure | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| +/- | python |
| CCCL C Parallel Library | |
| Catch2Helper |
🏃 Runner counts (total jobs: 1)
| # | Runner |
|---|---|
| 1 | linux-amd64-gpu-rtx2080-latest-1 |
* Add pytest-benchmarks for cuda_parallel * Implement review feedback --------- Co-authored-by: Ashwin Srinath <shwina@users.noreply.github.com>
Description
Closes #4338
This PR adds a benchmarking script that measures the time taken for compiling/running
cuda.parallelalgorithms. It uses pytest-benchmark and can be run using:The output looks like:
As an aside, this PR introduces a
.cache_clear()method to the algorithms that will clear the cached build objects.Checklist