Skip to content

Add pytest-benchmarks for cuda_parallel - #4357

Merged
shwina merged 2 commits into
NVIDIA:mainfrom
shwina:pytest-benchmarks
Apr 8, 2025
Merged

shwina merged 2 commits into
NVIDIA:mainfrom
shwina:pytest-benchmarks

Conversation

@shwina

@shwina shwina commented Apr 7, 2025 •

Copy link
Copy Markdown
Contributor

Description

Closes #4338

This PR adds a benchmarking script that measures the time taken for compiling/running cuda.parallel algorithms. It uses pytest-benchmark and can be run using:

$ pytest python/cuda_parallel/benchmarks/

The output looks like:

benchmarks/bench_cuda_parallel.py ................                                                                                                                                                                                                               [100%]


-------------------------------------------------------------------------------------------------------------- benchmark: 16 tests ---------------------------------------------------------------------------------------------------------------
Name (time in us)                                 Min                       Max                      Mean                  StdDev                    Median                     IQR            Outliers          OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
bench_reduce_iterator[10000]                  25.8586 (1.0)            233.2758 (1.92)            28.4497 (1.0)           13.7344 (1.0)             26.6053 (1.0)            0.8219 (2.16)         4;43  35,149.7240 (1.0)         256           1
bench_reduce_iterator[100000]                 28.3429 (1.10)        19,925.5217 (163.83)          31.2841 (1.10)         156.1783 (11.37)           29.1239 (1.09)           0.3814 (1.0)        8;1431  31,965.1410 (0.91)      16358           1
bench_reduce_pointer[100000]                  32.4203 (1.25)         2,231.3180 (18.35)           47.8151 (1.68)          42.4934 (3.09)            42.8949 (1.61)          16.8057 (44.07)       15;15  20,913.9118 (0.59)       8326           1
bench_reduce_pointer[10000]                   47.4523 (1.84)           121.6261 (1.0)             65.1374 (2.29)          31.8549 (2.32)            50.0162 (1.88)          25.7470 (67.51)         1;1  15,352.1524 (0.44)          5           1
bench_reduce_iterator[1000000]                51.9189 (2.01)         1,256.6689 (10.33)           53.8783 (1.89)          15.8784 (1.16)            52.8800 (1.99)           0.4009 (1.05)     112;1239  18,560.3330 (0.53)      11113           1
bench_merge_sort_iterator[10000]              72.7209 (2.81)           389.8563 (3.21)            77.6871 (2.73)          24.1193 (1.76)            73.9233 (2.78)           1.7280 (4.53)         7;20  12,872.1565 (0.37)        263           1
bench_merge_sort_pointer[10000]               84.9599 (3.29)           201.8781 (1.66)           112.0210 (3.94)          50.4748 (3.68)            88.9860 (3.34)          37.6576 (98.74)         1;1   8,926.9001 (0.25)          5           1
bench_reduce_pointer[1000000]                120.0140 (4.64)         1,389.6329 (11.43)          124.9122 (4.39)          34.2337 (2.49)           121.6559 (4.57)           3.7863 (9.93)        45;61   8,005.6241 (0.23)       3145           1
bench_merge_sort_iterator[100000]            456.9977 (17.67)        1,866.3011 (15.34)          467.0846 (16.42)         44.5679 (3.24)           461.0941 (17.33)          2.4529 (6.43)       93;161   2,140.9396 (0.06)       2013           1
bench_merge_sort_pointer[100000]             469.5770 (18.16)        1,657.2610 (13.63)          478.6672 (16.83)         39.0637 (2.84)           473.2024 (17.79)          2.9346 (7.69)       61;247   2,089.1341 (0.06)       1930           1
bench_merge_sort_iterator[1000000]         5,106.3080 (197.47)       5,839.5490 (48.01)        5,165.9092 (181.58)        94.8120 (6.90)         5,126.5850 (192.69)       101.0341 (264.92)        5;2     193.5768 (0.01)        134           1
bench_merge_sort_pointer[1000000]          5,265.3919 (203.62)       6,226.2788 (51.19)        5,334.6825 (187.51)       114.3260 (8.32)         5,291.1460 (198.88)       117.2586 (307.46)        7;2     187.4526 (0.01)        122           1
bench_compile_merge_sort_pointer         868,369.0950 (>1000.0)    871,388.9071 (>1000.0)    869,501.3781 (>1000.0)    1,645.4828 (119.81)     868,746.1321 (>1000.0)    2,264.8590 (>1000.0)       1;0       1.1501 (0.00)          3           1
bench_compile_merge_sort_iterator        886,552.9792 (>1000.0)    900,912.0190 (>1000.0)    892,531.8474 (>1000.0)    7,474.6370 (544.23)     890,130.5441 (>1000.0)   10,769.2798 (>1000.0)       1;0       1.1204 (0.00)          3           1
bench_compile_reduce_iterator            907,706.6532 (>1000.0)    926,061.2330 (>1000.0)    914,671.3087 (>1000.0)    9,945.3458 (724.12)     910,246.0397 (>1000.0)   13,765.9349 (>1000.0)       1;0       1.0933 (0.00)          3           1
bench_compile_reduce_pointer           1,008,807.4780 (>1000.0)  1,281,799.9166 (>1000.0)  1,100,455.5138 (>1000.0)  157,051.8913 (>1000.0)  1,010,759.1469 (>1000.0)  204,744.3290 (>1000.0)       1;0       0.9087 (0.00)          3           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

As an aside, this PR introduces a .cache_clear() method to the algorithms that will clear the cached build objects.

Checklist

  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@shwina
shwina requested review from a team as code owners April 7, 2025 16:16
@shwina
shwina requested review from gonidelis and leofang April 7, 2025 16:16
@github-project-automation github-project-automation Bot moved this to Todo in CCCL Apr 7, 2025
@cccl-authenticator-app cccl-authenticator-app Bot moved this from Todo to In Review in CCCL Apr 7, 2025

@leofang leofang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW I always find pytest-benchmark outputs confusing, but I am sure it is just a me problem... 😆

@github-actions

github-actions Bot commented Apr 7, 2025

Copy link
Copy Markdown
Contributor
🟩 CI finished in 1h 28m: Pass: 100%/1 | Total: 1h 28m | Avg: 1h 28m | Max: 1h 28m
  • 🟩 python: Pass: 100%/1 | Total: 1h 28m | Avg: 1h 28m | Max: 1h 28m

    🟩 cpu
      🟩 amd64              Pass: 100%/1   | Total:  1h 28m | Avg:  1h 28m | Max:  1h 28m
    🟩 ctk
      🟩 12.8               Pass: 100%/1   | Total:  1h 28m | Avg:  1h 28m | Max:  1h 28m
    🟩 cudacxx
      🟩 nvcc12.8           Pass: 100%/1   | Total:  1h 28m | Avg:  1h 28m | Max:  1h 28m
    🟩 cudacxx_family
      🟩 nvcc               Pass: 100%/1   | Total:  1h 28m | Avg:  1h 28m | Max:  1h 28m
    🟩 cxx
      🟩 GCC13              Pass: 100%/1   | Total:  1h 28m | Avg:  1h 28m | Max:  1h 28m
    🟩 cxx_family
      🟩 GCC                Pass: 100%/1   | Total:  1h 28m | Avg:  1h 28m | Max:  1h 28m
    🟩 gpu
      🟩 rtx2080            Pass: 100%/1   | Total:  1h 28m | Avg:  1h 28m | Max:  1h 28m
    🟩 jobs
      🟩 Test               Pass: 100%/1   | Total:  1h 28m | Avg:  1h 28m | Max:  1h 28m
    

👃 Inspect Changes

Modifications in project?

Project
CCCL Infrastructure
libcu++
CUB
Thrust
CUDA Experimental
stdpar
+/- python
CCCL C Parallel Library
Catch2Helper

Modifications in project or dependencies?

Project
CCCL Infrastructure
libcu++
CUB
Thrust
CUDA Experimental
stdpar
+/- python
CCCL C Parallel Library
Catch2Helper

🏃‍ Runner counts (total jobs: 1)

# Runner
1 linux-amd64-gpu-rtx2080-latest-1

@shwina shwina moved this from In Review to In Progress in CCCL Apr 8, 2025
@shwina shwina moved this from In Progress to In Review in CCCL Apr 8, 2025
@shwina
shwina force-pushed the pytest-benchmarks branch from ae5b5a7 to 66bda8f Compare April 8, 2025 12:33
@github-actions

github-actions Bot commented Apr 8, 2025

Copy link
Copy Markdown
Contributor
🟩 CI finished in 1h 33m: Pass: 100%/1 | Total: 1h 33m | Avg: 1h 33m | Max: 1h 33m
  • 🟩 python: Pass: 100%/1 | Total: 1h 33m | Avg: 1h 33m | Max: 1h 33m

    🟩 cpu
      🟩 amd64              Pass: 100%/1   | Total:  1h 33m | Avg:  1h 33m | Max:  1h 33m
    🟩 ctk
      🟩 12.8               Pass: 100%/1   | Total:  1h 33m | Avg:  1h 33m | Max:  1h 33m
    🟩 cudacxx
      🟩 nvcc12.8           Pass: 100%/1   | Total:  1h 33m | Avg:  1h 33m | Max:  1h 33m
    🟩 cudacxx_family
      🟩 nvcc               Pass: 100%/1   | Total:  1h 33m | Avg:  1h 33m | Max:  1h 33m
    🟩 cxx
      🟩 GCC13              Pass: 100%/1   | Total:  1h 33m | Avg:  1h 33m | Max:  1h 33m
    🟩 cxx_family
      🟩 GCC                Pass: 100%/1   | Total:  1h 33m | Avg:  1h 33m | Max:  1h 33m
    🟩 gpu
      🟩 rtx2080            Pass: 100%/1   | Total:  1h 33m | Avg:  1h 33m | Max:  1h 33m
    🟩 jobs
      🟩 Test               Pass: 100%/1   | Total:  1h 33m | Avg:  1h 33m | Max:  1h 33m
    

👃 Inspect Changes

Modifications in project?

Project
CCCL Infrastructure
libcu++
CUB
Thrust
CUDA Experimental
stdpar
+/- python
CCCL C Parallel Library
Catch2Helper

Modifications in project or dependencies?

Project
CCCL Infrastructure
libcu++
CUB
Thrust
CUDA Experimental
stdpar
+/- python
CCCL C Parallel Library
Catch2Helper

🏃‍ Runner counts (total jobs: 1)

# Runner
1 linux-amd64-gpu-rtx2080-latest-1



def bench_merge_sort_iterator(benchmark, size):
benchmark(merge_sort_iterator, size, build_only=False)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am out of my league here but are we sure we want to be maintaining a monolithic file for all the benchmarks? Python benchmarking infra looks simplistic rn as it's maturing but maybe it would be better if we start off by splitting them in separate files. Sth like:

benchmarks
   |---- pybench_util.py (for any common utility code)
   |---- reduce.py (or reduce_bench.py)
   |---- merge_sort.py

If not, maybe just renaming the monolithic file to something more specific atm.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • I split up bench_cuda_parallel.py into bench_reduce.py and bench_merge_sort.py
  • I put all common functionality in a conftest.py

"pytest-xdist",
"cupy-cuda12x",
"typing_extensions",
"pytest-benchmark",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we want pytest-benchmark in the test dep group? maybe start a different dependency group for benchmarking, again for future scaling reasons?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did consider this but it felt like a bit overkill. Also when in doubt, do as RAPIDS does

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oof, you win

)


def bench_compile_merge_sort_iterator(benchmark):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the fourth occurrence of the following structure:

def bench_compile_ALGO_KIND(benchmark):
    def setup():
        # This function is called once before the benchmark runs
        # to set up the environment.
        algorithms.ALGO.cache_clear()

    benchmark.pedantic(
        ALGO_KIND,
        kwargs={"size": 10, "build_only": True},
        rounds=3,
        iterations=1,
        setup=setup,
    )

seems like a common utility that can be placed in some header, like util.py (proposed below) in some form like:

def bench_compile_template(benchmark, ALGO,KIND):
...

or

def bench_compile_template(benchmark, ALGO, ALGO_KIND):
...

so we can avoid code duplication. Again, if these benchmarks are only intended for these 4 sets of algo_kind and there is not expected scaling, feel free to scrap my remarks.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great suggestion - I refactored this pattern into a fixture called compile_benchmark and put it in the conftest.py.

@shwina
shwina requested a review from gonidelis April 8, 2025 19:55

@oleksandr-pavlyk oleksandr-pavlyk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Thanks @shwina

@shwina
shwina enabled auto-merge (squash) April 8, 2025 21:34
@github-actions

github-actions Bot commented Apr 8, 2025

Copy link
Copy Markdown
Contributor
🟩 CI finished in 1h 31m: Pass: 100%/1 | Total: 1h 31m | Avg: 1h 31m | Max: 1h 31m
  • 🟩 python: Pass: 100%/1 | Total: 1h 31m | Avg: 1h 31m | Max: 1h 31m

    🟩 cpu
      🟩 amd64              Pass: 100%/1   | Total:  1h 31m | Avg:  1h 31m | Max:  1h 31m
    🟩 ctk
      🟩 12.8               Pass: 100%/1   | Total:  1h 31m | Avg:  1h 31m | Max:  1h 31m
    🟩 cudacxx
      🟩 nvcc12.8           Pass: 100%/1   | Total:  1h 31m | Avg:  1h 31m | Max:  1h 31m
    🟩 cudacxx_family
      🟩 nvcc               Pass: 100%/1   | Total:  1h 31m | Avg:  1h 31m | Max:  1h 31m
    🟩 cxx
      🟩 GCC13              Pass: 100%/1   | Total:  1h 31m | Avg:  1h 31m | Max:  1h 31m
    🟩 cxx_family
      🟩 GCC                Pass: 100%/1   | Total:  1h 31m | Avg:  1h 31m | Max:  1h 31m
    🟩 gpu
      🟩 rtx2080            Pass: 100%/1   | Total:  1h 31m | Avg:  1h 31m | Max:  1h 31m
    🟩 jobs
      🟩 Test               Pass: 100%/1   | Total:  1h 31m | Avg:  1h 31m | Max:  1h 31m
    

👃 Inspect Changes

Modifications in project?

Project
CCCL Infrastructure
libcu++
CUB
Thrust
CUDA Experimental
stdpar
+/- python
CCCL C Parallel Library
Catch2Helper

Modifications in project or dependencies?

Project
CCCL Infrastructure
libcu++
CUB
Thrust
CUDA Experimental
stdpar
+/- python
CCCL C Parallel Library
Catch2Helper

🏃‍ Runner counts (total jobs: 1)

# Runner
1 linux-amd64-gpu-rtx2080-latest-1

@shwina
shwina merged commit 1936c39 into NVIDIA:main Apr 8, 2025
@github-project-automation github-project-automation Bot moved this from In Review to Done in CCCL Apr 8, 2025
davebayer pushed a commit to davebayer/cccl that referenced this pull request Apr 11, 2025
* Add pytest-benchmarks for cuda_parallel

* Implement review feedback

---------

Co-authored-by: Ashwin Srinath <shwina@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

Add benchmarks for cuda.parallel

4 participants