Getting started

The purpose of these examples is to help demonstrates various transfer and timing techniques.

Transfer techniques

1. Manual cudaMemcpy
2. Manual cudaMemcpyAsync
3. Unified memory
4. Unified memory with cudaMemPrefetchAsync

Timing techniques

1. Chrono
2. CUDA events
3. NVTX markers

Usage

Each test does the follow:

1. Transfers two chunks of data to the GPU.
2. Run simple kernel on each chunk (mutually exclusive)
3. Transfer both chunks back to CPU.
4. Verify results

Throughput based on entire workflow.

Default transfer size is 1GB.

./cudaMemcpyAsync
Running with = 1073741824 B (1.07 GB)
Chrono: 194.463501 ms @ 5.521560 GB/s
Events: 195.145355 ms @ 5.502267 GB/s

or add number of values to transfer

./cudaMemcpyAsync 1000000000
Running with = 4000000000 B (4.00 GB)
Chrono: 725.302307 ms @ 5.514942 GB/s
Events: 726.420593 ms @ 5.506452 GB/s

Using NVTX markers

We must use Nsight Systems to see NVTX. Open *.qdrep file with Nsight Systems GUI.

nsys profile -s none -t cuda,nvtx --stats=true ./cudaMemcpyAsync
WARNING: Backtraces will not be collected because sampling is disabled.
Collecting data...
Running with = 1073741824 B (1.07 GB)
Chrono: 195.130112 ms @ 5.502697 GB/s
Events: 195.151413 ms @ 5.502096 GB/s
Processing events...
Capturing symbol files...
Saving temporary "/tmp/nsys-report-8b68-3d6e-a890-0843.qdstrm" file to disk...
Creating final output files...
Processing [==============================================================100%]
Saved report file to "/tmp/nsys-report-8b68-3d6e-a890-0843.qdrep"
Exporting 1532 events: [==================================================100%]
Exported successfully to
/tmp/nsys-report-8b68-3d6e-a890-0843.sqlite
CUDA API Statistics:
Time(%) Total Time (ns) Num Calls Average Minimum Maximum Name ------- --------------- --------- ------------- ----------- ----------- ------------------------
92.8 7,935,281,846 30 264,509,394.9 162,979,208 367,487,697 cudaStreamSynchronize 3.8 328,299,372 2 164,149,686.0 164,072,205 164,227,167 cudaHostAlloc 1.9 158,486,349 2 79,243,174.5 79,163,730 79,322,619 cudaFreeHost 1.5 130,534,008 2 65,267,004.0 701,703 129,832,305 cudaMalloc 0.0 1,454,937 2 727,468.5 691,604 763,333 cudaFree 0.0 401,077 60 6,684.6 2,374 26,921 cudaMemcpyAsync 0.0 373,116 30 12,437.2 4,637 31,135 cudaLaunchKernel 0.0 83,993 10 8,399.3 5,491 10,170 cudaEventRecord 0.0 18,491 5 3,698.2 3,500 3,931 cudaEventSynchronize 0.0 7,857 2 3,928.5 873 6,984 cudaStreamCreate 0.0 7,470 2 3,735.0 1,390 6,080 cudaStreamDestroy 0.0 5,474 2 2,737.0 397 5,077 cudaEventCreateWithFlags
0.0 2,602 2 1,301.0 417 2,185 cudaEventDestroy CUDA Kernel Statistics:
Time(%) Total Time (ns) Instances Average Minimum Maximum Name ------- --------------- --------- ----------- --------- --------- -----------------------------------------------------------------------
50.4 67,362,653 15 4,490,843.5 4,469,005 4,518,252 void VectorOperation<Add<float>, float>(int, float, float*, Add<float>)
49.6 66,399,104 15 4,426,606.9 4,391,821 4,462,860 void VectorOperation<Sub<float>, float>(int, float, float*, Sub<float>)
CUDA Memory Operation Statistics (by time):
Time(%) Total Time (ns) Operations Average Minimum Maximum Operation ------- --------------- ---------- ------------- ----------- ----------- ------------------
50.8 5,314,309,114 30 177,143,637.1 162,980,699 191,320,991 [CUDA memcpy DtoH]
49.2 5,142,402,501 30 171,413,416.7 168,996,768 173,680,811 [CUDA memcpy HtoD]
CUDA Memory Operation Statistics (by size in KiB):
Total Operations Average Minimum Maximum Operation -------------- ---------- ------------- ------------- ------------- ------------------
31,457,280.000 30 1,048,576.000 1,048,576.000 1,048,576.000 [CUDA memcpy DtoH]
31,457,280.000 30 1,048,576.000 1,048,576.000 1,048,576.000 [CUDA memcpy HtoD]
NVTX Push-Pop Range Statistics:
Time(%) Total Time (ns) Instances Average Minimum Maximum Range ------- --------------- --------- ------------- ----------- ----------- ------------
67.6 3,889,587,729 5 777,917,545.8 777,031,465 778,565,183 Process_Loop
21.7 1,247,130,464 5 249,426,092.8 247,611,878 250,019,880 Verify 10.7 616,467,941 5 123,293,588.2 120,541,180 132,044,582 Reset 0.0 101,082 5 20,216.4 16,613 22,645 H2D_A 0.0 93,874 5 18,774.8 18,020 21,060 Kernel_A 0.0 29,242 5 5,848.4 5,317 6,468 Kernel_B 0.0 23,568 5 4,713.6 4,374 4,970 D2H_A 0.0 18,801 5 3,760.2 3,283 4,147 H2D_B 0.0 14,849 5 2,969.8 2,680 3,182 D2H_B Report file moved to "/home/belt/workStuff/git_examples/transfer_examples/report4.qdrep"
Report file moved to "/home/belt/workStuff/git_examples/transfer_examples/report4.sqlite"