Your GPU stopwatch can stop before the work does

One workload, three boundaries

Our fictional workload takes 40 microseconds to submit. The host returns from its completion wait at 140 microseconds. A pair of device events spans 100 microseconds. These are invented teaching values, not measurements from a GPU. The difference is where each stopwatch starts and stops.

We built a small Python worksheet that keeps those boundaries separate. It includes eight cases, the complete saved output and fifteen authored edge controls. It runs on a CPU with the standard library; it does not call CUDA or PyTorch. The accompanying CUDA listing is source-only and was not compiled or executed.^[1][2]

Submission is an earlier endpoint

NVIDIA documents asynchronous CUDA calls: control can return to the CPU before the requested device work completes. PyTorch describes the same issue for GPU operations. A host timer around submission can therefore answer a useful question—how long did submission take—without answering when the device finished.^[3][5]

In the worksheet, the 40-microsecond submission interval makes no completion claim. The 140-microsecond host interval ends after a declared wait for the right event, covering both required phases. Remove that wait or point it at another event, and the completion result becomes unknown. A timestamp alone is not a completion marker.^[1]

The 100-microsecond device interval uses completed, timing-enabled events in one declared stream and clock instance. NVIDIA warns that other stream activity can affect event timing. This is not a claim of pure kernel instruction time or isolation from other work. PyTorch event timing must also be enabled explicitly.^[4][6]

Overlapping work does not add up to wall time

Two fictional phases last 80 and 60 microseconds. On their shared device timeline, one runs from 7,000 to 7,080 and the other from 7,040 to 7,100. Their durations sum to 140, but their covered span is 100. Adding them counts the 40-microsecond overlap twice.^[1]

The worksheet reports the sum, covered union and span separately. A gap remains a gap; it does not become active work. This arithmetic is about the declared intervals. It does not recover GPU utilization or a critical path from an incomplete trace. The phase sum matching the host interval in this fixture is a coincidence, not a timing formula.^[1]

Keep clocks attached to timestamps

The host clock begins at zero in our fixture, while the device clock has an arbitrary 7,000 offset. Subtracting one from the other would create a number with no established meaning. The worksheet returns unknown for different clock domains or instance epochs, including two host boots. Matching units do not establish clock alignment.^[1]

The device coordinates are also a teaching device. CUDA event elapsed-time APIs provide a duration; the example does not give you an absolute GPU timestamp that can be aligned with a CPU timer. Even the difference between two valid durations does not, by itself, identify a causal overhead component.^[4]

Write down what the timer excludes

The annotated CUDA example places initialization, allocation, the input copy and one illustrative warmup before the timers. It places the result copy, validation and cleanup after them. Its host submission timer includes event-record calls and launch/error-check overhead; its completion timer adds the wait. One warmup is not evidence of steady state.^[1]

The listing uses a specific event wait for its one-stream workload. A device-wide synchronization has a broader scope: PyTorch documents it as waiting for kernels in all streams on the selected device. Choose that scope deliberately, and retain failures and repeated samples when conducting an actual experiment. This package has no actual latency, cost or training-benefit result.^[7]

Download the worksheet, fictional inputs and saved result, then use the blank boundary form to specify your own start, stop, stream, clock and inclusion rules. A timing number is easier to compare when those choices travel with it.^[1][2]