One verification batch took 23× as long at a 24-thread limit

Measure the CPU comparison

GPU evaluation includes CPU work. We measured the comparison stage in Nodus' kernel evaluator. For a batch of 1,024 small cases, serial comparison took 0.120486 seconds. With a requested worker cap of 24, the same comparison path took 2.747845 seconds—about 22.81 times as long. The headline rounds that observed ratio to 23×.^[1]

What the timer includes

The diagnostic prepared each cohort once, then reused its input list while requesting worker caps of 1, 4 and 24 in that order. Each case supplied its reference array itself as the output to compare. These were successful reference-output controls: no generated kernel output, model call or GPU execution was involved.^[1][4][6]

The wall timer surrounded the complete comparison call and restoration of the worker-selection helper. In the parallel path, that included creating a fresh ThreadPoolExecutor, mapping the comparison work and waiting for the pool to shut down. The serial path used direct iteration. Fixture preparation and its input digest were outside the comparison timer. These numbers therefore describe the whole comparison call with its scheduling overhead, not a reused pool's steady-state throughput.^[1][4][6]

All nine observations

All nine observations follow. Each cell is one call; cases inside a batch are work items, not independent timing repetitions.^[1]

• * The eight-case cohort used a maximum of eight workers when 24 were requested. These are configured pool caps, not measurements of simultaneous thread activity or physical cores.
• Every comparison returned the expected successful verdict using the reference arrays themselves. This is a control of the comparison path, not a new kernel-correctness result.

The workload matters

The large-case cohort is a useful counterpoint. Four workers reduced its observed comparison time from 2.691225 to 1.803929 seconds. Requesting 24 raised it to 5.176855 seconds, but that cohort contained only eight cases: the harness capped the pool at eight workers. The tiny-case slowdown alone would miss this workload dependence.^[1]

This gives us a concrete reason to measure the CPU comparison stage with representative case sizes, batch sizes and pool lifetimes. A worker count that helps one batch can increase elapsed time for another. It does not identify a generally optimal worker policy or tell us how much an entire evaluation run would improve after a change.^[1]

What this diagnostic cannot establish

There was one observation per setting, in a fixed order, with no dedicated warmup or randomized repetitions. The cohorts ran in the same process, leaving cache and memory history as potential confounds. The container was configured with a 24-CPU quota and reported an affinity mask with 24 logical CPU IDs; that is not evidence of 24 physical cores. Underlying numerical-library thread settings and other host load were not recorded, so their configuration and effect are unknown.^[1]

The records do not establish whether the slowdown came from pool lifecycle overhead, contention, the GIL, numerical-library behavior or another effect. Those are possible explanations to investigate, not measured causes. This retained engineering diagnostic does not establish GPU speed, production throughput or model-training benefit.^[1]

Inspect the saved observations

The numerical record includes all nine observations, prepared-input digests and measurement scope. Its saved values reproduce the arithmetic. The private evaluator dependency closure is required to replay the original timings and is not released here.^[1][2][3]