The reward is part of the dataset

What exactly got better?

An agent writes a kernel that runs faster. That is a useful result. It does not, by itself, show that the training data improved the agent. The gain could come from a different model, a longer search, a weaker baseline, or a different machine. For a lab choosing kernel RL data, those distinctions determine whether a result is useful.

We are building a benchmark program around three separate questions: Can the agent produce correct, fast kernels? Can the environment produce reliable learning signals efficiently? Does training on the resulting data improve performance on work the model has not trained on? This article describes the research design. It reports no new Nodus training-uplift or provider-comparison result.

Use public benchmarks for the question they answer

KernelBench provides a recognizable starting point for evaluating GPU kernel generation from PyTorch workloads. Its fast_p metric combines correctness with a speed threshold relative to a reference. A report should identify the exact task revision, reference implementation, hardware, and generation budget alongside that score.^[1]

FlashInfer-Bench connects kernel definitions, implementations, workloads, and evaluation traces. Its production-oriented structure is useful when the question is whether an optimization matters in an inference workload. We want both broad task coverage and realistic library baselines; a win against one reference does not establish a win against every implementation.^[2]

Correctness deserves its own evaluation. KernelBench-Verified focuses on stronger validation, while BackendBench tests generated backend implementations against PyTorch behavior. These efforts are useful references for judging the evaluator itself, including whether wrong programs can receive attractive scores.^[3][4]

The reward is part of the dataset

For kernel RL, the recorded speedup can become a training reward. If timing conditions change the reward, increasing evaluations per second can increase the volume of unreliable feedback. That is a hypothesis we want to measure, not a claim that every shared GPU produces bad data.

Sharing mechanisms also differ. NVIDIA documents that GPU time-slicing does not provide the memory and fault isolation of MIG, and that requesting more time-sliced replicas does not guarantee proportional compute. An experiment must record the actual resource configuration rather than treat every advertised GPU fraction as equivalent.^[5]

Our proposed experiment fixes a set of candidate and reference kernels, evaluates them in isolation, and replays the same pairs under declared sharing and contention settings. Each shared run is compared with an isolated run on the same GPU and partition profile. A smaller MIG slice is a different target: a ranking change from a full GPU to that slice is not automatically measurement error.

• Verify correctness during timing as well as before it. Incorrect or skipped computation does not count as a speedup.
• Report false speedup labels, candidate-versus-reference ordering agreement, and ambiguous near-ties. Repeated timings of one candidate are not independent tasks.
• Report independently revalidated records per dollar and per elapsed hour, alongside failures, missing results, and accounting coverage.
• Where a foreground service shares the GPU, report its latency and error rate too. Background yield alone does not establish an acceptable configuration.

Idle capacity only counts when useful work survives

A GPU allocation can spend time waiting for compilation, retrying, or redoing work that will be lost at interruption. For intermittent capacity, our proposed unit is a completed, independently validated trajectory or environment artifact. We will count startup, checkpointing, interrupted attempts, and recovery in the cost of producing that unit.

Public cluster traces can help make repeatable workload scenarios. Microsoft’s Philly trace, for example, includes job attempts and utilization observations. Such traces need explicit replay assumptions: minute-scale utilization cannot establish subsecond spare capacity, and observed low utilization does not guarantee free memory or safe colocation.^[6]

A trace replay and a live provider experiment answer different questions. For provider comparisons, we will measure customer-visible time from submission to an accepted artifact and all-in cost for a matched workload. We will not turn those measurements into a claim about a competitor’s internal fleet utilization.

Then test whether the data teaches anything

The learning experiment comes next. Start with the same trainable checkpoint and compare a baseline data mixture with a Nodus-data mixture under a matched training budget. Hold out workload families or source lineages before training, freeze evaluation and candidate-selection rules, and record the full recipe. More shape variants of a training task are not a substitute for transfer to new work.

The outcomes are correctness-qualified performance on the held-out workloads, uncertainty across tasks and training runs, and total cost to reach the stated quality target. We will report regressions and inconclusive outcomes alongside improvements. Environment difficulty and a high rejection rate can help characterize a curriculum; they cannot stand in for this learning comparison.

What we plan to release

The useful public artifact is more than a chart. Each study should include a pinned protocol, the allowed public task and runtime identities, aggregation code, complete attempt accounting, and enough evidence to reproduce the stated claim. Private evaluation tasks can remain sealed, with their scope and limitations disclosed.

We are developing the measurement tools and checking the execution path before running these studies. When we publish measured results, the claim will name the workload, hardware, comparison, and budget it actually covers.

We have also shared an independent CPU reproduction of a reported normalization-input issue in SOL-ExecBench, together with a narrow candidate fix. The original reporter identified the issue; our contribution adds synthetic regression cases and explicit-input controls. GPU validation and maintainer acceptance remain open.^[7]

If your lab trains agents that write GPU kernels, we would like to compare against your evaluation protocol and discuss a scoped Nodus environment or data pilot. The first conversation should establish what improvement would make the data worth using.