Choose the benchmark for the claim

Start with the claim

This guide summarizes inspected public sources and original evaluation design. We did not run these benchmarks or train a model for this article.

A faster kernel is useful evidence. A lab buying training data needs another answer: does using that data improve its model on tasks it has not seen? A team buying GPU capacity needs to know how much correct work finishes within its deadlines. Those questions can share tools, but they need different experiments.

We built a downloadable comparison covering six public benchmarks or toolkits and two proposed study designs. Each row names the input, comparison, correctness gate, metric, denominator and measurement boundary.^[1]

Pick the right layer

For generated kernels, KernelBench compares outputs and runtime with a PyTorch reference. SOL-ExecBench adds a score anchored to an optimized baseline and an analytical hardware limit. The reference and hardware assumptions are part of the result.^[3][8][9]

The name TritonBench is ambiguous. THUNLP's project evaluates model-generated Triton operators. Meta/PyTorch's project benchmarks operator implementations. Naming only TritonBench leaves the reader without a clear evaluation definition.^[7][6]

FlashInfer-Bench organizes kernel definitions, workloads, solutions and evaluation traces. A workload entry can exist without a completed evaluation. MLPerf Inference measures a system under defined request scenarios and quality requirements; a kernel timing is a different measurement.^[10][11][12]

Keep the denominator attached

KernelBench's inspected fastp function counts correct entries whose speed ratio is strictly greater than p, then divides by the supplied n. At p=1, an exactly equal runtime does not count as faster.^[4]

Its analysis script patches missing evaluations as failures, but skips tasks whose baseline entries are missing. The final denominator is the number of aligned entries. Before reporting a whole-cohort result, check that the intended task set and baseline coverage actually match.^[5]

Here is synthetic arithmetic, not a measured result: four tasks have complete baselines; one is correct and 1.5 times faster, one is correct at equal speed, one is incorrect, and one has no result. The strict fast_1 score is 1/4, or 25%. Reporting only the two correct cases would change the question.^[1][4]

The data buyer needs a matched experiment

Our proposed data-utility evaluation compares specified training-data conditions using the same base model and controlled training and inference budgets. It freezes an independent held-out task bank before training and measures the paired change in correctness or correct-and-fast success.^[1]

The split follows source groups. Two failed candidates repaired into one accepted kernel are useful records, but they are not two independent tasks. A training loss decrease or a successful grader run does not establish held-out improvement. No Nodus training uplift is claimed by this guide.^[1]

Capacity needs a lifecycle measurement

For cold start, GPU partitions and idle capacity, our proposed endpoint is accepted work from a fixed offered workload and deadline. Record request, admission, readiness, completion and cleanup boundaries, along with retained state and every failed or unfinished attempt. Cost per accepted unit requires settled costs.^[1]

This is a proposed measurement design. The comparison file does not rank Nodus, Hyperbolic, SF Compute or Prime Intellect. That requires matched measurements on declared targets and supply conditions.^[1]

Use the comparison

Download the source-linked JSON and use its fields to write the claim before choosing a benchmark. The public rows describe reviewed revisions; the two Nodus study-design rows are explicitly marked as proposals. We include the small arithmetic example so its denominator can be checked without a GPU.^[1][2]

For a lab evaluation, specify the model, task family, held-out split, budget and success metric. Those choices define what a data sample can demonstrate.