More seeds won't test a parameter you never vary

The parameter matters more than another seed

This article is based on one retained CPU experiment and a separate internal review of its source and saved artifacts.

We ran 504 checks across 18 synthetic normalization configurations. Randomizing only the weights left every omitted-bias formula accepted. Randomizing only the bias left every omitted-weight formula accepted. Both varied together exposed all six selected altered formulas in this grid.^[1]

That is a specific coverage problem. If bias stays zero, removing the bias term produces the same value on that slice. More random x tensors, shapes or seeds do not restore the missing distinction. Acceptance there does not mean the output is wrong for that particular fixture; it means the test cannot establish behavior outside it.^[2]

What changed from our earlier reproduction

The starting point is zzw4257's original report in NVIDIA SOL-ExecBench issue #25. Our earlier follow-up reproduced an affine-input counterexample on CPU and, separately, one retained T4 fixture. It varied weights and bias together. Those results did not isolate which input dimension exposed which formula.^[4][5][6]

This new ablation separates four regimes: identity parameters, weight only, bias only, and both varied. For each configuration, we draw x, varied weights and varied bias once and reuse those exact tensors across the four regimes. The full source and retained tensor bytes are included.^[2][1]

The complete matrix

The grid has three shapes (1×8, 4×64 and 4×127), two dtypes (float32 and float16), and three seeds (7, 42 and 2026). That makes 18 correlated configurations. Each has four regimes and seven outputs: six altered formulas plus a copied-reference control. All 504 planned observations completed; none were missing or errored.^[1]

The figure reports altered-formula acceptances out of 18 configurations per cell. The complete totals are:^[1]

• Identity parameters: 108 of 108 altered-formula observations accepted.
• Weight only: 18 of 108 accepted, all from omitting bias.
• Bias only: 18 of 108 accepted, all from omitting weight.
• Both varied: 0 of 108 accepted; all six altered formulas rejected in all 18 configurations.
• Copied-reference controls: 72 of 72 accepted across the four regimes.

What acceptance means here

The reference is an eager layer_norm operation with epsilon 1e-5, followed by multiplication and addition in the selected dtype. We do not assert that this sequence is bit-identical to a fused affine LayerNorm kernel or an independent real-arithmetic oracle. The positive control is reference.clone(), which checks transport and checker consistency rather than an independently implemented correct kernel.^[2]

We used the unchanged public compute_error_stats helper at SOL-ExecBench commit a9fa0804c793d438e70850c33fe34426e66d53dd. It converts to float32 and applies the fixed 0.01 absolute / 0.01 relative tolerance with at least 99% matched elements, its nonfinite/all-zero checks, and no separate maximum-error cap. This is not ordinary torch.allclose. We did not tune tolerances or select new seeds after seeing the result.^[7][2]

Reproduce the result, keep its scope

The observed cohort ran locally on macOS arm64 with CPython 3.12.14, Torch 2.14.1 and Pydantic 2.12.5. It is CPU evidence. The package contains the unchanged runner and protocol, exact public checker import bundle and notices, complete plan/events/result, and portable JSON containing all 504 observations.^[2][1][3]

Mac package-version constraints are recorded, but they are not a wheel-hash lock or a claim of identical binary builds. An optional pinned Linux Torch 2.9.0+cu130 runtime is documented as a separate reproduction cohort. No Linux or CUDA execution of this 504-slot grid is included, and the earlier T4 fixture does not confirm it.^[3]

The six formulas are deliberately selected examples in one exposed development normalization family. Shapes, dtypes and seeds are not independent tasks. This does not measure benchmark-wide defect prevalence, exhaustive correctness, held-out transfer, kernel performance or model-training improvement.^[1]

Why this matters for kernel data

A feedback record can preserve the wrong distinction if its test inputs never exercise the relevant parameter. This experiment gives a concrete example and a reproducible way to inspect it. It does not establish that training on the resulting feedback improves a model; that needs a separate, matched data-utility study.^[1]

For a lab evaluation, the useful next question is which distinctions the task inputs and feedback actually test. We make the complete scoped evidence public so that question can be checked before interpreting a headline score.^[2]