Benchmarking honestly
Fair baselines and reproducible runs
What to measure and how to report it so others can trust the comparison.
6 minute read
Measure the right things
- Wall-clock time to a stated quality, for example “time to reach a 1% optimality gap”, not just total runtime.
- Solution quality with a named metric, and whether lower or higher is better.
- Everything counted. Include queue time, embedding or compilation, and classical pre- and post-processing for hybrid runs.
- Hardware and settings for every run: cores, solver version, time limits, parameters.
Use a strong baseline
Compare against a well-tuned classical solver, ideally the one a domain expert would actually use. If you can, run several random seeds and report the spread, not only the best run.
Make it reproducible
Attach code, a dataset or a synthetic generator, and a Super notebook link. Tick “Re-run on Super” when results were reproduced there. When someone else repeats your run, they can post a replication report in the comments.
You can attest a benchmark on Nera. A fingerprint of the results is recorded with a timestamp, so later changes are detectable. The data itself is not published.
Key points
- Report time to a stated quality, with full overheads.
- Beat a strong, tuned classical baseline.
- Share code, data or generators, and invite replication.
Check your understanding
Answer the questions above to finish.