Trust
Benchmarks
Every engine here makes predictions, and a prediction without a validation story is decoration. This page describes how each one is checked and what the checking can and cannot tell you.
HPLC
ground truth
Held-out
evaluation sets
Per-engine
metrics
Versioned
results
01
What we mean by a benchmark
A benchmark here is a comparison between what an engine predicted and what was subsequently measured, on material the engine had no access to when it made the prediction. That last condition is the one that does the work. A model evaluated on designs it was tuned against will look excellent and tell you nothing.
The ground truth is HPLC. Capsaicinoid concentrations in grown material, measured on tissue, converted to a heat rating. It is slower than everything else in the stack by several orders of magnitude, which is exactly why the computational layer exists — and exactly why the computational layer has to be checked against it.
02
What is tracked, engine by engine
Feasibility and ceiling
For flux analysis the questions are whether the feasibility verdict held and whether the predicted ceiling bounded the measured outcome. Tracked as agreement rates and as signed error against HPLC ratings, per host.
Predicted versus measured rating
Predicted SHU against HPLC SHU on held-out material, reported on a log scale because the ladder spans five orders of magnitude. Error at Tier 1 and error at Tier 5 are not comparable quantities.
Capsaicinoid composition
Predicted versus measured proportions across capsaicin, dihydrocapsaicin and the minor species. Composition is often predicted better than total, and the two are tracked separately for that reason.
Ranking quality
ESM-2 scores are evaluated as a ranking problem, not a regression. The relevant question is whether the candidates it put at the top outperformed the ones it did not, measured on variants that were actually built.
Confidence calibration
Whether a model's own confidence estimates track real reliability — whether low-confidence regions do in fact fail more often. Calibration matters more here than average accuracy, because decisions are made per-region.
Edit outcomes
Predicted on-target efficiency and off-target risk compared against sequencing from material that was edited. Cas-OFFinder sweeps are evaluated on whether flagged sites were the ones that mattered.
03
How evaluation is structured
- Ground truth
- HPLC heat rating, grown material
- Protocol
- Held-out — prediction precedes measurement
- Heat metrics
- Log-scale error, interval coverage
- Variant metrics
- Ranking quality on built variants
- Structure metrics
- Confidence calibration
- Stratification
- By tier, by host, by series
- Versioning
- Results pinned to engine versions
Results are stratified rather than averaged into a single figure, because a single figure would hide the thing most worth knowing. Performance in well-populated regions of the catalog is not performance at the sparse edges, and a host with dozens of measured entries is a different proposition from one with two. Aggregate numbers flatter the model; stratified ones are useful.
Every benchmark result is pinned to the engine version that produced the predictions. When an engine is updated the old results stay attached to the old version instead of being quietly restated, which means a comparison between versions is possible at all.
04
What benchmarks cannot tell you
Three limits are worth stating plainly. Measured material is unevenly distributed across the ladder and across hosts, so confidence is uneven in the same pattern. A design at the sparse edge is an extrapolation whatever the aggregate numbers say. Docking scores are ranking hypotheses, not affinities, and should never be read as measurements. And growing conditions contribute real variance to HPLC ratings, which puts a floor under how well any model can agree with them.
The platform’s response to all three is the same: keep measured and predicted values visually distinct everywhere they appear, show where the data is thin, and store every run so a claim can be traced to the computation behind it.
Ready to run it
Try this in the Splice Platform.
CPU engines are free to run. GPU engines run on prepaid credits.
Related