KVRM / comparison lab / October 2, 2026
Better decisions need better comparisons.
Start with measured outcomes, include the stronger controls, and keep the limits beside the numbers. This is a retrospective research replay—not a fresh vendor leaderboard.
586 synthetic cases · 8 domains · no production effects
118 more correct supported decisions than recorded Gemma plus checks.
KVRM selected the expected action in 410/410 supported cases; Gemma proposals with the same domain checks and deterministic runtime reached 292/410. Both had zero false accepts across 176 unsupported cases. The measured difference is useful routing—not an exclusive ability to block invalid actions.
Important: KVRM and the controls use explicit constraints; the archived model prompts did not contain those full executable specifications. Model outputs were not regenerated today. This does not establish superiority over Jev, current frontier models, OPA or production deployments.
The scorecard—including the stronger baselines
| Decision path | Correct supported routing | False accepts |
|---|---|---|
| KVRM hybridRerun; optional learned model disabled | 410/410 · 100.0% | 0/176 · 0.0% |
| Constraint scan + shared checksMinimal no-learning control, not OPA | 379/410 · 92.4% | 0/176 · 0.0% |
| Gemma4 e2b + shared checksRecorded proposals; validation rerun | 292/410 · 71.2% | 0/176 · 0.0% |
| Qwen3.5 0.8b + shared checksRecorded proposals; validation rerun | 224/410 · 54.6% | 0/176 · 0.0% |
| Gemma4 e2b aloneArchived April 2026 responses | 270/410 · 65.9% | 152/176 · 86.4% |
| Qwen3.5 0.8b aloneArchived April 2026 responses | 200/410 · 48.8% | 176/176 · 100.0% |
The control matters: shared checks remove false accepts from both model paths. A minimal constraint scan gets 379/410 correct and ties KVRM in 6/8 domains. KVRM adds 31 correct cases across customer support and content moderation in this replay. This is not proof that a well-engineered rules system cannot match it.
What ran—and what did not
The runner aligns every output by domain and case ID, checks labels and support flags, and recomputes counts from per-case results. Model-only arms use archived April responses. The other four arms run against public revision 85fe4dc8a4f5a7010dd19f98bd8a934554fb8663. Only an in-memory executor is used.
Shared checks include the same domain unsupported-context checks, deterministic validator and fallback path used by KVRM. Confidence thresholds are not tuned on evaluation labels. The constraint scan chooses an action only when exactly one non-fallback action satisfies the constraints; otherwise it uses the validated fallback. It has no training or label access, but shares the support evaluator and domain checks. These controls are not independent OPA or Cedar implementations.
Correct supported routing means the executed action matches the expected action. False accept means final status “executed” on an unsupported-labelled case. Validated handoffs are separate outcomes. Zero observed false accepts is not proof of zero production risk.
These are researcher-authored structured cases. Historical model requests have no input hashes: IDs and labels match, but we cannot independently verify the original payloads were unchanged. Prompts, policy knowledge, run dates and architectures differ. No latency, cost, customer impact or statistical generalization claim follows from this replay.
Per-domain results and ties
| Domain | KVRM | Constraint scan |
|---|---|---|
| soc | 88/88 | 88/88 |
| sre | 94/94 | 94/94 |
| drone | 98/98 | 98/98 |
| grid | 18/18 | 18/18 |
| finance | 18/18 | 18/18 |
| medical | 18/18 | 18/18 |
| customer support | 36/36 | 24/36 |
| content moderation | 40/40 | 21/40 |
Where is Jev in this comparison?
Not measured yet. No Jev responses were generated or substituted with mocks. Jev Choice returns a distribution over options: a fair comparison must include using that distribution with policy constraints, not deliberately throwing alternatives away.
The next head-to-head should compare Jev alone with abstention, Jev plus a policy engine that can select another eligible option, a structured-output LLM plus that policy, KVRM without Jev, and Jev feeding KVRM. Include a strong rules-only implementation. Give every arm the same policy knowledge, facts and held-out cases; keep tuning on development data.
Report correct permitted decisions, false allows, unnecessary review, failure handling, end-to-end p50/p95 latency and actual usage cost. Publish losses and ties. API access and a bounded budget are needed before the Jev run; no vendor-wide superiority claim is currently supported.
Reproduce this replay
git clone --filter=blob:none --no-checkout https://github.com/robertcprice/KVRM.git kvrm-comparison
cd kvrm-comparison
git checkout 85fe4dc8a4f5a7010dd19f98bd8a934554fb8663
python3 -m venv .venv
. .venv/bin/activate
pip install -e ./kvrm-core
curl --fail https://blackweb.ai/downloads/kvrm/compare-decision-stacks.py -o compare-decision-stacks.py
python compare-decision-stacks.py --repo . --output replay.jsonSource is pinned; this command does not pin every dependency version. No API key required. Inspect downloaded code before running it.