Methodology

Why a CTO should trust our numbers

Anyone can report a good accuracy number. The hard part is making it structurally impossible to report a false one. Our evaluation harness was built so that the most common ways accuracy numbers lie are not policy violations but type errors: the code refuses to compute them.

The nine-stage Closed Loop

The Platform runs the same loop for every vertical and every camera:

Ingest

Pull frames from the Operator's own storage through a scoped read adapter. Sampled cameras only; footage stays where it lives.

Sample

Build the stratified sample across site type, camera geometry, and lighting, so results predict the fleet.

Label

Build ground truth in three layers: automated teacher labels on easy frames (training fuel only, never evaluation), multi-frame consensus, and human-verified labels anchored to the Operator's own operational records. Every label carries a provenance flag and a role flag from the moment it is created.

Fine-tune

Adapt our founder-owned base models per camera cluster.

Evaluate & auto-tier

Score every camera on held-out footage, apply the Stopping Rule when gains flatten, and issue the Tier Decision.

Deploy

Push adapted models through a versioned registry to the Operator's cloud.

Drift-monitor

Watch deployed models through proxy signals (read-rate drop, confidence shift, consensus disagreement, lighting change) and enqueue re-adaptation on breach.

Active learning

Mine the frames where the model is uncertain or disagrees with consensus, and route exactly those to human labeling, so each cycle is cheaper than the last.

Repeat

The loop runs again as the fleet and seasons change — cheaper each cycle.

The honesty gates

Three gates are enforced in code, not in process documents.

GATE 1 / ENFORCED IN CODE

The label-provenance gate

Every label is stamped auto, consensus, human, or billing-anchored at creation. The evaluation harness refuses, in code, to compute a Tier's headline accuracy from auto-generated labels. This kills the classic failure mode of teacher-labeled evaluation: the automated teacher fails on the same hard frames the student fails on, so auto-labels are wrong exactly where the model is wrong, and the score inflates precisely on the cameras that matter. With the gate, that circularity is structurally impossible. Headline numbers on struggling cameras come from human-verified ground truth or they do not exist.

GATE 2 / ENFORCED IN CODE

The role gate

Every label also carries a train or eval role, orthogonal to provenance. Provenance answers “is this label trustworthy enough to score against?”; role answers “was this held out from training?” If one frame ever acquires conflicting roles, the harness raises an error instead of silently letting evaluation data leak into training fuel. Eval leakage, the other classic way to inflate a number, is impossible by construction.

GATE 3 / ENFORCED IN CODE

The deploy-lineage gate

Every model we deploy declares the full lineage of its training data and base weights. The deploy stage refuses lineage that is undeclared, and refuses lineage that carries non-commercial or otherwise incompatible license terms. Much of the computer-vision ecosystem quietly ships models trained on research-only datasets or non-commercial checkpoints; that is a contractual time bomb we refuse to plant in your stack. If we cannot show clean lineage for a model, the platform will not deploy it to you, no matter how good its numbers are.

Everything downstream of these gates is reproducible: seeded runs, pinned commits, exact commands, published in evidence reports our own engineers can re-execute.

Case studies in method

The write-ups below are anonymized summaries of our internal evidence reports. Each one demonstrates how we measure, not how well a product performs. All of them ran on publicly available research-licensed benchmark datasets, which are contractually barred from commercial use. Nothing in this section is a performance claim, commercial evidence, or a promise about your fleet. Commercial results come from one place only: your footage, under contract, against your Sellable Bar. That is what the Tier-Map Audit measures.

Case study 1: proving generalization without letting the eval leak

Question: When a fine-tuned plate reader scores well, is it reading, or did it memorize the answers?
Method: On a public research LPR benchmark, the harness split the data by unique plate string, so no plate string could ever appear in both training and evaluation; the role gate enforces this split in code. Headline accuracy was computed only from trustworthy-provenance, eval-role labels; synthetic training fuel entered as auto-provenance, train-role data and was therefore structurally barred from every headline number by both gates. The report also published its negative controls: configurations where the model demonstrably memorized its training data while held-out accuracy stayed flat, shown side by side with the configuration that genuinely generalized. Held-out decodes were printed verbatim, misses included.
What held: The gates made the memorization and generalization cases cleanly distinguishable, and the report's own caveats section flagged the small evaluation size and a bug in the loop's lift bookkeeping rather than quietly working around them. The bug was then fixed and documented.

Case study 2: four pre-registered A/B experiments, four published negative results

Question: Can purely synthetic, license-clean training data close the gap to real field imagery for plate reading?
Method: The team ran four controlled A/B experiments on synthetic training fuel: three successively more realistic glyph renderers, then a physically ordered capture-chain degradation pass (defocus, motion blur, resolution loss, compression, sensor noise) added to the most realistic renderer. Each experiment changed exactly one variable, ran seeded and reproducible through the full nine-stage loop, and was scored on two independent held-out real-image domains through the provenance and role gates. Critically, the kill criterion was pre-registered: before the final run, the team committed in writing to abandoning this line of work if the result came back negative.
What held: All four experiments came back negative, and all four were published as evidence reports, with a capacity control in each proving the model had genuinely learned its training data, so the negative result was a finding about generalization, not a broken pipeline. The pre-registered kill criterion was honored: the team stopped investing in that lever instead of tuning until something looked publishable.

We tell this story on our website on purpose. A measurement company that buries its negative results is not a measurement company. The discipline that produced and published those four reports is exactly the discipline that produces your Tier Map. That candor is the product.

Case study 3: a leak-proof split, verified on the bytes

Question: In an attribute task (clothing color) where one person appears in many crops, how do you guarantee the evaluation is honest?
Method: On a research pedestrian-attribute benchmark, roles were assigned per subject, not per image, so every crop of a person inherits that person's role and no individual straddles the train/eval boundary. The disjointness was not assumed from the splitting code; it was verified directly on the stored bytes, with exact counts: disjoint key sets, every file claimed by exactly one label, zero orphans. Results were compared against a provable chance bound rather than a flattering baseline, across multiple seeds, and the report explicitly separated the claim that survived the sweep from a loop-reported convenience metric whose baseline term turned out to be unstable. The unstable metric was reported as buggy, then fixed.
What held: Subject-level splitting plus byte-level verification is the same standard your Tier-Map Audit gets: held-out means held out, and “verified” means we ran the check, not that we trust the config file.

Book a Tier-Map Audit What the Audit delivers