Checks that lie: how we verify the gates are actually guarding

Start with a concrete one.

Step 25 of our flow has an electromigration gate. Step 33 has a power-budget gate. Both are blocking — if they do not pass, the flow is not supposed to continue.

We ran them against a completely empty directory. Both returned 0. Both passed.

And the last line each of them printed was this:

INCOMPLETE: electromigration was NOT screened — missing authority: per-layer Jmax
INCOMPLETE: total power was NOT compared against anything — missing authority: L19...

They knew they had checked nothing. They said they had checked nothing. But the flow reads the exit code, not the prose. So an empty directory passed two blocking gates.

Nobody was being lazy. Whoever wrote these gates honestly printed "I do not have enough information" — and then classified that as "not a failure", which routed it down the return-0 path. The honesty was in the text and not in the exit.

We call this a check that lies. It is not malicious. It is a mechanism with no way to tell the truth.

Why this is particularly fatal in an AI flow

A human running a flow glances at the output. Seeing "NOT screened", a human stops.

An AI agent does not. It reads the exit code and proceeds to the next step.

Worse: the entire point of what we are building is letting machines take a chip through the flow with nobody watching. And "nobody watching" is only safe if the gates tell the truth. A gate that lies is a small blemish when someone is looking; when nobody is, it is the one place in the chain where a false thing becomes a true one — and every report upstream and downstream faithfully copies that "pass" forward.

That is why we have spent so much effort here. It is not quality control. It is the precondition for the road being walkable at all.

63 × 9: the nine questions we ask of every step

2026-08-20 update: the flow has since grown to 68 steps — five conditional tape-out steps (0.5ic, 15.5ic, 26.5ic, 37.5ic, 37.5ip) were added in August 2026. The census numbers below are left exactly as they were measured, over the 63 steps that existed at the time. Re-running the census over 68 steps gives a different set of numbers, and this post does not claim them.

At the time our flow had 63 steps. In July 2026 we stopped asking "does this step have a test" and started asking the same eight questions of every one of them — later nine. 63 × 9 = 567 cells, each of which needs an answer.

D1　wiring — is the gate actually wired into the flow, or does it merely exist?
D2　falsifiable — can it ever go red? Is there any input that reddens it?
D3　outputs_produced — are the outputs it declares actually produced?
D4　criteria_match — is what it checks the thing it claims to check?
D5　deps_correct — is what it blocks on in the right order?
D6　skip_discipline — when may it skip, and does it say so when it does?
D7　outputs_list_complete — is the declared output list missing anything?
D8　missing_caught — when something is absent, does it notice?
D9　content — does it read the content, or only confirm a file exists?

D9 came last and is the hardest. The first eight ask "is this gate properly connected and does it move"; D9 asks "did it actually look inside". A gate can be green on D1 through D8 and then fail D9 because it only ever confirmed the file was present and never read a byte of it.

There is a wrong turn of ours here worth recording. I first designed D9 to compare against known-correct answers — that is, to require an oracle. The owner sent it straight back:

Of course we can use an oracle while we are training and converging ourselves. But when it is actually running, where does an oracle come from?

He was right. A real project does not ship with an answer key. So D9 was rewritten as five oracle-free criteria: can the report be self-consistent, can it be cross-checked against another artefact, can it be bound to the spec, can it be bounded by physics (for example: no branch of a grid can carry more current than the supply injects), and only then, expert judgement. After the rewrite the number of reviewable cells went from 22 to 63.

Twelve shapes of lying

After enough collisions you notice the ways a gate lies are a finite set, and each leaves a mechanical trace. Here are the twelve we have catalogued:

Says it did not check, exits as if it did. stdout leads with INCOMPLETE / NOT CHECKED; rc is 0.
Passes over a zero denominator. Prints "0 runs", "0 libraries", and returns PASS. A zero is not a result.
Passes on an empty directory. It certifies a project that does not exist.
Neuter its decision and no test dies. It has never had a control.
Make it always say NO and no test dies either. It has no positive control — it is a ban, not a check.
Delete the output it declares and it stays green. We call this RULER-BLIND: the ruler cannot measure the thing it says it measures.
Runs, produces a verdict, and is wired where it can never block. On a dashboard it looks like coverage; it stops nothing.
Its input selector can reach the test fixture tree. Data the harness manufactured, reported as production output.
It reads a report it wrote itself.
Running it dirties the tree it is judging. The instrument perturbs its subject.
Spell the same directory differently and the answer changes. The population depends on how the caller typed the path.
It measures a proxy, not the property. Wanting to know "is this rail actually reached", and measuring "who owns this net".

The twelfth is the hardest to catch and the most common. Every step is correct; it is simply answering a different question.

From collisions to a carpet sweep

The problem is that we found all twelve of those by colliding with them. Fix one, and days later another surfaces.

Then the owner asked a very direct question: "Can't we just sweep every gate and decide up front whether it has this problem?"

We can. And it turned out we already owned most of the parts — twelve detectors running independently, with nobody ever having asked how a single gate scores across all twelve. Which is exactly why we could only find these by collision.

So we wrote a census: take every gate clause the flow declares, run a battery of probes against each, and issue one scorecard per gate.

One thing there matters more than the tool itself, and I want to state it plainly: we first verified that the census tool does not lie.

A detector that has never fired is indistinguishable from no detector. So we calibrated it against the two gates above — in both directions. The fixed versions must read CLEAN (they did). Restore the pre-fix versions and it must catch them (it did, reproducing the original evidence verbatim). Only after both directions did we trust its numbers.

What the first full sweep found

136 gate clauses, four seconds-each probes:

LIAR      18   (15 of them blocking)
SUSPECT   32

That number needs discounting, and the discount has to come from us.

Of the 18, 5 had already been adjudicated, and "passes on an empty directory" is the correct answer for them — a gate checking that a forbidden artefact is absent should pass when nothing is there. Our probe produces false positives on fail-safe gates.

After that discount, 13 have never been adjudicated by anyone, and 10 of those are blocking. Ten gates whose whole purpose is "do not proceed unless this passes" will wave an empty tree through, and nobody ever decided that was correct.

The 32 SUSPECT are all one shape: their input selectors do unbounded directory walks with no guard in the file restricting them to tracked, published artefacts. That does not make them wrong. It makes them capable of picking up fake data with nothing to stop them. That has already bitten us once.

Seven of the twelve shapes are not in the census yet — they need mutation runs and are expensive. So this number can only go up.

Mines we stepped on

A few that actually changed how we work:

A test that passed only because another test quietly repaired the tree first. One test asserts that every skill file carries a particular section. It was green. But the file shipped without that section — an earlier test in the same file had rewritten it mid-run. Deselect just that repairing test and the assertion goes red immediately. So "15 passed" was 14 honest passes and one hitchhiker.

Running the test suite rewrites published evidence. We left a full suite running on a clean tree and checked halfway through: it had overwritten the gate reports of three published cells, rewriting "where this result came from" to point at whatever temporary directory last ran the tests. That path will not exist tomorrow.

A "fetch real data" function that returned the suite's own fixture. It looks for three named paths, and failing that, walks the whole tree. After those three paths were withdrawn for unrelated reasons, the first file the walk found was a fixture the tests had manufactured — and two tests with real in their names asserted properties of production extraction output about it. It was caught by pure luck: that fixture happened to contradict the assertion. A fixture that agreed would have shipped green.

And one of them is mine. I wrote a census program to verify something else, and it returned empty. Fifty-six agents' worth of completed work, discarded by a type error in my own aggregation code. I recovered it from the journal and fixed the aggregator — but the lesson was immediate: the thing that can lie is not only the subject under test. It is also the instrument, including the one I wrote.

An honest close

This census cannot tell you whether a gate's rule is correct. Deciding whether a given DRC rule is the right one for a process needs an expert; that is a different problem.

It answers a narrower question that can be mechanised: is this gate capable of lying? — will it wave through nothing, can it go red at all, does what it says match what it returns.

A gate that passes this census can still be wrong. A gate that fails it cannot be trusted even when its rule is right.

And we are not finished. Seven shapes are not yet in the sweep, the 32 SUSPECT have not been adjudicated one by one, and the ten blocking cases are still sitting there waiting to be repaired. We are publishing the number because it is true, not because it is flattering.

If there is one thing to take from this, it is probably this: once you start letting machines decide things where nobody is watching, the first question is not "did it get it right" but "if it got it wrong, what would tell me". And that thing which would tell you needs verifying too — by making it fire.

Further reading: the 68 steps are on the flow page; the per-cell gate status is on the gate matrix; all the code, including every issue mentioned here, is on GitHub.