Honesty is the moat: anti-fabrication, deterministic gates, and the holes I fell into
A passing simulation doesn't mean the FPGA works, and it certainly doesn't mean the silicon works. The most dangerous thing AI does isn't getting something wrong — it's telling you, with total confidence, that it got it right. This is the story of the holes I fell into, and why I made honesty structural: no real evidence, no PASS.
The failure mode a software person fears most
After a lot of years writing software, the thing I've learned to fear in AI coding isn't that it gets things wrong. It's that it will almost always tell you "it worked." You ask it to write some code, run the tests, fix a bug, and back comes a tidy summary: "Done, tests passing." Confident tone, clean formatting. Then you open the log and the tests never ran, or the exit code was 0 while the output is full of errors, or it quietly skipped the exact step it was supposed to verify.
In pure software, this kind of hallucinated PASS is annoying but survivable — you run it again, put it through CI, ship it, and it surfaces sooner or later, usually at a cost of minutes to hours. But new as I am to the IC flow, it took me almost no time to see the difference: in chip design, a fake PASS can get someone killed. A mask set costs an astronomical sum; a single tape-out takes months. A passing sim doesn't mean your RTL will work on an FPGA; a working FPGA doesn't mean it will work as silicon. Every layer down, physical reality takes another bite — parasitics, timing, DRC, LVS. If the AI lies to me at any station along the way, I might not find out until the most expensive one.
So I decided early: for this flow to be trustworthy, honesty has to be structural — it can't depend on whether the model is in a good mood today.
The 30/30 that honest re-scoring turned into 0/10
Phase 1 is the entrance to the whole flow: it takes a piece of natural-language intent and unfolds it into layered L1–L27 design JSON, and everything downstream — RTL, simulation, synthesis, sign-off — eats that document. Get this step wrong and everything after it is hollow.
When I first trained the Phase 1 output, the scores looked great — a batch of prompts came back 30/30, all passing. I was rather pleased with myself. Then one day I stopped and asked a simple question: this 30 — who's grading it, and what is it actually measuring?
The answer was embarrassing. That 30/30 was measuring "is the format right, are the fields filled in," not "does the content actually pin the design down." It was a test I set myself, graded myself, and handed myself the marks on. When I switched to an honest standard — really asking "could a downstream engineer who knows nothing get the RTL right from this L-doc?" — and re-scored, it fell from 30/30 to 0/10.
0/10. Not 3, not 5 — zero. It was the most clarifying slap I'd taken on the whole project: all my earlier "progress" had been measuring the wrong thing entirely.
Oddly, hitting zero was a relief, because from there every point was real. The way back up was the dumb, honest way: turn each failure into a rule a program can check. A missing interface-timing definition becomes a coverage-loop rule that FAILs when a required class of field is absent. Two submodules whose port names don't line up become a hard precondition: ports are defined once, in one canonical place, before any RTL. Output that "looks complete" but is a hollow shell becomes a gate: the top level must instantiate everything, no stubs. Phase 1 was rebuilt one point, one gate at a time — and every point now has a program behind it that actually runs and reproduces.
The anti-fabrication doctrine: no real evidence, no PASS
That episode is why I carved a doctrine into the plugin's spec description: anti-fabrication — provenance audit-chain, real-artifact + SHA256 attestation, gate self-assertion, canonical-flow compliance, chip-agnostic source guard — no fabrication, no hallucinated PASS. Broken out, it's a few principles that interlock:
- Deterministic programs, not LLM judgment. Every check is a Python program that emits exactly one of three verdicts:
PASS/PASS_WITH_WAIVERS/FAIL. An LLM saying "looks fine" is never allowed to count as a pass. The judgment can be made by AI; the ruling is always made by a program. - Real artifact + SHA256. If a step claims it produced a file, that file must actually exist, have its own bytes, and be bound by SHA256 — not a hollow shell, and not a symlink pointing elsewhere pretending something is there.
- A gate is not allowed to certify itself. This is the most insidious hole: a step writes its own
{"pass": true}, and a lazy downstream gate reads that boolean as a pass. That's the defendant acting as his own judge. A dedicated check scans the flow definitions and FAILs any gate that trusts a flag the same step wrote. - Canonical-flow compliance. The whole of Phase 2 and Phase 3 isn't "done" because a few key gates ran — every required step must have its output file, and its gate predicate must actually pass.
In one line: to report success, first lay the evidence on the table — and the evidence has to be independently verifiable by a program that doesn't trust you.
These aren't hypotheticals. One concrete example: while forking an open-source LVS tool (netgen), I hit a case where the transistor properties didn't line up at all, yet it happily reported Circuits match uniquely. A silent fake PASS on LVS is exactly the kind that lets a chip whose layout and schematic disagree sail all the way to tape-out — so it got fixed to rule, correctly, that they do NOT match.
The governance hole: 36 FAILs, and the gate was lying
The second hole scared me more, because it wasn't "I don't know how" — it was "my own system lied to me." For a stretch I was running a case that came back FAIL 36 times in a row. I naturally assumed the RTL was wrong, so round after round I changed the RTL, changed the testbench, changed it until I questioned my sanity — and the number wouldn't move.
Then I stopped looking at the design under test and looked instead at the machinery doing the judging. There was the real disease. The problem wasn't in the design; it was in governance. A gate that was supposed to stand guard, when the tool it depended on simply wasn't there, chose to silently tolerate it — the tool wasn't installed, wasn't run, and instead of erroring the gate waved the flow through. So the whole thing marched forward in a state where nothing had actually been verified, and blew up much later somewhere else. All 36 of my fixes were in the wrong place, because the signal the system fed me was fake from the start.
That is what a governance hole is: a seam where a system can report a pass while nothing was really verified. It's far more dangerous than an ordinary bug, because it systematically points you the wrong way. The fix matched the doctrine exactly: turn "silent tolerance" into a hard FAIL. Tool absent means FAIL; the harness has to do a byte-exact self-verify proving it really ran and really compared, not hand back an empty rc=0.
This is the whole case for honesty in one image: a fake PASS steals dozens of rounds spent fixing the wrong thing; an honest FAIL takes you straight to the disease.
Measuring the runner, not the LLM
The same principle settles a subtler honesty problem in benchmarking. The scores we publish measure what the deterministic runner — this product chain — can do, not what a bare LLM with the same tools can finesse out of a lucky prompt.
Take an external benchmark like CVDP. The solver is only allowed to read input.prompt plus input.context; the hidden test harness, the cocotb testbench, and the golden solution are all off-limits oracle. And this isn't enforced by willpower — a regression guard proves it: run with the oracle and run without it, and the emitted bytes are identical. If peeking would change the output, the guard FAILs. Honesty here isn't a slogan; it's a structural invariant. Under that constraint the clean, blind pass rate converges honestly — and every result file records exactly which image and model produced it, so anyone can wipe a machine and reproduce the same number.
Honesty is a moat, not a virtue
If this year left me one sentence, it's this: in a field you don't know, that's long and hard and easy to fudge, honesty is the only thing that actually moves you forward. I'd rather have the truth of 0/10 than the self-deception of 30/30 — because the zero was real ground to build on, and the thirty was sand.
That's why I say honesty is a moat, not a virtue. Anyone can bolt more features onto a flow; what's hard to copy is a flow that refuses to lie to you, one gate at a time. Vibe-IC has iterated from v1.0.0 to v1.4.x across roughly 600 commits, and most of those commits fix an entire class of fake PASS — each one turning a hole I fell into into a rule the flow will never fall into again. A human engineer grows a memory from lessons that hurt; this plugin turns those lessons, one gate at a time, into rules that can't be repeated. That accumulation is the moat.
That's the discipline. Next, I want to carry this "no false PASS" principle all the way onto real hardware — a verification loop running on an actual FPGA board, where the AI programs the SOF itself, reads the LEDs with a camera, and tunes until it converges. That's where you see that "trust" isn't something you say; it's a machine reading a real result and deciding whether to go again.