Evaluation · fully blind benchmarks + real-silicon sign-off
Every number here is a clean-room blind run — the authoring agent sees only each problem's prompt, and is graded by each benchmark's official upstream testbench. On top of the benchmarks, 8 real ICs are driven end-to-end (spec → GDSII) through real open-source sign-off. Nothing is claimed that isn't measured, and residual misses are documented, never silently dropped.
Open Benchmarks · fully blind · official upstream testbenches
Every score below is a clean-room blind run on the latest plugin: the authoring agent sees only each problem's prompt — never the testbench, golden, or sibling solutions — and is graded by each benchmark's official upstream testbench. Latest runs (2026-07): Claude Fable 5 driving plugin v1.4.81 + the forked vibeic-eda toolchain. Reproducible run logs live in vibeic-bench.
Same harness · different author LLM
The clean-room harness is model-agnostic: the deterministic gate (sole emit path) and the official upstream scorer stay fixed — only the authoring LLM is swapped. Same dataset, same blind rule. This isolates authoring-model quality from harness quality.
| Benchmark · blind · same gate + scorer | Claude Fable 5 | Kimi K3 |
|---|---|---|
| VerilogEval-v2 (156) | 98.08% (153/156) | 94.23% (147/156) |
| VerilogEval-Human (156) | 98.72% (154/156) | 95.51% (149/156) |
Read-out: under the identical Vibe-IC harness, Claude (Fable 5) is currently the stronger RTL author — leading Kimi K3 by ~3–4 points on VerilogEval. The two are near-parity on combinational / single-register problems; the gap opens on hard multi-state FSM depth, where the authoring model carries the load. (Authoring-model comparison — not a change to the headline reference above.)
Plus: CVDP 243/302 = 80.46% official-compliant blind pass@1 on the public no-commercial code-generation set (measured on Opus 4.8 · plugin v1.2.63 — a separate earlier campaign, not the v1.3.88 runs above) — prompt+context-only (the hidden cocotb harness + golden are off-limits oracle, proven byte-identical by a regression guard); and a MetRex premise-dissolving demo — instead of predicting post-synthesis area, Vibe-IC runs MetRex's exact yosys+sky130 recipe and reproduces its golden area to a 1.15% median (90% within 5%). Honest caveats included: residual misses are documented dataset defects or spec ambiguities, never silently dropped.
Across 6 design classes — cryptographic primitive, secure processor, RISCV CPU (Verilog / SystemVerilog / VHDL / SpinalHDL), RISCV SoC, mixed-signal ΔΣ-ADC, and an edge-AI inference accelerator. Every step runs real open-source signoff inside the vibeic-eda container — yosys + OpenROAD + klayout + magic + netgen + ngspice + iverilog+SDF + SymbiYosys.
| IC | PDK | State | Evidence |
|---|---|---|---|
| spm | IHP-SG13G2 | converged | PASS_WITH_WAIVERS · plugin v1.5.58 |
| spm | sky130A | converged | PASS_WITH_WAIVERS · plugin v1.10.18 |
| spm | GF180MCU | converged | PASS_WITH_WAIVERS · plugin v1.9.96 |
| sha256 | sky130A | in progress | no landed evidence yet |
| caravel_user_project | sky130A | landed, not converged | FAIL · plugin v1.9.43 |
| edge_llm_accel | NanGate45 | in progress | no landed evidence yet |
| edge_llm_matmul_accel | NanGate45 | in progress | no landed evidence yet |
| ibex | sky130A | in progress | no landed evidence yet |
| opentitan_aes | sky130A | in progress | no landed evidence yet |
| subservient | sky130A | in progress | no landed evidence yet |
| subservient | GF180MCU | in progress | no landed evidence yet |
| u_hawaii_adc | sky130A | converged | PASS · plugin v1.9.86 |
A cell counts as converged only when its landed evidence says so — the verdict is read from the run's own completion audit, not from a status table. Cells with no landed evidence show as in progress; they are never counted as passing and never removed from the denominator. Updated 2026-08-12 14:17 UTC+08:00.
SHA-256 ×3 variants · SPM
PicoRV32 · CV32E40P · Ibex · SERV
Subservient (SERV-SoC) · NEORV32 (VHDL via GHDL)
U-Hawaii ΔΣ-ADC — A1-A9 + M1-M4 tracks exercised on real ngspice/klayout artefacts
DarkRISCV (BRAM-as-flops over-utilisation) · VexRiscv (SpinalHDL out of scope)
SERV via SymbiYosys + RVFI BMC depth=10 on the real RTL
edge_llm_accel — 1.36M-cell INT4 GEMM engine, the 8th benchmark IC (below)
8th benchmark IC · first run on a NEW PDK — NanGate45 / FreePDK45 (open 45nm)
Prompted by the Kimi K3 “48-hour chip” demo, we designed our OWN same-class chip — a 64×64 weight-stationary INT4 systolic GEMM core (4096 MAC/cycle) with a 20-bank ~195 KB SRAM scratchpad and fused dequantization — and drove it docs→GDS through the standard Vibe-IC front door on the SAME open PDK (NanGate45), the first time this PDK ran in our flow. 3/3 RTL modules generated from the L1–L9 design documents, zero reused IP. Doc-driven dual-track verification caught a real pre-silicon RTL bug (a half-rate weight-load chain that silently degraded the array to 32×64 and leaked state across runs) — fixed and re-proven bit-true before synthesis.
| Metric | Kimi K3 demo | Vibe-IC edge_llm_accel |
|---|---|---|
| Std cells | 1.46M | 1,356,030 + 20 SRAM macros (93%) |
| Design area | 3.981 mm² | 3.07 mm² @ 55% util (die 5.76 mm²) |
| Clock | 100 MHz | 100 MHz MET · SPEF WNS +1.08 ns · TNS 0 |
| Route DRC / antenna | — | 0 / 0 |
| Functional verification | — | 180 random 64×64 tiles bit-true · 9 seeds · 1 real bug caught pre-silicon |
| Wall-clock (docs → GDS, incl. all convergence) | 48 h | ~14 h |
Honest scope: NanGate45 / FreePDK45 is a simulation-grade, non-foundry 45nm enablement (fictional process, no LVS deck, abstract FakeRAM SRAM macros) — so this is synth → macro place → PnR → CTS → detailed-route-DRC-clean → GDS, i.e. “tape-out simulation”, the SAME level as the Kimi demo; real foundry sign-off is demonstrated separately on sky130A / GF180MCU / IHP-SG13G2 / the commercial 180nm PDK above. The educational FreePDK45 KLayout deck reports 23,082 items, every one attributed (15,814 = the deck’s 200 nm well-separation reading vs the library’s abutting-row wells; 7,247 = its simplified flat antenna model where OpenROAD’s hierarchical check reports 0; 20 = exactly the 20 abstract SRAM macros; 1 metal item). Autonomy models differ: Kimi = one LLM iterating alone for 48 h; Vibe-IC = a deterministic runner chain + gated AI convergence — and the run distilled 11 chip-agnostic tool/flow fixes back into the plugin. Full evidence: benchmark-data/ic/edge_llm_accel (RESULT.md) in the public repo. Every figure is measured, not claimed.
Commercial-PDK sign-off · a 180nm NDA foundry PDK · native, no Calibre license
Beyond the open-PDK runs, the same ICs are driven through a real commercial 180nm foundry flow. DRC runs the foundry's own Calibre .rule deck (224 layers / 4533 rules) NATIVELY on the forked KLayout engine (svrfdrc) — no Calibre license. Honest status: two digital ICs reach production-grade sign-off; the analog ΔΣ-ADC reaches real corner-simulation; the crypto core's backend is clean and its only residuals are documented open-source-tool scale floors — never silently dropped.
| IC | Class | GDSII | Sign-off DRC | LVS | STA | Status |
|---|---|---|---|---|---|---|
| spm | SPI peripheral | 2.16 MB | 4533 / 4533 | MATCH | MET · +5.55 ns | Converged |
| subservient | RISC-V SoC (SERV) | 6.3 MB | 4533 / 4533 | MATCH | MET · +2.73 ns | Converged |
| u_hawaii_adc | ΔΣ-ADC + LDO (analog) | ldo 432,726 B · ΔΣ 79,410 B | A6 downstream | — | 9-corner PVT | A8 hardmacro GDS |
| sha256 | Crypto hash | 84–90 MB | 4528 / 4533 | dev+net match | MET · +17.06 ns | Backend green |
The sign-off target is a 180nm commercial NDA foundry PDK. spm & subservient: PASS_WITH_WAIVERS — GDSII + native DRC 4533/4533 + full LVS MATCH (KLayout NetlistComparer + netgen, 0 power shorts) + STA MET; only an FPGA-board hardware waiver remains. u_hawaii_adc: the LDO reaches a full 9-corner TT/SS/FF × (−40/27/125 °C) real-ngspice PVT sweep (Vout ≈ 1.199 V) on BOTH open IHP-SG13G2 and the commercial 180nm PDK; it walls downstream at A6 layout parasitic-verification (real layout pending). sha256: backend is clean (0 router-DRC, STA MET) — the 5 firing sign-off-DRC rules are 4 FEOL over-fires on foundry-qualified std-cell interiors plus 1 metal-density gap; LVS matches device-for-device (108,150) and net-for-net (55,541) with 3 top-level pins a known NetlistComparer artifact; and the single-threaded svrfdrc runtime (~6 h on 90 MB) is a documented open-source-tool scale floor — a spatial-tiling parallelization is in progress. Every figure is measured, not claimed.
Open-PDK Benchmark IC campaign · sky130A / GF180MCU / IHP-SG13G2 / NanGate45 · 2026-07
A wider matrix than the single-PDK results above: the same 9 designs driven across 4 open PDKs. Each (IC × PDK) cell is a distinct, independently-graded result. Status here is re-derived from raw run artifacts, never from a run's own self-report.
| IC × PDK | GDS | Sign-off DRC | LVS | STA | Status |
|---|---|---|---|---|---|
| spm × IHP-SG13G2 | 822,084 B | 592 / 592 | MATCH | MET · +6.14 / +0.17 ns | Converged |
| spm × sky130A | 730,458 B | 145 / 145 | MATCH | MET · +4.56 / +0.33 ns | Converged |
| spm × GF180MCU | 1,180,456 B | 763 / 763 | MATCH | MET · +1.73 / +0.57 ns | Converged |
| sha256 × sky130A | present | — | — | setup gap | Converging |
| caravel_user_project × sky130A | — | — | — | — | Converging |
| edge_llm_accel × NanGate45 | — | — | — | — | Converging |
| edge_llm_matmul_accel × NanGate45 | — | — | — | — | Converging |
| ibex × sky130A | — | — | — | — | Converging |
| opentitan_aes × sky130A | present | — | — | setup gap | Converging |
| subservient × sky130A | — | — | — | — | Converging |
| subservient × GF180MCU | — | — | — | — | Converging |
| u_hawaii_adc × sky130A | analog track | — | — | PVT sweep | Converging |
Progress 2026-07-31 — see text.
3 of 12 cells are independently re-derived PASS as of 2026-07-24: spm × IHP-SG13G2 (DRC 592/592) and spm × sky130A (DRC 145/145) — the same "checked-clean/total" claim strength as the 4533/4533 commercial-PDK number, just a smaller deck (a property of open vs. commercial PDKs, not a thinner check). The other 9 cells remain in active convergence with genuine, documented residuals. Full evidence: benchmark-data/ic/spm/ and benchmark-data/BENCHMARK_IC_CAMPAIGN_STATUS.md in the public repo.
Lower the barrier from decades of training to a conversation.
That column compares Vibe-IC to the traditional commercial flow. For a comparison against the two open projects it gets named next to most often — OpenECOS and OpenROAD — including where Vibe-IC is measurably behind them, see Similar projects.