V2 · October 2026 · six agent systems · 36 scored runs

V2 results

V2 rebuilt the score after V1's open-source timer turned out to miss whole classes of circuit paths. Every design is now timed by the FPGA vendor's own tool, with memory stalls present in the timed circuit, and ranked on five hidden programs the agents never see. Six agent systems (a model inside its vendor's command-line agent) each ran six times from the same starting core, 15 rounds of 3 attempts per run.

Result Three tiers: Opus 5.5 and Sonnet 5.5 (Claude Code), then GPT-6.1 Sol and GPT-6 Astra (Codex CLI), then GPT-5.5 and GPT-6 Luna (Codex CLI). Pre-registered test, Opus 5.5 vs GPT-6.1 Sol: 1.32x, 95% CI 1.18 to 1.47, p = 0.0005.

Every scored run

Held-out score per run

One dot per scored run (six per system), the bar is the system's geometric mean. Held-out score = the design's maximum clock frequency (Gowin EDA, GW2A-18) times the geometric mean of iterations per cycle on five benchmark programs the agents never saw. Dashed lines: VexRiscv MaxPerf, the best of ten human-designed reference cores on the same flow; the textbook edit (V0 plus an iterative divider); V0, the starting core.

Final designs

Six systems, six runs each

Geometric means over 6 runs; Fmax and LUT4 are medians. Rank interval: 95% bootstrap. Blank cells do not apply. Harness 2.8.2 for every scored run.
System (agent CLI)	Held-out iter/s	Rank interval	CoreMark iter/s	Fmax (MHz)	Area (LUT4)
Opus 5.5 (Claude Code)	7,024	1 to 2	280	107.8	3,161
Sonnet 5.5 (Claude Code)	6,418	1 to 2	259	103.8	4,232
GPT-6.1 Sol (Codex CLI)	5,320	3 to 4	215	90.7	4,716
GPT-6 Astra (Codex CLI)	5,014	3 to 4	201	83.6	5,546
GPT-5.5 (Codex CLI)	3,595	5 to 6	143	63.9	2,948
GPT-6 Luna (Codex CLI)	3,538	5 to 6	137	64.6	2,934
VexRiscv MaxPerf (human, best of ten reference cores)	6,216			87.8
textbook edit (V0 with an iterative divider)	2,645		103.7	46.6	2,541
V0 (every run starts here)	309		12.1	5.4	12,312

The pre-registered comparison, Opus 5.5 against GPT-6.1 Sol, separates: ratio 1.320 [1.183, 1.473], p = 0.0005. Of the 12 further pairs (Holm-corrected), 9 separate. The three that do not are the pairs inside each tier: Opus 5.5 vs Sonnet 5.5, GPT-6.1 Sol vs GPT-6 Astra, and GPT-5.5 vs GPT-6 Luna. Six runs per system resolve differences of about 20%, not the 6 to 10% gaps inside a tier.

Baselines and controls

What the agents beat

Textbook edit: V0's single-cycle divider replaced by a radix-2 iterative one, the first move most agents make (34 of the 35 designs accepted in round 1 change the divider). It passes every gate and scores 2,645 held-out iter/s, 8.6x V0, all of it clock frequency. Every scored run beats it; system means are 1.3x (GPT-6 Luna) to 2.7x (Opus 5.5) its score.
Random-mutation control: No model: 1 to 3 random single-line edits per attempt (swapped operators, changed constants), 135 attempts over 3 runs. 0 accepted: 117 fail formal verification, 12 fail co-simulation against an instruction-set simulator, 6 pass every gate but change nothing the design does. The gates do not hand out improvements to blind edits.
Lessons ablation: GPT-6.1 Sol run six more times without the lessons file (one-line notes an agent writes after each attempt for later rounds). Full vs no lessons: 1.02x, 95% CI 0.91 to 1.14, p = 0.68. Not distinguishable; the interval rules out a gain from the lessons above about 14%.
Human reference cores: Ten open-source RV32IM cores built on the same flow with the same stall model. VexRiscv MaxPerf (6,216) falls between the first and second tiers; the other nine score below every second-tier system.

Robustness

Does the ranking hold?

Place-and-route settings. All final designs rebuilt under the six Gowin placement and routing settings that change a build: the same system order, Opus 5.5 vs GPT-6.1 Sol between 1.28x and 1.33x (every p < 0.001), and the same 9 of 12 separating pairs under every setting.

Unseen programs. On 15 more Embench-IoT programs no agent saw, the system order, the rank intervals and the three indistinguishable pairs are the same; all 740 design-program runs give correct results.

Another FPGA. Every design synthesized for an AMD Artix-7 with Vivado. The split between the top four and the bottom two holds; the order inside the top four does not (Opus 5.5 vs GPT-6.1 Sol: 1.03x, p = 0.80). The agents' designs gain less from the faster fabric than the human cores (Fmax ratio 1.17 vs 1.67), which suggests that fifteen rounds of tuning against one vendor's timing report fit a design to that part. The V2 ranking is a ranking on the Gowin contract.

Held-out score on each FPGA, geometric means over 6 runs; Fmax ratio Artix-7 / Gowin.
System	Gowin GW2A-18	Artix-7 200T	Fmax ratio
Opus 5.5	7,024	6,724	0.96
Sonnet 5.5	6,418	7,511	1.17
GPT-6.1 Sol	5,320	6,543	1.23
GPT-6 Astra	5,014	6,236	1.24
GPT-5.5	3,595	4,385	1.22
GPT-6 Luna	3,538	4,245	1.20
VexRiscv MaxPerf (human)	6,216	10,246	1.65
VexRiscv NoCache (human)	4,524	8,772	1.94

Correctness

What the gates missed

Three gaps in the correctness checks surfaced during and after scoring; none changes a scored result.

Instruction fetch address. No gate proves that the fetch address stays inside memory. A bounded formal check written after scoring (any 64-word program whose own branches stay in range, arbitrary memory stalls, 20 cycles from reset) passes 36 of 44 designs and finds one failure: an Opus 5.5 design fetches from below address 0 on a mispredicted path, 9 cycles after reset. Its results stay correct. Seven larger designs time out without a counterexample.
Multiply and divide arithmetic. The per-round formal gate replaces multiplier and divider results with stand-in formulas, so co-simulation checks the arithmetic. The vendored formal specification for signed DIV and REM computes the unsigned result, so the contract's full-arithmetic check fails every correct divider.
Formal saw different code. Two final designs gave the formal tool a different version of one block than synthesis and simulation got. Both pass all 53 checks when formal sees the synthesized version.

From V1 to V2

What changed, and what V1 got wrong

V1	V2
Open-source timer (nextpnr): misses whole path classes, a 32-bit divider timed at 123 MHz that the vendor tool times at 6.9 MHz	Vendor timer (Gowin EDA) for scoring and for the agents
Memory stalls simulated but absent from the timed circuit, so stall logic was free	The same stall generator in simulation and in the timed circuit
Scored on CoreMark, the program the agents optimize	Scored on five hidden programs; CoreMark stays the optimization signal
Any improvement accepted	A 4.6% margin, set from measured placement noise
2 to 3 runs per model, best run reported	6 runs per system, pre-registered tests, geometric means
Read access differed by CLI (Codex agents could read anything)	One OS account per concurrent run, the same rules for every CLI, hidden programs unreadable
Random-mutation control: 0 of 135 accepted, but a seeding bug made it 3 draws repeated 45 times	135 independent draws, 0 accepted

The leaderboard on the home page is V1's: its numbers come from the open-source timer and are not comparable with these.

Data

Everything is in the repository

Pre-registration, amendments and analyses (index in its README)
Every run: final RTL, per-attempt log, lessons file, full agent transcripts, git history
Day-by-day research notes, including every incident that stopped a campaign
What V2 does not fix