PhysDesignBench
GitHub

PhysDesignBench

Can AI agents go from a specification to a validated design?

30 tasks in electromagnetic, photonic and multiphysics engineering. Agents use real simulators under a fixed budget; an independent evaluator rebuilds every submission and checks it against fabrication rules and physical limits.

Explore the leaderboard →Browse the tasks
Mean normalized scoreOverall · % · 95% CI
  1. 1GPT-6 AstraCodex47.5plus or minus ±9.0
  2. 2Opus 5.5Claude Code39.9plus or minus ±8.8
  3. 3GPT-6.1 SolCodex39.5plus or minus ±9.4
  4. 4Fable 5.1Claude Code33.5plus or minus ±8.4
  5. 5Sonnet 5.5 (high)Claude Code32.2plus or minus ±8.6
  6. 6Gemini 4 Argonmini-SWE-agent25.2plus or minus ±9.0
  7. 7Muse Wonder (high)Muse Code24.2plus or minus ±8.9
  8. 8GPT-5.6 SolCodex23.3plus or minus ±8.1
  9. 9Kimi K3.1 ProClaude Code22.2plus or minus ±8.1
  10. 10GLM 5.3 Flash (max)mini-SWE-agent15.1plus or minus ±7.4
  11. 11GLM 5.3 Flash (high)Claude Code12.5plus or minus ±6.3
Design runs, probe by probe

Watch the designs evolve.

Each frame is the agent's layout at one evaluator probe, drawn at constant physical scale, beside its best valid score so far.

  • Microresonator modulatorClaude Code · Opus 5.5
    ⤢
  • Four-channel WDM channelizerCodex · GPT-5.6 Sol
    ⤢
  • 4×4 interferometer meshCodex · GPT-5.6 Sol
    ⤢
  • 6×6 interferometer meshClaude Code · Opus 5.5
    ⤢
  • MZI-coupled ring filterClaude Code · Opus 5.5
    ⤢
  • Cascaded MZI-ring filterClaude Code · Opus 5.5
    ⤢
  • Cascaded MZI-ring filterCodex · GPT-5.6 Sol
    ⤢
  • Four-mode quantum projectorCodex · GPT-5.6 Sol
    ⤢
  • Eight-mode quantum projectorCodex · GPT-5.6 Sol
    ⤢
  • Microring Fourier spectrometerCodex · GPT-5.6 Sol
    ⤢
  • Serial dual-ring MZI filterClaude Code · Opus 5.5
    ⤢
  • Ku-band waveguide filterCodex · GPT-6.1 Sol · RF preview
    ⤢
  • Dual-band waveguide filterCodex · GPT-6.1 Sol · RF preview
    ⤢
How a task works

From specification to a rebuilt, scored design.

Read the method →
  1. 1

    Specification

    A function, physical constraints and an objective J (lower is better).

  2. 2

    Agent designs

    Edits code, runs domain simulators and checks its own work for up to 120 minutes.

  3. 3

    Evaluator probes

    Up to 40 (Design) or 30 (Reimplementation) scored checks of intermediate candidates.

  4. 4

    Independent rebuild

    Only the design crosses over. A fresh container rebuilds it and applies every gate: DRC, LVS, electrical, thermal, numerical.

  5. 5

    Score

    S = 1 / (1 + J / c). A design that fails any gate scores 0; S = 0.5 matches the reference.

What we found

Valid designs are the bottleneck.

See every outcome →

2–16of 16

Design tasks with an eligible design

Depending on the configuration. Even the strongest miss some: only GPT-6 Astra found an eligible design on every Design task.

1–3of 10

Reimplementations that beat the paper

Every Reimplementation submission is valid, but agents rarely recover the performance the original authors published.

$5vs $11 per task

Similar scores, different costs

GPT-6.1 Sol and Opus 5.5 are 0.4 points apart overall, yet cost $5 and $11 per task and take 120 and 92 minutes.

Questions

Reading the results.

Why are Design scores so low?

Most of the gap is validity, not polish. A design must pass up to about 50 gates (design rules, connectivity, electrical and thermal limits, numerical checks) before it earns any score. Configurations found an eligible design on between 2 and 16 of the 16 Design tasks.

Can agents game the evaluator?

Agents never grade their own work. The workspace and the evaluator run in separate containers; only the design file crosses over. Metrics, spectra or files the agent writes are ignored, and the score is recomputed from a fresh rebuild bound to the submission's hash.

What does S = 0.5 mean?

S = 1 / (1 + J / c). For Reimplementation, c is the published device's objective, so S = 0.5 matches the paper and S > 0.5 beats it. For Design, c is a fixed engineering scale per task.

Is every task solvable?

Yes. Every Design task has an expert reference design that passes all gates, and every Reimplementation task is anchored to a published device. GPT-6 Astra also produced an eligible design on all 26 tasks.

Which score is on the leaderboard?

The same as Table 3 of the paper: the best eligible evaluated candidate for Design tasks and the final submission for Reimplementation, with invalid outcomes scored zero. The leaderboard also shows a final-submission-only score.