Mebbian helped an AI score 89% on a public building-plan test, up from 82%
We gave the same AI the same 12 sets of building plans twice. Once on its own, and once with Mebbian’s notes on each set.
The test
Every set of building plans starts with a list of its pages, like a table of contents. Sometimes the list and the pages disagree. A page is on the list but missing from the set, or a page has a different title from the one on the list. Someone who prices the job from the list can miss work.
The test asks an AI to find these mismatches in 12 real sets of plans. It is a public test called AEC-Bench, made by Nomic AI.
How we ran it
- 1The AI took the test on its own.
- 2The same AI took the same test again, with Mebbian’s notes on each set of plans.
- 3The test’s own grader scored both runs. Each set ran 3 times both ways.
Every try
One dot is one try. With Mebbian’s notes, the AI got two more sets right in every try. The other ten sets scored the same both ways.
What it means
When Mebbian finds a mismatch, the AI reports it every time. On the two sets where Mebbian missed the problem, the score stayed the same. The gain comes from what Mebbian catches.
The same problems in real plans
These plans went out to real bidders. Each one has the same kind of problem the test looks for.
- Reviewer-led studyThree rooms point to plumbing sheet P2.20. The set stops at P2.19.UT Austin KIN North Tower renovation · Austin, TexasRead the review →
- Bid Engine reviewHow much wind should the roof supports resist?Harmony Irving gym and storm shelter · Irving, TexasRead the review →
- Reviewer-led studyThe fresh-air setting says “#REF!” instead of a number.Corpus Christi art museum · Corpus Christi, TexasRead the review →
- Reviewer-led studyThe pump note says 208 volts. The power supply says 480.Lockhart Aquatic Facility · Lockhart, TexasRead the review →
- Reviewer-led studyThe lighting legend points to a fixture schedule. The sheet has none.San Antonio Main Plaza renovations · San Antonio, TexasRead the review →
- Reviewer-led studyThe drawings say traction elevators. The specification says hydraulic.Galveston public housing rehab · Galveston, TexasRead the review →
- Reviewer-led studyThe contractor buys the oven. No sheet says which oven.Joplin community supervision center · Joplin, MissouriRead the review →
Each review states how it was made. The Harmony review came from Mebbian’s Bid Engine. The other six were led by a Mebbian reviewer with AI assistance. All drawings come from public bid documents.
Questions
What is AEC-Bench?
AEC-Bench is a public test for AI on real building plans. Nomic AI made it. Each task gives the AI a real set of plans and one job, and the test grades the answer the same way every time.
What did the test ask?
The AI had to find where a set of plans and its list of pages disagree. For example, a page is on the list but missing from the set, or a page has a different title from the one on the list.
How much did Mebbian help?
The same AI scored 81.9% on its own and 88.9% with Mebbian's notes. It ran 12 sets of plans, 3 times each, both ways.
Did Mebbian change the test?
The plans, the task and the grading are the same as the published test. Mebbian adds one thing: its notes on each set of plans.
Can someone check this?
Yes. The test is public, so anyone can run the AI on its own. The full results for every try are available from contact@mebbian.ai.
Technical details for reviewers
Setup
Task family intradrawing/sheet-index-consistency. Graded by the benchmark’s own verifier: a planted problem counts as found when a line of the answer contains one of its keywords, and the set score is the share found.
| Item | AI alone | With Mebbian |
|---|---|---|
| AI | One AI model | Same model |
| Review made by | None | Mebbian's engine |
| Agent | The benchmark's own agent | Same agent, plus two additions |
| Instructions | The task's own instructions | Same, plus one paragraph |
| Extra input | None | Mebbian's review of the same drawing set, as one file |
| Grader | The benchmark's own grader | Same |
| Tries | 3 per set, 36 in total | 3 per set, 36 in total |
The Mebbian run adds one short paragraph to the task instructions. It tells the model that Mebbian has already reviewed the drawing set, that the review can misread text, and that the model must confirm each issue against the PDF before it reports it.
Metrics
| Set | AI alone | Alone, mean | With Mebbian | Mebbian, mean |
|---|---|---|---|---|
| 1bb649-mechanical | 0.5, 0.5, 0.5 | 0.500 | 1, 1, 1 | 1.000 |
| 3908-bid-set-prints | 1, 1, 1 | 1.000 | 1, 1, 1 | 1.000 |
| bidset-stadium | 1, 1, 1 | 1.000 | 1, 1, 1 | 1.000 |
| broughton-meadow-habitat | 1, 1, 1 | 1.000 | 1, 1, 1 | 1.000 |
| darrington-library | 1, 1, 1 | 1.000 | 1, 1, 1 | 1.000 |
| h59-chiller-replacement | 0, 1, 1 | 0.667 | 1, 0, 1 | 0.667 |
| kenai-rec-center | 1, 1, 1 | 1.000 | 1, 1, 1 | 1.000 |
| mechanical-set-2022 | 1, 1, 1 | 1.000 | 1, 1, 1 | 1.000 |
| rces-modular-bid | 1, 1, 1 | 1.000 | 1, 1, 1 | 1.000 |
| reid-hall-permit-bid | 1, 0.5, 0.5 | 0.667 | 1, 1, 1 | 1.000 |
| saluda-hall-generator | 1, 1, 1 | 1.000 | 1, 1, 1 | 1.000 |
| sta-cooling-tower | 0, 0, 0 | 0.000 | 0, 0, 0 | 0.000 |
| Sum of means | 9.833 | 10.667 | ||
| Score (sum ÷ 12) | 81.9% | 88.9% |
Rerun
Benchmark base commit 3f50ae8, Harbor 0.3.0. Requires Docker and an API key for the test model.
git clone https://github.com/nomic-ai/aec-bench cd aec-bench uv run harbor trials start \ -p tasks/intradrawing/sheet-index-consistency/<task> \ --agent-import-path <agent> \ -m <model-id>
Each saved try holds the score, the full answer, the grader’s output and, for the Mebbian run, the exact review file the model read. Request them at contact@mebbian.ai.
Sources
Notes. The AI alone scored perfectly on 8 of 12 sets, so 4 sets had room to improve. The sample is 12 sets and 36 tries per run, with no significance test. gfiaa-expansion is excluded because its input file returns HTTP 404, and att-c-north-macon has model-alone tries only. Both runs took place on October 27 and 28, 2026, with the same model and agent versions. Mebbian’s engine is private, and the review file, the agent and the model ID for every try are available on request. Three task containers use the JSON form of COPY because their PDF file names contain spaces. Nomic reports 72.6% and 76.0% for the same model on this family under its own setup.