- Blog
- Six Months Ago, We Told You to Wait. Not Anymore.
Six Months Ago, We Told You to Wait. Not Anymore.

Why we told you to wait
Just a few months ago, when a client asked whether they could hand their drawings over to AI for analysis (LLMs in particular), we often advised them to hold off.
Some of it already worked. We could usually handle the text in drawings and tender documents without trouble. The drawings themselves were another story: you had to train a vision model for each type of plan, annotate datasets by hand, and development could take months. For most projects, the payoff didn’t justify the effort.
We preferred to say so plainly rather than sell a project that wouldn’t deliver.
What changed
Two things shifted at the same time.
The models. Large multimodal models now read a drawing the way a technician would. They recognize symbols, read dimensions, and match a legend to the elements on the page. What used to require a vision model trained on your own data now works almost out of the box.
The agents. An agent no longer just answers a question: it plans, breaks the drawing into zones, checks its own work, and corrects its mistakes. A year ago, every step had to be supervised. Now you give it a goal and some tools, and it finds its own way.
These advances have tipped the cost-benefit equation. A plan that used to take hours now takes minutes, for a few dollars in model usage fees.
Plan2Trace: recovering the data in a PDF plan
Most plans are drawn in software like AutoCAD, then circulated as PDFs or scans. Once a plan is exported to PDF, it’s just an image, so engineers have to extract the data by hand whenever they need to prepare an estimate or redraw a plan.
Plan2Trace is the prototype we built to recover that data. There’s nothing magical about it: it combines off-the-shelf models with an agent-based approach inspired by a talk from DB Engineering & Consulting (Deutsche Bahn) at LangChain’s Interrupt conference, originally designed for railway plans. We adapted it to other types of drawings: structural steel, piping, and architecture.
It works in three steps:
- Upload a plan as a PDF or image.
- Add a legend, or an example of an already-annotated plan, to show the agent what you expect. For measurements to be accurate, the plan must state its scale.
- Get back an annotated file, ready to import into software like AutoCAD.
No model to train, no dataset to build. The annotated example is enough to steer the agent toward each client’s own conventions.
Office floor plan annotated in 23 steps.

The results
Our first tests cover several types of plans: residential, office, structural steel, and municipal networks. The preliminary results are very promising: annotation accuracy exceeds 90%.
We tested the tool on two real-world datasets:
- Water and sewer networks of the City of Trois-Rivières. Six layers of open data published on Données Québec: water mains, sewer mains, valves, manholes, fire hydrants, and catch basins. The connections between pipes come from the city’s hydraulic model.
- Swiss Dwellings. Real apartment layouts, from which we draw a reproducible sample of 2,000 floor plans, evaluated using the same method as our other floor plans.
Water and sewer networks: synthetic rendering based on open data from the City of Trois-Rivières.

To measure accuracy, we compare each plan annotated by the agent against a reference version, element by element. An element counts as correct only if it has the right type, sits in the right place (within a few pixels), has measurements that match the reference, and carries the correct value or label. For lines such as walls, beams, or pipes, we check that the path follows the reference. For links between elements, such as a pipe connected to a manhole, we check that both ends are correct.
The score combines how much of what should have been found the agent actually found, and how much it added by mistake. But the most useful output is the corrected plan: every miss and every error is marked right where it occurs, so you can see at a glance where the agent goes wrong.
Comparison with the reference: X marks show elements added by mistake; dotted circles show missed elements. Plan taken from Swiss Dwellings v3.0.0 (Standfest et al., Archilyse AG), CC BY 4.0.

The platform isn’t built for just one type of plan. Moving to a new context means swapping the legend or the example you provide, not rebuilding the solution.
That said, it will never be perfect. The agent sometimes misses measurements. And on a drawing, a line doesn’t always touch the element it should connect to, which can cause the agent to miss a link. So an engineer stays in the loop: they review the annotations in a tool they already know and make corrections as needed. Their role shifts from data entry to validation.
Under the hood: LangChain and LangSmith
Two tools connect the models to the plans. LangChain is an open-source library for linking an AI model to tools. LangSmith, from the same company, makes it easy to trace the actions an agent takes.
Swap models without rewriting everything. New models come out almost every month. With LangChain, we plug in the newcomer, rerun our tests, and compare.
See the agent think. LangSmith traces every step of the agent: the zone it’s examining, the tool it calls, the decision it makes. When an annotation is wrong, we know exactly where the error crept in.
What’s next?
If you were ever told to wait, now is the time to ask again.
Do you have plans to analyze, drawings to digitize, or annotation conventions of your own? Send us a sample plan: we’ll run it through Plan2Trace and show you the result.

Send us a plan
