PhiloLabs Deploys AI Agents to Build 3D San Francisco in Two Hours for $33
A team of Claude Fable 5.1 agents constructed a detailed Three.js replica of Union Square using geographic data and reference images, then caught visual flaws through automated screenshot comparison that traditional testing would have missed.
PhiloLabs conducted an experiment to test the limits of AI coding agents on a project requiring more than just functional code. The company deployed Claude Fable 5.1 agents to construct a 3D browser-based recreation of San Francisco's Union Square, drawing from real geographic data and photographic references. Within two hours, the agents had assembled a working Three.js implementation featuring 453 building footprints, 75 custom façades, 129 named storefronts, plus 220 pedestrians and 109 vehicles navigating the scene, including the iconic cable cars on Powell Street.
The project consumed approximately 8 million tokens and incurred roughly $33 in API costs. Beyond simply achieving a working application, PhiloLabs integrated Playwright into the development workflow to identify visual problems that conventional testing methods would overlook.
Playwright as agent vision
PhiloLabs organized the reconstruction effort across specialized subagents responsible for geographic research, building geometry, textures, storefronts, and other scene components. Once the application ran in the browser, Playwright captured screenshots from 34 predetermined camera angles and compared them against photographs of the actual Union Square.
The agents generated 147 comparison sheets that revealed discrepancies between technically correct code and visual accuracy. "In all, the agents produced 147 comparison sheets, making it easier to spot things that were technically correct but still looked wrong." Issues included buildings positioned correctly but with incorrect proportions, or storefronts appearing on the wrong side of a street. Consistent camera positions across iterations also enabled tracking of changes between development cycles.
Agents reviewing agents
PhiloLabs assigned specialist agents to evaluate the output, with some concentrating on architectural and geographic accuracy while others assessed technical art and interactions. These reviewers produced nine reports documenting issues with the Union Square build.
The reports functioned as a task list for subsequent development iterations. Development agents addressed the reviewers' observations and reran the scene to implement corrections.
This approach addresses a fundamental challenge in automated testing: not all errors translate into measurable test cases. While verifying that a building occupies the correct coordinates is straightforward, determining whether a street authentically resembles Union Square requires visual judgment that traditional testing frameworks cannot easily capture.
Filling gaps in source data
The agents lacked a finished 3D model to simply replicate. Instead, they synthesized open geographic data—primarily OpenStreetMap and USGS elevation datasets—with information about the actual location, then converted this material into geometry, façades, and interactive elements suitable for browser deployment.
Significant data gaps remained throughout the process. Geographic information could pinpoint a building's location without revealing its façade appearance, while photographs captured only the visible portions from specific angles. When neither source provided complete information, the agents made independent decisions about how to proceed.
Agent-based review does not guarantee accuracy when source material is incomplete. Reviewers can overlook the same issues that initial agents missed, particularly when working from insufficient reference data.
What $33 buys
"Eight million tokens is a lot of model activity for a single application. Yet the reported API cost for the Union Square run was about $33." PhiloLabs achieved this cost efficiency through several strategies: distributing work among subagents, executing tasks in parallel, and leveraging cached context to avoid redundant processing of the entire project.
The Union Square experiment remained relatively contained in scope. Screenshot-based validation becomes less practical as agents tackle more complex applications. Spline recently rebuilt its 3D editor using Claude Code agents, but the resulting interface displays only a portion of what those agents produced. Issues embedded within code or triggered by actual user interactions may never surface in screenshots.
Source: The New Stack