AgentLab
A runtime that executes AI agents across browser, sandbox and desktop, captures evidence for every claim, recovers from tool failures and scores each run.
- 3
- Execution environments
- 5
- Scoring dimensions
- 2
- Model providers, routed
- 1
- Command to demo
- Role
- Author
- Year
- 2026
- Type
- Agent execution & evaluation runtime
01Overview
AgentLab runs AI agents in real environments and keeps everything that happened. Every tool call, every intermediate result, every recovery from failure, and every source the final answer rests on is persisted as a run you can replay.
It came out of a frustration with agent scripts: they produce an answer and throw away the reasoning. When the answer is wrong there is nothing to inspect, and when it is right there is no way to prove it.
The result is closer to a lab notebook than a chatbot — a runtime, an evidence system, an evaluator and an explorer UI for reading runs back.
02The problem
A typical agent script loses tool calls after execution, returns an answer with no sources, terminates on the first tool failure, and gives you no way to compare one run against another. Debugging means reading logs and guessing.
Three things had to be true instead: the trace must survive the process, answers must cite the evidence that produced them, and a failed tool call must be a recoverable event rather than the end of the run.
03Architecture
- 01
Three environments behind one interface
Browser (navigate, click, extract), sandbox (execute, python, git) and desktop (screenshot, type, click). Each has a mock implementation that runs offline and instantly, and a Solari implementation that executes in real cloud environments. Agent code is written once against the interface.
- 02
Model routing with fallback
A ModelRouter fronts the providers — Gemini as primary, Groq as fallback — so a provider outage degrades the run instead of ending it.
- 03
Evidence system
Answers carry inline citations. "Cursor Pro is $16/mo [source:1]" links back to the exact browser page the figure was read from, so a claim can be checked rather than trusted.
- 04
Recovery-aware loop
Tool results branch: success feeds back into the model, failure routes into recovery and retries against the environment. Cascading failures are detected rather than silently absorbed.
- 05
Five-dimension evaluation
Every run is scored on completion (50%), efficiency (25% — tool success rate and step count), resilience (15% — tool diversity, no cascading failures), tool usage (10% — coverage and execution time), and recovery. The output is a single graded score with the components visible.
- 06
Explorer UI
A separate app renders the persisted run: execution timeline, evidence, evaluation breakdown, and the final answer with its citations resolved.
04Key decisions
Mock-first environments
Every environment has an offline implementation, so the whole system demos and tests in under a second with no API keys. Cloud execution is a swap, not a prerequisite.
Runs as files, not a database
A run serialises to a single JSON document. It can be committed, diffed, attached to an issue or replayed months later without standing up infrastructure.
Evaluation as a weighted rubric, not an LLM judge
Deterministic scoring across five dimensions means two runs are comparable and a regression is measurable. An LLM judge would have been faster to write and impossible to trust as a baseline.
05Stack
- TypeScript
- Node.js
- tsx
- Gemini
- Groq
- Solari