Observatory Lab runs an autonomous, evidence-driven research program on multi-agent AI systems. We build small controlled experiments, label what we find by evidence class, and never claim more than the data supports.
Multi-agent systems already mediate real work. Agents coordinate, negotiate, propagate each other's mistakes, interpret authority, and operate with varying degrees of oversight. Most public discussion of this is speculation.
We replace speculation with controlled comparison: seeded tasks with verifiable ground truth, deterministic evaluators, and a public record of what ran, what failed, and what remains uncertain.
Honesty is a design constraint. Infrastructure tests are never reported as findings. Failed runs are recorded, not hidden. Exploratory results are labeled exploratory. Nothing is claimed novel without review.
Free and open resources only. The program runs exclusively on free tiers and open-source tooling, and publishes its methods so others can replicate every experiment.
How distributed agents organize, share state, and assemble private information into joint answers — and what each coordination structure costs.
How agent groups disagree, align, and converge — and when consensus becomes confident, shared error.
How false claims travel through agent networks, when verification regimes detect them, and why detection is not the same as correction.
How agents interpret conflicting instructions, ambiguous priorities, and rules that compete — and how they attribute the rule they followed.
How much oversight agent systems actually need, what different supervision cadences catch, and what they cost.
Reproducible by construction. Seeded task generators, Python-verified ground truth, deterministic evaluators, and per-call transcripts with recorded configuration for every experiment.
Controls before conclusions. Deterministic stub agents validate the harness itself before any model run; stub outputs are never reported as evidence.
Labeled evidence.
ExploratoryReplicated ReproducedInconclusiveFindings move between classes only through replication. An inconclusive or failed experiment is a published outcome, not a hidden one.
How to read this page. Every cell shows planned and completed denominators, separates control trials from injected-fault trials, and names the model that ran it. "Exploratory" means small samples on one model and one task family: observations, not established patterns. "Inconclusive" means the experiment cannot answer its question as run. An infrastructure error (quota, timeout) is reported as an incomplete execution, never as a result. No experiment so far measures civilization formation, agent corruption, or long-horizon goal retention; where a test touches a related question (did agents keep the instructed priority? was a planted false claim adopted?), we report exactly that measurement and nothing more.
| Experiment | Track | Model | Planned | Completed | Status |
|---|---|---|---|---|---|
| EXP-A01 | Coordination structure | Gemini 3.1 Flash Lite2 passes, 3 seeds x 3 structures | 18 trials | 171 infrastructure error | Exploratory |
| EXP-B01 | Consensus mechanisms | Gemini 3.1 Flash Lite1 seed x 8 items x 4 mechanisms | 32 items | 239 infrastructure errors | Mediator arm incomplete |
| EXP-C01 | Error propagation | Gemini 3.1 Flash Lite2 passes, 3 seeds x 3 regimes x injection on/off | 36 trials | 333 infrastructure errors (pass 1) | Exploratory |
| EXP-C01 | Error propagation | Llama 3.1 8B fp8 (Workers AI)3 regimes x 10 seeds x injection on/off | 60 trials | 60 | Inconclusive - floor effect |
| EXP-C01 fix | Instrument validation | Llama 3.1 8B fp8 (Workers AI)control-only, 10 seeds | 10 trials | 10 | Completed - stopped per pre-registration |
| EXP-C01 rep. | Error propagation | Gemini 3.1 Flash Litepre-registered replication, 3 regimes x 10 fresh seeds x injection on/off | 60 trials | 6010 infrastructure-error trials, labeled (primary analysis) | Replicated in setting (primary analysis) |
| EXP-D01 | Instructions & authority | Gemini 3.1 Flash Lite2 passes, 3 seeds x 2 priority conditions | 12 trials | 12 | Exploratory |
| EXP-E01 | Supervision cadence | Gemini 3.1 Flash Lite3 seeds x 4 cadences | 12 trials | 111 infrastructure error | Exploratory |
| EXP-S01 | Society line - cooperation & charters | Llama 3.1 8B fp8 (Workers AI)pre-registered, 3 paired seeds x 2 conditions | 6 runs | 6 | Exploratory - claim not met |
Source: first-batch report, EXP-A01 · data
Source: first-batch report, EXP-B01 · data
Watch one of these trials, step by step - a false claim travels the chain →
Source: first-batch report, EXP-C01 · CF-Llama instrument-adequacy report · Gemini replication cell report (60/60) · data
Source: first-batch report, EXP-D01 · data
Source: first-batch report, EXP-E01 · data
What comes next. The pre-registered 60-trial replication of EXP-C01 on Gemini (10 fresh seeds, adoption/citation split kept) ran on 2026-10-28 on the free tier: under the primary (keep-first) analysis declared at closeout, attainable-ceiling adoption was 8/10, 8/10 and 7/10 of injected trials across the three regimes with 0 detections anywhere - the pilot pattern (near-universal propagation once injected, near-zero detection) replicated in the same setting, where "replicated" refers to the pre-registered contradiction check only. A post-hoc sensitivity analysis (retry-after- infrastructure-failure semantics; 3 superseded trials) gives 9/10, 9/10 and 8/10 with 0, 1 and 0 detections - the one detection did not prevent adoption. n=10 per cell, one model, one task family: a replicated result in the same setting, not a general effect. After that: EXP-B01 with harder items below the ceiling, EXP-E01 with a fallible model-based supervisor, and an EXP-A01 cost curve as agent count grows. Full methods, pre-registrations, run logs and transcripts: GitHub and Hugging Face.
Replay all six runs - reserve, actions and verbatim discussion →
Source: pre-registration · cell report · data
First experimental batch complete - October 2026. Five pilot studies, one per research track, executed on real model calls at zero cost (free tiers only), plus one pre-registered 60-trial cross-provider cell, one instrument-fix validation, and one pre-registered six-run society-line cell. All findings are exploratory or inconclusive as labeled above: small samples, single model per cell, single task family per track. They are inputs to replication design, not conclusions.
Completed 2026-10-28: the pre-registered EXP-C01 Gemini replication (60/60 trials; primary analysis keep-first as declared at closeout: 10 infrastructure-error trials labeled, kept in all denominators; post-hoc retry-semantics sensitivity: 7). Everything this program publishes - methods, data, transcripts, failures - stays public on GitHub and Hugging Face.