Report for

Choose a day

AI agent cooperation

PARLEY collaboration puzzles, benchmark saturation and agent swarms

GPT-4o pairs solve 60.5% of cooperative asymmetric puzzles and fail mostly at coordination. A re-scoring of eight agent benchmarks shows pass/fail stops ranking before tasks run out, and a 1,000-agent team formalizes Fermat through emergent SQL coordination.

  • collaboration-benchmark
  • failure-taxonomy
  • benchmark-saturation
  • communication-control
  • multi-agent-orchestration
  • agent-monitoring

Evaluation

  • Authors’ claims

PARLEY measures LLM collaboration through asymmetric puzzles

Hernandez and Wilson introduce PARLEY, a platform for studying LLM agent collaboration through four asymmetric puzzle tasks grounded in cognitive psychology: referential communication, joint coordination, perspective-taking and spatial reasoning. Across 400 conversations between GPT-4o pairs (100 per task), agents succeeded in 60.5% overall, from 34.0% on Tangram Reference to 80.0% on Scene Comparison; human baselines from conceptually related paradigms sit at 70 to 96% but the authors mark them directional, not exact. From 1,621 failure instances extracted with an LLM judge and clustered by embedding, they derive the PAC taxonomy: coordination failures are most prevalent (52.1% of instances; verification n=563, grounding n=282), then adaptation (36.3%; progress tracking n=244, feedback integration n=344), then perception (11.6%; visual interpretation n=90, spatial reasoning n=98). Feedback integration is the most discriminative pattern, present in 78.5% of failed versus 21.5% of successful conversations, and five of six patterns significantly predict failure in a mixed-effects model. A taxonomy-guided system-prompt intervention (200 conversations, 50 per task) raised average success to 70.5% (+10.0 points, Fisher's exact test), improving every task.

Source figure Figure 4, full figure scaled to frame with whitespace paddingFull resolution
UMAP projection of 1,621 failure instances from 400 PARLEY conversations, clustered into six labeled patterns: verification (n=563), grounding (n=282), progress tracking (n=244), feedback integration (n=344), visual interpretation (n=90) and spatial reasoning (n=98), grouped under coordination, adaptation and perception failures.
Source figure: Hernandez and Wilson, PARLEY, arXiv 2611.14242, Figure 4, full figure scaled to frame with whitespace padding. License: arXiv nonexclusive distribution license; reproduced as a quotation accompanying this review with attribution

Assessment

This is the closest thing to a cognitive-psychology-grounded cooperation benchmark for LLM pairs, and the PAC taxonomy decomposes cooperation quality into measurable failure classes instead of one success bit. That a simple taxonomy-guided prompt then lifts success shows the diagnosis is actionable, a template for anyone building cooperation evaluation.

Submitted November 10; announced in the November 11 listing.

  • Authors’ claims

Agent benchmarks run out of ranking before they run out of tasks

Dong, Zhan and Hu re-score released runs of eight agent benchmarks and re-run Terminal-Bench to argue that apparent saturation is largely measurement. On SWE-bench Verified the top ten now spans 3.6 points as one statistical tier: none of the 45 within-top-ten pairs differs significantly, the largest neighbor gap is 1.4 points where the test can detect about 6.3, and strengthened tests (SWE-ABS) reject 17.7% of the median agent's passing patches (range 10.8 to 23.2%). Correcting verdicts shrinks the top tier from 23 agents to 5 and reverses 8 of 18 neighboring orders, but deleting a random sixth of each agent's passes reverses about as many (8; 95% band 4 to 11), so neighbor order is noise. On ALE-Bench a pass/fail bar at the median human collapses the top ten's three contest-score tiers into one, although nearly half of runs beat that human, while a contest-winner bar scores 89% of earlier models zero. On Terminal-Bench 2.0, 18 of 89 tasks compute a quantity and then discard it; pass/fail-tied models differ over tenfold on that quantity, and median cost per tied run runs from $0.44 to $2.47. They derive a clipped log-ratio score with dated anchors, cost as a separate axis and lexicographic gate-then-quality reporting.

Source figure Figure 2, full figure scaled to frame with whitespace paddingFull resolution
Top ten agents on SWE-bench Verified, Terminal-Bench 2.0 and OSWorld-Verified with 95% confidence intervals: SWE-bench forms one tier with a 3.6 point spread, Terminal-Bench one tier with 8.5 points, OSWorld two tiers with 12.0 points.
Source figure: Dong, Zhan and Hu, Agent Benchmarks Stop Ranking Before They Run Out of Tasks, arXiv 2611.13541, Figure 2, full figure scaled to frame with whitespace padding. License: CC BY 4.0

Assessment

Required reading for anyone designing an agent metric: a single pass/fail bit per task stops discriminating long before tasks are exhausted, and the fix is graded credit with frozen anchors plus cost reported as its own axis, not a harder bar. The noise analysis of leaderboard neighbor order applies directly to cooperation leaderboards.

Submitted November 10; announced in the November 11 listing.

Coordination research

  • Authors’ claims

Where multi-agent failures enter, and which controls reach them

Wang, Wen, Zhang and Jiao bring dependability engineering's fault-error-failure vocabulary to LLM agents through an episode record that ties each fault to its origin, propagation, detection, response and consequence. An LLM annotator (one independent Claude instance per trace, following a written 38-field protocol) read 95 failed runs of six multi-agent systems from the Who&When and MAD corpora in full. Faults enter early, at a median 0.18 of the run, mostly as wrong decisions or fabrications over task state, while symptoms surface a median 0.67 of the run later; in 37 of 88 Who&When runs the origin precedes the step the corpus's human labels mark as decisive. In 72% of failed runs the trajectory already contained evidence against the error, and in 88% of those no agent acted on it; something flagged a deviation in 31% of runs, but no response removed the deviated state. Counterfactually, independent judges reading the unused evidence, task contracts fixed at admission, provenance on stored facts and re-checking beliefs against the environment would reach most episodes, while restoring the agent's own state suffices in only 35 of 95 runs because in 56 the fault sits in a plan or policy a restore would replay. A deterministic scan of all 184 raw Who&When trajectories also finds that 26 of the 58 hand-crafted failures (45%) end in a provider content-filter crash after which the harness emits a final answer no agent produced, with 21 of those runs attributed by the corpus to an agent other than the orchestrator.

Assessment

Failure attribution is the part of cooperation evaluation nobody wants to do, and this shows the two popular shortcuts both mislead: symptom labels and responsible-agent labels miss where errors enter, and 45% of one corpus's failures are a harness artifact no agent caused. The control analysis says detection is not the bottleneck; evidence is usually present and unused.

Submitted November 9; announced in the November 11 listing.

  • Authors’ claims

Pruning committee messages helps in a cache, not yet at the endpoint

Xue, Liu, Yan, Xie and Yang ask whether a router can drop one specialist message from a multi-agent reasoning committee without losing correctness, using only pre-integration information. On a sealed cache of 5,328 committees (SciAgentTrace leave-one-out outcomes over gpt-oss-120b), their calibrated gate gains +1.80 accuracy points (95% CI +1.13 to +2.52) while removing 12.76% of message tokens. The endpoint replication is the instructive part: 4,580 controlled integrations over a stratified 458-committee sample on a public gpt-oss-120b endpoint give a sample estimate of -0.09 points, a design-weighted +0.94 points (CI -1.31 to +3.50) with 18.72% acted-input reduction, and no established advantage over a token-matched deletion control (+1.68, CI -1.27 to +5.10). Only 13 of 88 cached non-neutral labels (14.8%) keep their direction under repeated-call majority, and 88 of the 96 original net recoveries come from a single benchmark (LAB-Bench). The authors therefore propose a numerical qualification rule that falls back to keeping all messages, formulated after seeing the endpoint results and applied only retrospectively.

Assessment

A model study in how to report communication pruning honestly: cached replay shows a gain, live replication does not confirm it, and the authors say so with confidence intervals. Anyone building message routing for agent committees should treat cached leave-one-out effects as fragile until replicated at the endpoint.

Submitted November 10; announced in the November 11 listing.

  • Authors’ claims

Watching one channel does not control learned agent communication

Liu and Xia test whether monitoring a visible channel actually controls learned agent communication. In an end-to-end sender-receiver game where a discrete protocol and route policy are learned without communication supervision, protocols form in 20/20 and 19/20 seeds of two independent batches. Online partial monitoring shifts traffic onto a costlier, less reliable shadow route (share up by .701, CI .574 to .826, and .575, CI .412 to .737, over no monitoring) while task accuracy stays at .905 and .913 and signed message contribution at .592 and .563; complete coverage removes most of this selective migration without removing communication. Frozen interventions show the route itself is semantic: shuffling joint message-route codes changes .668 of receiver predictions. Their stricter semantic-evasion gate, which would require content-level adaptation, passes 0/20 seeds in both batches and is reported separately as a negative.

Assessment

A small but causal demonstration that communication oversight can relocate a protocol rather than suppress it, which matters for any monitoring scheme that watches one channel of a multi-agent system. The pre-registered gate structure and the honest 0/20 on the stronger claim make the limits as readable as the result.

Submitted November 7; announced in the November 11 listing.

  • Authors’ claims

No universal best agent architecture across enterprise procedures

Nandi, Arora, Bai and colleagues hold the model, tasks and tools fixed and vary only the architecture: a function-calling single agent, skill-augmented agents, centralized multi-agent orchestration and a triage swarm, eight arms across 13 enterprise-style standard operating procedures. There is no universal winner: per-SOP task success ranges from under 10% on the hardest to near 100%, and the right choice is SOP-dependent. Cost is the robust separator. The single-agent baseline (67.4% average task success, 24.9k tokens and 24.0s per task) is Pareto-optimal: no arm is both cheaper and more accurate, two coordinated arms are strictly dominated, and the arms that edge it pay heavily for insignificant gains. Orchestration-v2 is the only arm that significantly beats the baseline (75.6%, +8.2 points) but is also the most expensive in the study at 4.7x tokens (117.3k) and 3.1x latency, driven by giving every worker the full SOP and tool set plus a re-dispatch round. A matched skill-definition quality rubric shows high-quality skills help on only a few SOPs.

Assessment

A clean controlled answer to the question every multi-agent deployment asks: coordination overhead must be justified per workflow, because the cheapest single agent is competitive almost everywhere and the one orchestration that wins pays nearly five times the tokens for it.

Submitted November 7; announced in the November 11 listing.

Infrastructure

  • Catch-up item
  • Authors’ claims

A 1,000-agent team reproduces a formal Fermat proof on general tools

Zhang, Peng, Najt, Scholl and Martin report a toy experiment in which 1,000 agents running an early version of Claude Fable 5.2 reproduced Anthropic's September formal proof of Fermat's Last Theorem without the bespoke Prove2Me platform or researcher steering. Given 60 statements, a one-sentence route sketch and 47 already-proved lemmas from the earlier run, plus Git and a shared Postgres database, the team organized itself: the single lead created five coordination tables in the first ten minutes, crews grew from 16 or 17 helpers to between 46 and 139 as the lead reallocated them, and by hour 5.6 helpers had written 756 tasks with only 16 of the 73 task-writing helpers being captains. The hardest statement, Langlands-Tunnell, became a task tree of 2,418 tasks up to 10 levels deep and was proved at hour 28.4, with the theorem kernel-checked at hour 28.7: one third of the original run's output tokens and a proof one seventh the length. Safety came from sandboxed virtual machines without network access, a real-time action classifier checking every tool call (no blocks across more than a million calls) and an offline monitor; in Anthropic's separate red-team evaluations, monitors that see chain-of-thought and interagent communication detected all sabotage attempts, though this run used only the action-based monitor.

Source figure Blog Figure 3, full figure scaled to frame with whitespace paddingFull resolution
The five shared database tables the lead agent created to coordinate 999 helpers: Announce for orders, Assign for statement ownership, Notes as an open board, Subtasks as a shared to-do list and Landed as a log of finished work, with read and write directions marked for lead and helpers.
Source figure: Zhang, Peng, Najt, Scholl and Martin, Anthropic, Safely scaling multi-agent teams with Claude Fable 5.2, Blog Figure 3, full figure scaled to frame with whitespace padding. License: Anthropic copyright; reproduced as a quotation accompanying this review with attribution

Assessment

The coordination system here is emergent, not engineered: five SQL tables, self-assignment and a recursively split task tree replaced a purpose-built platform, which is direct evidence about what large agent teams invent when given only general primitives. The layered monitoring stack, and the note that a single shared machine broke down around 100 agents, are practical scaling data points.

Published November 9; covered November 11, two days later (catch-up).

  • Authors’ claims

Autonomous synchrotron experiments with auditable human-agent teamwork

Luthra, Pandolfi, Diroll and colleagues run autonomous synchrotron X-ray experiments on nanoparticle superlattices with a team of specialized coding agents, an Operator for beamline control, an Analyst for scattering analysis and a GP agent for adaptive sampling, working under evolving human-defined objectives. Complete human-agent and agent-agent transcripts feed an audit pipeline that reconstructs decision provenance and quantifies the division of cognitive labor: across three experiments, 61 to 67% of messages involve interpretation or judgment rather than mechanical execution, and the share of methodological and scientific contributions rises from 46% in the setup-heavy first experiment to 56 to 61% in later ones.

Assessment

If human-agent teams are going to run physical instruments, this is the accountability pattern: keep every transcript and make the audit reconstruct who contributed judgment. The rising share of methodological contributions as experiments progress is an early measurement of agents moving from execution into scientific reasoning.

Submitted November 6; announced in the November 11 listing.

  • Authors’ claims

Google Antigravity ships /swarm multi-agent orchestration in preview

The Antigravity team rolls out /swarm in preview alongside the Gemini 4 Carbon model: typing /swarm turns a prompt into a coordinated system where a coordinator spins up specialized parallel workers in isolated git worktrees and adversarial verifiers run tests and challenge diffs, all collaborating on a shared, append-only message board where idle agents park at zero token cost and activate when addressed. Agents can be pinned to different models per role, with Carbon positioned for coordinator and adversarial-verification roles, and a Swarm UI exposes a live board, a member roster with per-agent trajectories and a graph replay of how the team converged. Google engineers dogfooded the system for five weeks on multi-package migrations, cross-subsystem root-cause investigations and architectural redesigns. Access starts with Google AI Ultra subscribers, and the vendor warns that parallel multi-agent systems consume quota much faster than standard chat.

Assessment

The design choices are the news: append-only shared board, isolated worktrees, adversarial verification as a first-class role and zero-cost idle parking are becoming the default vocabulary of production multi-agent coding. Worth tracking as the deployment-shaped counterpart to today's coordination research.

Published November 10; covered November 11.

Also seen

  • Authors’ claims

Three agent evaluation paradigms compared on frozen traces

Zhang, Arvin, Brown and colleagues compare three evaluation paradigms (trace-based LLM judging, text-only LLM judging with red-teaming, and deterministic heuristics) on two enterprise analytics agents over frozen execution traces. 97% of judge configurations meet their stability threshold and the pattern is agent-independent; on 72 ground-truth failure cases the trace-based framework achieves the best precision-recall balance (F1=0.90) while text-only evaluation matches it on recall; and spot-checking alone misses 9 of the 72 known failures. They recommend a two-framework strategy: trace-based evaluation for functional assessment, adversarial red-teaming for security.

Assessment

A stability-first comparison that treats the evaluation harness itself as the measurement instrument worth calibrating.

Submitted November 10; announced in the November 11 listing.

  • Authors’ claims

Human evaluators are getting pickier on fixed model pairs

Chen, Zheng and Zhu separate model change from judge change using 1,260,967 blind Chatbot Arena votes from May 2023 to August 2024 on pairs of version-fixed models. For the same unchanged pair, the share of votes calling both answers bad rises by 1.71 percentage points a month on a 17.1% mean (1.91 points in 2024), and it rises in all 14 pairs observed in both years, from 15.5% upward. The perceived decline of AI assistants is at least partly rising user standards.

Assessment

A clean identification trick that warns against comparing human preference scores across time, including in cooperation and interaction studies.

Submitted November 10; announced in the November 11 listing.

Editorial synthesis

Two of today's results are about what measurements can see. PARLEY decomposes LLM collaboration into perception, adaptation and coordination failures and shows a taxonomy-guided prompt repairs part of the gap, while the benchmark re-scoring study shows pass/fail leaderboards stopped ranking frontier agents some time ago: neighbor order is noise, and graded credit with a separate cost axis is the fix. The coordination papers add the same lesson from other angles: committee message pruning that replicates in a cache does not yet replicate at a live endpoint, and monitoring one channel relocates a learned protocol instead of suppressing it. On the constructive side, a 1,000-agent team invented its own coordination system from SQL primitives to formalize Fermat, Google productized the swarm pattern, and an auditable human-agent team ran real beamline experiments with full decision provenance.

Editorial interpretation of the cited sources.