Report for

Choose a day

AI agent cooperation

When agents should stop, what audits miss, where facts vanish

Passing tests do not settle when an agent should stop. Clinical benchmark audits find the largest hindsight effect in case text, not simulated labs. Two shared-memory studies show why collecting more facts need not preserve disagreement.

  • anytime evaluation
  • clinical hindsight
  • correction utility
  • conflict-preserving memory
  • evidence handoffs

Evaluation

  • Authors’ claims

Done, but still working: evaluate the patches before the final one

Yang and colleagues report an anytime evaluation of about 6,000 coding-agent runs spanning eight open models and two scaffolds on SWE-bench Verified. Replaying commands and scoring intermediate patches permits paired comparisons of stopping or earlier-patch selection against finishing the same run, without new trajectories when replay preserves the relevant states. The hindsight cost saving available at passing self-tests ranges from under 1% to 57% across model and scaffold settings. For Qwen3.6-27B in mini-SWE-agent, stopping after two passes on an unchanged patch saves 22.8% of inference cost at a 1.0-point loss in local resolution in a preregistered replication of 386 runs. The uncertainty-based test does not establish non-inferiority at the 2-point margin. The paper decomposes quality loss into rule firing, later patch changes and net repair per change.

Adapted from a source figure Figure 1, complete two-panel chart, caption excluded as a crop adaptation; proportions preserved and whitespace padded to 1200x600. Descriptive all-run estimates at c=0.1 under corrected scoring, including derivation runs.Full resolution
Two stopping rules across five model and scaffold combinations: inference cost saved on the left and change in corrected local resolution on the right. Stopping after two passes costs 2.8 points for Qwen3.6 in SWE-agent but 0.3 in mini-SWE-agent; DeepSeek-V4-Flash gains 0.9 points.
Adapted from a source figure: Zerui Yang, Done, But Still Working: Anytime Evaluation of When Coding Agents Should Stop, arXiv 2611.22901, Figure 1, complete two-panel chart, caption excluded as a crop adaptation; proportions preserved and whitespace padded to 1200x600. Descriptive all-run estimates at c=0.1 under corrected scoring, including derivation runs.. License: CC BY 4.0

Assessment

Intermediate-state evaluation distinguishes wasted work from work that repairs a patch. Paired replay is especially useful for evaluating stopping policies, but it is not a replacement for fresh runs when an intervention changes the agent's behavior.

Submitted November 17; appeared in the November 18 listing.

  • Authors’ claims

HindAudit locates the biggest hindsight effect in the case text

Zhu and colleagues report HindAudit, separating three channels in clinical-agent benchmarks: the diagnosis supplied to a simulator, retrospective simulator context, and agent-facing case text. On a cohort of 27,489 MIMIC-IV emergency admissions, five registered tests did not detect diagnosis leakage into simulated laboratory results; giving the diagnosis to five simulators had negligible effects, while prompted gatekeepers diluted real diagnostic signal. The larger effect was in case construction: across four LLMs, benchmark-style summaries yielded 39.7% zero-shot accuracy and discharge histories 45.3%, versus 25.4% from a presentation-time reconstruction based on triage records. The initial case-text analysis was exploratory, with the gap reproduced in a registered held-out-patient replication. Six interactive agents also scored higher from benchmark text, while their ranking changed little.

Adapted from a source figure Figure 1, entire three-channel diagram, caption excluded as a crop adaptation; proportions preserved and whitespace padded to 1200x600. Diagram values are rounded source annotations.Full resolution
HindAudit separates the diagnosis line, retrospective simulator history and agent-facing text. The diagram contrasts retrospective discharge-summary construction with a presentation-time reconstruction from emergency-department triage.
Adapted from a source figure: Meilu Zhu, Where Does Hindsight Enter Clinical Agent Benchmarks? An Audit of Simulators and Case Construction, arXiv 2611.22241, Figure 1, entire three-channel diagram, caption excluded as a crop adaptation; proportions preserved and whitespace padded to 1200x600. Diagram values are rounded source annotations.. License: arXiv perpetual non-exclusive license; assessed quotation basis: one unchanged source figure within substantive analysis, attributed and linked. Cropping and whitespace framing only.

Assessment

An audit can target the wrong channel. The useful contribution is the separation of case construction from simulator behavior, and absolute score inflation from changes in model ranking.

Submitted November 17; appeared in the November 18 listing.

  • Authors’ claims
  • Catch-up item

Consistency is not correction: check what the gate can change

Sethi reports an exact decomposition of correction utility into gated recoveries minus gated harms, separating answer ranking, selective prediction and improvement after intervention. For a correct replacement-only gate, the initial error count bounds the beneficial discordant pairs it can produce. In the paper's 60-answer diagnostic set, four initial errors are reachable while six one-directional changes are needed for the paired exact test to reject at 0.05, leaving even a perfect gate unable to produce a significant improvement. In a small-model MiniHouse suite, a perturbation-agreement verifier reaches within-policy AUROC 0.990 yet changes no episode outcomes relative to single-sample decoding at 4.32 times the call count. In the controlled 840-episode study, none of the verifier-driven policies replaces the initial proposal on the 60 GSM8K decisions.

Adapted from a source figure Figure 1, complete two-panel power and error-budget chart, caption excluded as a crop adaptation; proportions preserved and whitespace padded to 1200x600.Full resolution
Power of the 60-task paired test and its reachable region, alongside the mismatch between six required discordant pairs and four initial errors.
Adapted from a source figure: Khushal Sethi, Consistency Is Not Correction: Separating Confidence, Abstention, and Recovery in LLM Agents, arXiv 2611.21651, Figure 1, complete two-panel power and error-budget chart, caption excluded as a crop adaptation; proportions preserved and whitespace padded to 1200x600.. License: CC BY 4.0

Assessment

A confidence signal can be excellent at ranking and useless as a controller. Reporting gate firing, reachable errors, harms and recoveries makes a null result interpretable before more evaluation budget is spent.

Catch-up: submitted October 15; first encountered in the November 18 listing.

Coordination research

  • Authors’ claims

Naming the bad teammate is easier than recovering around it

Heo and colleagues report 720 episodes in a custom four-player Overcooked environment, crossing three maps, four teammate conditions and two monitoring prompts, with 30 trials per cell. The main experiment uses gpt-5.4-mini throughout and injects one fixed-policy misaligned teammate: speech-action mismatch, role invasion or aggregate-progress disruption. Eventual verbal attribution reaches 90%, 88% and 97% respectively in the weak-sensitivity correction analysis of 90 episodes per troll condition (30 per map), but an utterance combining attribution with situational diagnosis appears in only 61%, 54% and 31%. Recovery is clearest for the speech-action mismatch and weakest for aggregate disruption. Stronger attribution prompting improves the aggregate-disruption condition from 2.7 to 4.8 soups, while reducing troll-free performance from 8.0 to 5.0 soups, a 37.5% drop.

Adapted from a source figure Figure 4, entire recovery chart, caption excluded as a crop adaptation; proportions preserved and whitespace padded to 1200x600. Error bars are 95% bootstrap intervals from 10,000 resamples.Full resolution
Post-detection recovery in serves per game-time bin for Reneger (L1), Misallocator (L2) and Disruptor (L3), with 95% bootstrap confidence intervals. Reneger is clearly above zero; the other intervals cross zero.
Adapted from a source figure: Dongseok Heo, Bad Apples in the Kitchen: How LLM Teams Detect, Correct, and Calibrate Against Their Misaligned Members, arXiv 2611.22342, Figure 4, entire recovery chart, caption excluded as a crop adaptation; proportions preserved and whitespace padded to 1200x600. Error bars are 95% bootstrap intervals from 10,000 resamples.. License: CC BY 4.0

Assessment

Robustness needs separate measures for detection, recovery and false-alarm cost. The design makes that tradeoff visible instead of treating more explicit blame as an unconditional improvement.

Submitted November 17; appeared in the November 18 listing.

  • Authors’ claims

DebateGraph: preserve disagreement before the debate sees it

Khan and colleagues report a 39-run gated ablation of five shared-memory mechanisms for multi-agent debate. In the graph backend at retrieval cap 200, surfacing the ranked conflict set instead of collapsing it to one winning fact improves answer token-F1 by 0.340, with a 95% interval of 0.267 to 0.414, and reduces successful attacks from 22 to 7 of 122 poisoned items under the same prompt budget, using 11,318 tokens per query for collapse and 11,550 for surface. The pre-declared cap of 40 had hidden most planted contradictions from retrieval and measured a smaller effect: 0.043 F1 overall and 0.084 on poisoned items. At cap 200, the write trust-gate and router provenance features instead cost 0.075 and 0.096 F1 in their respective contrasts. Surfacing helps on a second retrieval substrate, with a positive HotpotQA accuracy effect on its pre-specified discriminating subset; the full-sample interval includes zero.

Source figure Figure 2, complete original PNG; resized proportionally and whitespace padded to 1200x600. Exact label families: full_context, graph, hybrid and vector. The two HotpotQA runs are excluded.Full resolution
Accuracy-cost frontier for 37 LoCoMo runs. The x-axis has a break between retrieval and full-context regimes. The surfaced-conflict graph configuration reaches 0.456 F1 at about 11,500 tokens per query, while full-context configurations remain more accurate and cost much more.
Source figure: Ziaullah Khan, Surfacing Disagreement Beats Resolving It: A Gated Ablation Study of Provenance-Weighted Memory for Multi-Agent Debate, arXiv 2611.22461, Figure 2, complete original PNG; resized proportionally and whitespace padded to 1200x600. Exact label families: full_context, graph, hybrid and vector. The two HotpotQA runs are excluded.. License: arXiv perpetual non-exclusive license; assessed quotation basis: one unchanged source figure within substantive analysis, attributed and linked. Cropping and whitespace framing only.

Assessment

The storage layer can destroy precisely the evidence a debate is meant to adjudicate. The retrieval-cap audit is as important as the positive result: an apparently weak mechanism may never have had access to its target.

Submitted November 17; appeared in the November 18 listing.

  • Authors’ claims

Delegation retrieves more conflicts but hands fewer of them through

Zhang and colleagues report controlled numerical-conflict injection into 11 Harvey LAB document-review tasks, comparing no, optional and required delegation across GPT-5.6 Luna, GPT-6 Luna and Claude Haiku 5.5, with two runs per model. Pooled over 438 conflict observations per policy at comparable generation spending, required delegation increases retrieval of both conflicting claims from 50.2% to 68.7% relative to no delegation, while judged conflict detection falls from 16.2% to 13.2%. Under required delegation, only 35.2% of pairs retrieved by workers survive together in worker reports; pair recall at lead evidence is 24.4%. Matched comparisons associate keeping both claims within one worker context with better detection, and splitting them across workers with worse detection, relative to single-agent execution.

Source figure Figure 3, complete original three-panel PNG, proportions preserved and whitespace padded to 1200x600. Panel (a): Executions using workers and Worker Pair Recall; panel (b): Source Pair Recall and Judge Pass; panel (c): Source to Lead and Lead to Final pair-recall losses.Full resolution
Three panels show model-specific delegation choices, source-pair recall versus judge pass, and pair-recall losses before the final report. Each model has 22 task executions and 146 conflict observations per policy; the axes and denominators differ.
Source figure: Hejia Zhang, Multi-Agent Memory Inconsistency: Where Factual Conflicts Disappear in LLM Agents, arXiv 2611.22245, Figure 3, complete original three-panel PNG, proportions preserved and whitespace padded to 1200x600. Panel (a): Executions using workers and Worker Pair Recall; panel (b): Source Pair Recall and Judge Pass; panel (c): Source to Lead and Lead to Final pair-recall losses.. License: CC BY 4.0

Assessment

Evidence collection and evidence preservation are different capabilities. Profiling source reads, worker reports, lead evidence and final reports exposes losses a task-level pass rate would conceal.

Submitted November 17; appeared in the November 18 listing.

Infrastructure

  • Authors’ claims

GPT-6.2 release puts long-horizon agent cost on the comparison sheet

OpenAI reports the release of GPT-6.2 Astra and Sol. On its announcement's Agents' Last Exam comparison, Astra scores 5.2 points above GPT-6 Astra at less than one-third the cost. It also reports lower-token long-horizon workflows and model availability through its API and Work/Codex products. The announcement states that the comparator results for Claude Fable 5.2 exclude nine tasks that used a fallback model.

Assessment

A relevant new baseline for agent evaluation, particularly when task quality and aggregate inference cost are reported together. This is a model-release claim, not evidence of improved multi-agent cooperation.

Announced November 17; new since the previous daily lab-blog check.

Also seen

  • Authors’ claims

Recovered capacity stays idle unless agents can spread demand

Ezaki and colleagues report simulated shared-server choices in which agents must discover that a server has recovered despite a stale status report. For Claude Haiku 5.5, a rule reminder prompts no agent to try the recovered server, while a successful-request record prompts all to choose it. All five tested models still leave capacity unused after evidence sharing. With GPT-6.1 Sol and Gemini 3.5 Flash-Lite, distinct numerical cues spread simultaneous requests and increase completions without much change in overall choice frequencies.

Assessment

Availability knowledge does not solve congestion: coordination also needs agents to distribute demand.

Submitted November 17; appeared in the November 18 listing.

  • Authors’ claims

Look-ahead-free trading agents fail to beat the simple bets

Wang reports a look-ahead-free comparison before 598 earnings announcements and after 409 bad-news drops in large US stocks. Across five models from four vendors, staged agents do not beat simple buy-all-event baselines. Ticker-and-date-only probes show a GPT-based model predicting announcement direction more accurately before its training cutoff than after it.

Assessment

Event-agent evaluations need simple-action baselines and identity-blind, post-cutoff tests before apparent alpha is credited to reasoning.

Submitted November 17; appeared in the November 18 listing.

  • Authors’ claims

Federated trust infrastructure for cross-organization agents

Nguyen and Husain propose Agent Trust Infrastructure, combining organizational certificate authorities, zero-trust checks and remote attestation as an external enforcement layer. They report a simulated eight-organization federation spanning six frameworks and five-agent teams. The security experiments use two organizations, 14 attack scenarios and six benign tasks; across the 11 scenarios comparable under every configuration, containment rises from 40% to 80% with all enforcement mechanisms enabled.

Assessment

An architecture proposal focused on the identity and configuration checks needed when agents cross organizational boundaries.

Submitted November 17; appeared in the November 18 listing.

Listen

Listen: who measures trust when agents trade without a broker?

Obadia and Haimes discuss the coordination layer needed for agents to negotiate and transact across organizations without a central intermediary. Obadia explains ARIA's planned Arena as an environment for testing whether trust infrastructure survives adversarial pressure. Haimes challenges profit as the core score and asks whether it makes safety secondary; Obadia describes separately measuring security and leaves room to revise the metric. They also discuss responsibility when an agent breaks a deal. The episode runs 1:40:49; publisher transcript entry points are 34:39 for the Arena, 40:29 for profit versus safety, and 54:38 for liability.

Assessment

The most useful part is the disagreement about what a coordination benchmark should reward. It connects evaluation design to incentives, negotiation and accountability rather than treating successful communication as the outcome.

Catch-up: published November 9; not covered in the November 9 through 17 reports. Added November 18 during recovery of omitted listening.

Listen: long-running agents need observable outcomes, not only traces

Holdstock-Brown and Bronsdon discuss the infrastructure for agents that retain state over months: durable execution, sub-agent orchestration, sandboxes and the growing API load generated by agents. Holdstock-Brown argues for evaluating production behavior using outcome events, such as whether a review passed or a change was merged, alongside conventional agent evaluations. About 53 minutes. Publisher-linked segments cover agent-driven API usage at 9:12, load compounding at 13:16, and outcome-event monitoring at 15:52.

Assessment

A practical complement to benchmark papers: it asks what a deployed system can observe cheaply and continuously, and where an outcome proxy is more useful than another model judging a whole trajectory.

Catch-up: published November 12; not covered in the November 12 through 17 reports. Added November 18 during recovery of omitted listening.

Editorial synthesis

A final score hides the route to it. Today's evaluations trace intermediate patches, separate hindsight channels, and count whether a proposed correction can ever change enough outcomes to be measured. The coordination studies make a related distinction: finding a problem is different from preserving its evidence and reorganizing around it. More monitoring or more workers can make the system worse when those handoffs are not measured.

Editorial interpretation of the cited sources.