Report for

Choose a day

AI agent cooperation From the report of

One malicious delegate, replayed incidents and the cost of compaction

  • Authors’ claims

After compaction, the best model keeps only 68.5% of solved tasks

Chen and colleagues report CompAct-Bench, which scores working-context compaction by its downstream effect: take trajectories the executor solves in three pre-compaction screening runs, compress the history to a controlled 10% character-level retention target, and let the same executor continue. Across twelve frontier models as compactors on 200 instances spanning BrowseComp, SWE-bench and an internal CompanyBench, average post-compaction accuracy falls to 56.7%, and the best compactor, Claude Opus 5, reaches 68.5% overall, with coding at 81.2% far above workspace tasks at 40.0%. Failure analysis separates omissions, which cause repeated work and budget exhaustion, from state distortion, which turns tentative evidence into conclusions and triggers premature termination.

Adapted from a source figure Table 2, complete results table on the 200-instance evaluation set, caption excluded as a crop adaptation; proportions preserved and whitespace padded to 1200x600.Full resolution
Post-compaction task accuracy for twelve models across Search, Coding, Workspace and Overall columns. Claude Opus 5 leads overall at 68.5%; coding stays near 67 to 81% for all models while workspace drops as low as 8.0%.
Adapted from a source figure: Jiayi Chen et al., CompAct-Bench, arXiv 2611.25673, Table 2, complete results table on the 200-instance evaluation set, caption excluded as a crop adaptation; proportions preserved and whitespace padded to 1200x600.. License: CC BY 4.0

Assessment

This prices the alternative to structured handoffs like yesterday's semantic context handoff: compress a solved trajectory and even the strongest model loses about a third of task completion. Workspace tasks, not coding, are where compaction hurts most, which is where long-horizon personal agents actually live.

Submitted November 19; appeared in the November 20 listing.