Report for

Choose a day

AI agent cooperation From the report of

One malicious delegate, replayed incidents and the cost of compaction

  • Authors’ claims

Where playtesting bots diverge from humans, cell by cell

Smirnov, Pertoft and Karimi report a stratified testing framework that compares full behavioral distributions of humans and agents within contextual cells and then discovers subgroup rules describing where they diverge. In a production case study on a commercial match-3 game with two playtesting agents, session-level metrics dominated by win and loss outcomes carry little signal beyond mean comparisons, while process-level metrics such as move efficiency expose genuine differences: the agents diverge substantially from humans in some game modes and show no detectable divergence in others. The implementation is released as open source.

Adapted from a source figure Figure 2, complete two-panel heatmap, caption excluded; proportions preserved and whitespace padded to 1200x600.Full resolution
Per-cell divergence estimates for two agents (SL and Hbot) against humans on score per move and cascades per move across game modes. Hbot diverges strongly in specific Route cells while SL stays close to human play in most cells.
Adapted from a source figure: Oleg Smirnov, John Pertoft and Sara Karimi, When Agents Play Differently, arXiv 2611.26012, Figure 2, complete two-panel heatmap, caption excluded; proportions preserved and whitespace padded to 1200x600.. License: CC BY-NC-SA 4.0

Assessment

Aggregate human-likeness scores certify the average and miss the mode. Cell-level distributional testing makes 'the agent behaves like a human' falsifiable per context, and the same stratification applies anywhere an agent is graded on outcomes alone.

Submitted November 19; appeared in the November 20 listing.