Thesis

AI Evaluation Is Moving Toward Adversarial Question Generation

SepaRank has models generate and answer challenges, replacing a fixed question bank with one-to-many separation and calibration scoring. The evaluation bottleneck may be shifting to protocol design.

By All We Need Editorial System

AI evaluation is developing a testable new direction. As fixed human-written question banks become harder to extend quickly enough to separate frontier models, the core asset may shift from additional static questions to protocols that let models design challenges, distinguish rivals, and expose calibration failures. SepaRank is not a final solution, but it turns that idea into an operational experiment.

Question Banks Are Becoming a Constraint

Traditional benchmarks rely mainly on questions written in advance by people, then rank models through accuracy or preference votes. Chatbot Arena extends that model through anonymous pairwise comparisons and has collected more than 240,000 votes. HLE uses roughly 2,500 multidisciplinary questions contributed by nearly 1,000 experts. Both remain useful, but question creation, review, and contamination control depend on scarce human labor and naturally move more slowly than model development.

SepaRank Changes Who Gets to Ask

SepaRank uses a one-to-many protocol in which participating models answer questions and generate challenges intended to separate a target model from its peers. Variance, Brier loss, and adaptive weighting score discrimination and calibration. The paper reports experiments involving 11 models across 10 games of 20 rounds each. The mechanism is not merely mutual testing: it rewards challenges that consistently reveal capability differences while also measuring whether models state commitments and confidence honestly.

Dynamic Protocols Can Also Be Gamed

The evidence does not show that fixed benchmarks have become obsolete. HLE finds substantial remaining headroom on carefully reviewed expert questions, while most evaluated models recorded RMS calibration errors above 70%. MathDuels had already used model-generated mathematics problems, an independent verifier, and a Rasch model, so SepaRank extends an existing research direction rather than creating one from scratch. Collusion, invalid questions, and opponent-specific overfitting could all make a dynamic ranking diverge from general capability.

What to watch next

The decisive test is whether independent teams can reproduce the ranking on models excluded from the original experiment and whether the results correlate with HLE, Chatbot Arena, and real-world task performance. Stable rankings across different participants, prompts, and round counts—without rising rates of invalid questions or dishonest commitments—would strengthen the case. Otherwise, adversarial psychometrics may prove to be another game that models learn to optimize.

Sources