Consider a familiar pattern: a frontier model achieves a new state-of-the-art score on a widely-used benchmark, and the result is celebrated across the AI community as evidence of rapid progress. But within weeks, independent researchers raise concerns that significant portions of the test set may have appeared in the model's training data. The score, once a headline, becomes a cautionary tale. This pattern has played out repeatedly since 2023, and it reveals something deeper about how we measure progress.

This episode is not an anomaly. It is the logical endpoint of a system in which evaluation has become a bottleneck rather than a compass. Static benchmarks, for all their contributions to the field, are buckling under the weight of modern AI development. They were designed for a world of slower iteration, smaller models, and more constrained capabilities. That world no longer exists.

At Etude AI, we believe that evaluation is not merely a measurement problem. It is a design problem. The way we evaluate AI systems shapes what gets built, how resources get allocated, and which capabilities are prioritized. If the instruments are flawed, the entire feedback loop degrades. This essay explores why static benchmarks are failing, what dynamic evaluation might look like, and how a practice-based approach could chart a path forward.

The Cracks in Static Benchmarks

The standard model of AI evaluation goes something like this: a research team curates a dataset, defines a metric, publishes a paper, and the benchmark becomes a shared yardstick for the field. MMLU measures knowledge breadth. HumanEval tests code generation. MATH evaluates quantitative reasoning. Each serves a purpose, and collectively they have driven enormous progress. But several structural weaknesses have become impossible to ignore.

Data contamination

The most immediate problem is contamination. Modern language models are trained on trillions of tokens scraped from the open web, and benchmark datasets are part of that web. When a model has seen the test questions during training, its performance on those questions tells us very little about genuine capability. It is the difference between a student who understands calculus and one who has memorized the answer key.

Contamination is not always intentional. In many cases, benchmark items propagate through blog posts, forums, and educational materials long before they appear in a training corpus. The sheer scale of modern pretraining data makes it practically impossible to guarantee that no overlap exists. Decontamination techniques help, but they are imperfect and often applied inconsistently.

Goodhart's law

There is a deeper issue at play. When a measure becomes a target, it ceases to be a good measure. This principle, known as Goodhart's law, has become the defining challenge of AI benchmarking. Labs optimize for benchmark scores because those scores drive funding, media coverage, and recruitment. The result is a perverse incentive structure in which evaluation artifacts are gamed rather than genuinely solved.

Consider code generation benchmarks. HumanEval, introduced by OpenAI in 2021, consists of 164 hand-written Python programming problems. It was a valuable contribution at the time, but its small size and well-defined structure have made it vulnerable to overfitting. Models can be fine-tuned on problems structurally similar to HumanEval items, inflating scores without meaningfully improving general programming ability. The benchmark's signal has eroded even as its prominence has grown.

When a measure becomes a target, it ceases to be a good measure. This is the defining challenge of AI benchmarking. Goodhart's Law, adapted

Narrow scope and brittleness

Static benchmarks also suffer from a fundamental mismatch with the capabilities they aim to measure. Real-world tasks are open-ended, multi-step, and context-dependent. A benchmark, by necessity, is constrained. It must have a fixed set of questions, a defined scoring rubric, and reproducible evaluation conditions. These constraints make it a poor proxy for the kinds of complex, messy problems that AI systems are increasingly being deployed to solve.

A model that excels at multiple-choice knowledge questions may still struggle to synthesize information across sources, handle ambiguity gracefully, or adapt its reasoning when presented with novel problem structures. The gap between benchmark performance and real-world utility is not a minor calibration issue. For many applications, it is a chasm.

The Case for Dynamic Evaluation

If static benchmarks are the sheet music, dynamic evaluation is the live performance. It introduces variability, responsiveness, and a feedback loop between the evaluator and the evaluated. Several threads of research point toward what this might look like in practice.

Adversarial generation

One approach is to generate evaluation items adversarially, using one model to probe the weaknesses of another. This is conceptually similar to red-teaming, but applied systematically to capability assessment. Instead of relying on a fixed test set, the evaluation harness creates new challenges designed to be maximally informative about a model's boundaries.

Adversarial evaluation has the attractive property of being contamination-resistant by construction. If items are generated fresh for each evaluation run, there is no test set to leak into training data. It also naturally adapts to the frontier: as models improve, the adversary generates harder challenges.

The challenge, of course, is ensuring that adversarially generated items are fair and meaningful. A sufficiently creative adversary could produce pathological edge cases that no reasonable system should be expected to handle. The art is in constraining the adversary to produce items that are difficult but genuine, probing real capabilities rather than exploiting implementation quirks.

Adaptive difficulty

A related idea is adaptive testing, borrowed from psychometrics and educational assessment. In computerized adaptive testing (CAT), the difficulty of each question is calibrated to the test-taker's estimated ability. A student who answers an easy question correctly receives a harder one next; a student who struggles receives an easier one. This approach yields more precise ability estimates with fewer questions.

Applied to AI evaluation, adaptive difficulty could dramatically improve efficiency. Instead of running a model through thousands of items at all difficulty levels, an adaptive harness could zero in on the model's capability boundary in far fewer steps. This is particularly valuable when evaluation compute is expensive, as with large-scale agent evaluations or multi-turn dialogue assessments.

# Conceptual: adaptive evaluation loop
def adaptive_evaluate(model, item_bank, n_rounds=50):
    ability_estimate = 0.0
    history = []
    for _ in range(n_rounds):
        # Select item near estimated ability boundary
        item = select_item(item_bank, target_difficulty=ability_estimate)
        response = model.generate(item.prompt)
        correct = score(response, item.rubric)
        history.append((item, correct))
        # Update ability estimate via IRT model
        ability_estimate = update_estimate(history)
    return ability_estimate, history

Temporal freshness

A third dimension of dynamic evaluation is temporal. The world changes, and so should our benchmarks. A model evaluated on knowledge questions from 2023 may appear knowledgeable simply because its training data includes the answers. Questions that reference recent events, emerging scientific findings, or evolving cultural contexts are inherently harder to contaminate and more relevant to real-world deployment.

Temporally fresh evaluation also addresses a subtler problem: the implicit assumption that the skills being measured are stable over time. The tasks that matter for AI systems in 2026 are not the same as those that mattered in 2022. Evaluation frameworks must evolve alongside the systems they assess and the applications they are deployed in.

Ecological Validity: Benchmarks That Reflect Reality

Dynamic evaluation addresses the contamination and staleness problems, but there is a more fundamental question: are we measuring the right things? Ecological validity, a concept from psychology and human factors research, asks whether the conditions of a test reflect the conditions of real-world performance.

Most AI benchmarks have low ecological validity. They present problems in isolation, with clean inputs, well-defined outputs, and no time pressure or resource constraints. Real deployment is messier. A coding assistant must navigate ambiguous requirements, integrate with existing codebases, handle interruptions, and explain its reasoning. A medical AI must cope with incomplete patient histories, conflicting evidence, and the need to communicate uncertainty to clinicians who may not have AI expertise.

The gap between benchmark performance and real-world utility is not a minor calibration issue. For many applications, it is a chasm.

Improving ecological validity means building evaluation scenarios that capture this complexity. Multi-turn interactions rather than single-shot prompts. Noisy and incomplete inputs rather than clean, curated ones. Tasks that require coordination across tools, retrieval of relevant context, and graceful handling of ambiguity. These are harder to standardize, harder to score, and harder to scale. But they are far more informative about actual system capability.

The agent evaluation paradigm is a step in this direction. Benchmarks like SWE-bench, which asks models to resolve real GitHub issues, and WebArena, which tests web navigation in realistic browser environments, move beyond isolated question-answering toward situated task completion. They are imperfect, but they represent a meaningful shift in the field's understanding of what evaluation should look like.

The Tension Between Reproducibility and Validity

There is a genuine tension at the heart of this conversation. The scientific method depends on reproducibility. If an experiment cannot be repeated with the same results, its conclusions are suspect. Static benchmarks are reproducible almost by definition: the same questions, the same scoring, the same conditions. This is a feature, not a bug.

Dynamic evaluation, by contrast, introduces variability. If the test items change with every run, how do we compare results across time or across labs? If the difficulty adapts to the model, how do we rank models against each other? These are not trivial objections. A field that abandons reproducibility risks losing its empirical foundation.

The resolution, we believe, lies not in choosing one side but in designing evaluation systems that accommodate both concerns. Several strategies are promising:

  • Anchored dynamic benchmarks. Include a small, fixed "anchor" set alongside dynamically generated items. The anchor provides cross-session comparability; the dynamic items provide contamination resistance and freshness.
  • Statistical equivalence. Rather than reusing identical items, generate items from distributions with verified statistical properties. If two test forms are drawn from the same difficulty distribution, they are comparable even if the specific items differ.
  • Evaluation protocols, not just datasets. Publish the generation and scoring algorithms, not just the data. This allows anyone to reproduce the evaluation process while generating fresh items.
  • Versioned releases. Publish periodic snapshots of dynamic benchmarks, providing fixed reference points while maintaining the ability to update between releases.

None of these approaches fully resolves the tension, but together they offer a framework for evaluation that is both rigorous and adaptive. The goal is not to replace reproducibility but to expand our definition of it.

Practice-Based Evaluation: The Etude AI Approach

Our name is not accidental. In music, an etude is a composition designed to develop a specific technical skill through focused practice. It is both a test and a training exercise, a piece that reveals ability in the process of building it. We believe AI evaluation should work the same way.

Practice-based evaluation, as we define it, has three core properties:

Iterative. Evaluation is not a one-time event but an ongoing process. Each round of evaluation generates information that shapes the next round. Models are not simply scored; they are studied, with evaluation data feeding back into our understanding of their strengths and weaknesses.

Self-improving. The evaluation system itself learns and adapts. As new failure modes are discovered, new evaluation items are generated to probe them. As models develop new capabilities, the benchmark evolves to assess them. The evaluation harness is a living system, not a static artifact.

Capability-mapped. Rather than producing a single aggregate score, practice-based evaluation generates a detailed capability profile. Like a music teacher who can identify that a student's arpeggios are strong but their sight-reading needs work, our evaluations aim to decompose performance into meaningful, actionable dimensions.

In music, an etude is both a test and a training exercise. We believe AI evaluation should work the same way.

In practice, this means building evaluation pipelines that combine fixed anchor items with adversarially generated challenges, adaptive difficulty calibration, and detailed performance decomposition. We use item response theory models to estimate latent ability along multiple dimensions, and we continuously validate our evaluation items against real-world task performance to ensure ecological validity.

This is more complex than running a model through a fixed test set and reporting a number. But complexity is warranted. The systems we are evaluating are themselves complex, and our instruments must match that complexity to remain informative.

Future Directions

The ideas outlined here are not speculative. Many of the building blocks already exist. But assembling them into a coherent evaluation ecosystem requires sustained effort and community coordination. We see several promising directions for the field.

Living benchmarks

A living benchmark is one that updates itself continuously, drawing on a stream of fresh data, community contributions, and automated item generation. Unlike a static benchmark that decays as models are trained on its contents, a living benchmark maintains its discriminative power indefinitely. The challenge is governance: who decides what goes in, how quality is maintained, and how changes are communicated to the community.

We are particularly interested in benchmarks that draw on naturally occurring data streams. Open-source bug reports, newly published scientific papers, recent legislative changes, and emerging code patterns all provide raw material for evaluation items that are inherently temporal and resistant to contamination. The key is building the infrastructure to transform these raw signals into calibrated, scored evaluation items at scale.

Community-driven evaluation

The current model of benchmark development is centralized: a research lab creates a benchmark, publishes it, and the community uses it. This model has scaling problems. No single lab can cover the full breadth of capabilities that matter, and the lag between benchmark creation and adoption creates a window during which evaluation is effectively blind.

A more distributed model would allow domain experts, practitioners, and researchers to contribute evaluation items, scenarios, and scoring rubrics. A clinician might contribute medical reasoning cases that test diagnostic skill. A software architect might contribute system design problems that require trade-off analysis. A linguist might contribute translation challenges that probe cultural nuance.

Crowdsourced evaluation has its own challenges, including quality control, adversarial submissions, and coordination overhead. But platforms like Chatbot Arena have demonstrated that community-driven evaluation can produce rankings that are both robust and widely trusted. The next step is extending this model beyond preference ranking to structured capability assessment.

Evaluation as a service

If evaluation is too important to be static and too complex to be done ad hoc, perhaps it should be treated as infrastructure. Evaluation-as-a-service platforms would provide standardized, up-to-date evaluation pipelines that any developer can access. Rather than each lab maintaining its own evaluation suite, a shared service could ensure consistency, freshness, and breadth.

This model also opens the door to evaluation composability. A developer building a medical chatbot could combine a general language capability assessment with a specialized medical reasoning module and a safety evaluation layer, all running through a unified pipeline. The evaluation itself becomes modular, matching the increasingly modular nature of AI system development.

The history of science is, in large part, a history of measurement. The telescope did not just observe the heavens; it changed our understanding of what was there to observe. The microscope did not just magnify cells; it revealed an entire domain of biology that had been invisible. Evaluation plays an analogous role in AI. The way we measure these systems shapes our understanding of what they can do, what they cannot do, and what they might do next.

Static benchmarks served the field well in an era of slower progress and smaller models. But the pace of AI development has outstripped the pace of evaluation innovation. Scores saturate, datasets leak, and the gap between measured and actual capability grows wider. Closing that gap requires a fundamental rethinking of how evaluation works: not as a snapshot, but as a process; not as a fixed test, but as a living instrument; not as a single number, but as a rich, multidimensional portrait of capability.

At Etude AI, we are building the tools and frameworks to make this vision real. We believe that better evaluation is not just a technical improvement. It is a prerequisite for building AI systems that are genuinely capable, reliably safe, and meaningfully aligned with the complex, messy, ever-changing world they are meant to serve.

The etude is not the performance. It is the practice that makes the performance possible.