Measuring and Mitigating Solution Mode Collapse in RLVR

Suppose you ask an AI to solve a math problem. It gets the answer right. Then you ask it to find another way to solve it, and it gives you essentially the same solution. That's a little disappointing. There might be five perfectly good ways to solve the problem, but you're only seeing one.

But if we're using AI to explore what's possible, getting one right answer isn't always enough. For example, a student may need another explanation, a programmer might want an alternative implementation, or an architect may need other designs that satisfy the same constraints. We need AI systems that can help us find alternatives, not just a single right way.

One puzzle, six correct answers

Consider a triangle with three numbered corners: node 1 at the top, node 2 at the bottom left, and node 3 at the bottom right. Color the corners with blue, orange and teal so that connected corners have different colors. Every pair is connected. How many different valid colorings are there?

Solution. There are three choices for node 1, two remaining choices for node 2, and one for node 3: 3 × 2 × 1 = 6 colorings. For example, blue / orange / teal and orange / blue / teal are both valid in node order 1 / 2 / 3. Because the nodes are numbered, swapping their colors counts as a different solution.

An AI that always returns coloring A can be perfectly correct while leaving five valid alternatives unexplored.

This example separates two questions: (1) How often is the model able to answer correctly? (2) How many different correct solutions can the AI produce?

Correctness does not tell the whole story

This is the problem of mode collapse: when an AI’s outputs become concentrated on a few solutions. Here, a “mode” means a distinct answer or approach, defined by the task. The illustrative graph below shows two AIs with 90% accuracy. One spreads correct answers evenly across A–F; the other mostly repeats A.

We found this pattern across all seven frontier AIs we tested, spanning GPT, Claude, Grok, Kimi and DeepSeek. In our recent study, we tested models on puzzles with multiple verifiable solutions, including graph coloring. Across these AIs, correct answers clustered around a few solutions.

For example, we asked OpenAI’s GPT-5.6 Sol to solve 128 graph-coloring problems eight times each. All 1,024 answers were correct. But each problem had 4–12 valid colorings, and the AI found only 1.16 distinct solutions per problem on average. When we compared pairs of answers to the same problem, 95.1% repeated a coloring.

Earlier research found that training AI with human feedback made answers less varied on summarization and instruction-following tasks. Other researchers saw a similar loss of variety in writing tasks, as after AI edits, different people’s essays became more alike in style and meaning. Our puzzles give us a way to measure the gap between the answers the model gives and the set of correct solution modes.

Why solutions can become concentrated

Why does mode collapse happen? We suspect that part of the problem comes from how we train these models. In the setup we study, the AI first generates answers and then a checker scores those answers. The AI is then updated based on the checker-provided scores. However, a checker that only rewards correctness gives every valid solution the same score. It doesn’t care whether the answer is new or one that the AI has given a thousand times.

The training loop reinforces this problem. If the AI already tends to give one solution, that solution shows up more often in training and, as a result, gets more opportunities to be reinforced. As Explainer figure 4 shows, an answer that gets ahead early in training can keep becoming more prevalent until other correct solutions rarely show up.

More attempts help, but preferences persist

If we ask the model more times, does this solve the problem? Sometimes. But extra attempts draw from the AI’s pre-existing preferences. If one alternative has a one-in-a-thousand chance on each independent attempt, even 100 attempts give only about a 9.5% chance of seeing it.

Both toy AIs are correct on 90% of independent attempts. One spreads its correct answers equally across A–F; the other assigns 97% to A and 0.6% to each alternative.

Explainer figure 5 shows how an AI’s underlying answer distribution affects how many different correct solutions it finds over repeated attempts. The curves compare an evenly spread AI with one that mostly repeats the same solution.

Try it: same accuracy, different variety

Both toy models are correct 90% of the time. Change how their correct answers are distributed, then increase the sampling budget.

Even across six modes97% on one mode

Independent attempts: 8

With eight attempts: about 1.21 distinct correct modes under concentration, versus 4.37 under even sampling.

Analytical illustration, not a model run or a measured frontier-model result. Expected distinct modes = Σ[1 − (1 − 0.9q)ⁿ], where q is a mode’s conditional probability given a correct answer. Percentages in the bars are conditional on correctness.

Keeping alternatives in the training data

Our study asks whether we can prevent mode collapse. We ask, when an AI discovers a different way to solve a problem, can we make sure that the solution remains part of the AI’s training data? We call this approach mode replay. We save one example of each new correct solution, then make sure the AI practices it as often as the others. In our experiments, the AI got more answers right and found more ways to solve the problems. We can’t replay a solution it never finds, though. And the rules are much clearer for these puzzles than for something like writing, where two answers can look different and still say the same thing.

Sometimes you need a second explanation before a problem makes sense. Or your code works, but you want to see whether another implementation uses less memory or is easier to read. An architect can have a good design and still need another one when the budget changes. The point of asking for alternatives isn't just to get another answer. It's to have something to compare, so we can decide for ourselves what to do next.

Explainer figures 1, 2, 4 and 5 are explanatory illustrations. Explainer figure 3 reports results from our study.

What does ModeBench count as a different solution?

ModeBench pairs a correctness verifier with a task-specific mode identity. Two valid outputs count as the same mode when the verifier assigns them the same identity. A different wording alone is insufficient. Depending on the domain, a mode can mean a different valid answer or a different computational route to the same answer.

The five domains cover graph coloring, Countdown arithmetic, executable Python, algebraic action sequences (MathIR), and constrained planning (PantryPlan). The benchmark guide documents their validators and mode definitions.

The paper reports success and diversity separately. pass@K asks whether K attempts include a correct answer. Pairwise correct-mode diversity (PCMD) asks how often two correct responses belong to different modes. If all correct responses use one mode, PCMD is zero; it is undefined when there is insufficient correct-answer support. See Section 2.1 and Appendix A for estimators and eligibility rules.

How does Re:Max preserve discoveries?

For each prompt, retain a verified exemplar of each mode the policy discovers. Rehearse the retained modes uniformly alongside fresh reinforcement learning updates. This gives a rare discovery a continued training signal instead of allowing frequent modes to dominate the replay distribution. Re:Max combines this mechanism with MaxRL; the maintained Re:Dr variant uses Dr.GRPO.

The replay bank is built from generated, verified answers. It is not a catalogue of every possible solution, and it cannot rehearse a mode that has never been discovered. The Re:Max guide distinguishes maintained training recipes, saved-result reproduction, and summary-only historical comparisons.

What should we conclude—and what remains open?

The frontier-model comparison describes concentration under a particular evaluation protocol; it does not establish how those proprietary models were trained. The training experiments establish results for the tested tasks, objectives, and scales. They do not guarantee that replay preserves every mode or that the same verifier-based approach transfers directly to open-ended writing.

The reported frontier example uses 128 Level-3 graph-coloring prompts with eight responses per prompt. Those observed values are separate from the toy distributions above. Likewise, graph-coloring mode identity treats the numbered nodes as distinct; changing the equivalence rule would change the diversity question.

For a technical starting point, read ModeBench and the metrics, the replay method, and the experiments.

@inproceedings{rlvrmodecollapse2026,
  title = {Measuring and Mitigating Solution Mode Collapse in RLVR},
  author = {Liv G. d'Aliberti and Marwa Abdulhai and Sofiia Druchyna and Peter Henderson and Manoel Horta Ribeiro},
  booktitle = {The 6th Workshop on Mathematical Reasoning and AI at NeurIPS 2026},
  year = {2026},
  url = {https://arxiv.org/abs/2610.11064}
}