Research explained
Illustrated guides to Liv d’Aliberti’s research on RLVR solution diversity, aha moments in reasoning models, and behavior-consistent reinforcement learning.
How do we tell whether an AI system reasons reliably? These guides explain three complementary questions from my research, using figures from the papers, concrete examples, and the limits of what each experiment establishes.
Start with the question that matters to you: can one model find multiple correct solutions, does reconsideration improve an answer, or do independently trained agents behave similarly?
Measuring and Mitigating Solution Mode Collapse in RLVR
Can a reasoning model get more answers right while finding fewer ways to solve a problem?
Correctness and solution diversity are different properties. On ModeBench, RLVR can concentrate a model onto fewer correct solution modes even as accuracy holds or improves. Re:Max keeps one verified example per discovered mode and rehearses those modes uniformly, improving success and diversity in the tested settings.
The Illusion of Insight in Reasoning Models
Do reasoning models really have “aha” moments—and do those moments help?
A mid-trace change of approach is not, by itself, evidence of useful self-correction. Across more than a million reasoning traces, spontaneous shifts were rare and seldom improved accuracy. In separate intervention experiments, triggering reconsideration when token entropy was high improved accuracy in the evaluated settings.
Behavior-Consistent Deep Reinforcement Learning
Can independently trained reinforcement learning agents learn to behave similarly?
Similar returns do not imply similar behavior. Q-value Expectile Disagreement (QED) adapts the entropy temperature using critic disagreement to encourage consistency across training runs. On 18 continuous-control tasks, suitable settings reduced cross-run policy divergence by about two orders of magnitude while retaining comparable returns; stronger constraints introduce performance and sample-efficiency tradeoffs.
The guides accompany the research papers. The publication record on each guide includes all authors, the full abstract, the current citation, and links to available code and data. For my other work on AI, reinforcement learning, privacy, and autonomy, see all publications.