Research explained · 2026
Behavior-Consistent Deep Reinforcement Learning
Can independently trained reinforcement learning agents learn to behave similarly?
Workshop on Automated Reinforcement Learning at RLC 2026 · Accepted to AutoRL 2026
The finding
Similar returns do not imply similar behavior. Q-value Expectile Disagreement (QED) adapts the entropy temperature using critic disagreement to encourage consistency across training runs. On 18 continuous-control tasks, suitable settings reduced cross-run policy divergence by about two orders of magnitude while retaining comparable returns; stronger constraints introduce performance and sample-efficiency tradeoffs.
Two successful agents can still behave differently
Train the same reinforcement learning algorithm several times with different random seeds. Even when the resulting policies earn similar rewards, they can choose different actions and produce different trajectories. Return alone does not describe how reproducible their behavior is.
Behavior-Consistent Deep Reinforcement Learning formalizes a complementary objective: learn policies that perform well and have similar behavior distributions across independent training runs.
A policy may remain stochastic while satisfying this objective. Consistency across runs is different from demanding a deterministic action sequence within every rollout.
The last quantity summarizes the paper’s reported two-order-of-magnitude reduction, not a guarantee for every task, temperature, or constraint strength. The main experiments report returns, symmetric KL, uncertainty intervals, and tradeoffs together.
Why might maximum-entropy RL help?
Maximum-entropy reinforcement learning balances reward with a preference for broader action distributions. In the paper’s theoretical analysis, this can anchor independently learned Boltzmann policies to a common uniform prior. Increasing temperature can reduce the extent to which differences in learned Q-values produce different action distributions.
For the Boltzmann policies considered in the analysis, choosing temperature in proportion to disagreement between Q-functions bounds pairwise policy KL divergence.
That is an assumption-specific theoretical result. It is not an unconditional guarantee for arbitrary neural policies trained with the practical algorithm.
An intuition: bring two action distributions toward a shared prior
Start with two toy policies that prefer opposite actions. Increase their mixture weight on the same uniform distribution.
Conceptual illustration using linear probability mixtures. It is not Boltzmann temperature tuning, a QED implementation, or a model of the observed performance tradeoff. It shows only how the chosen distributions’ KL changes. One-way KL is displayed; the paper evaluates pairwise symmetric KL.See behavior-consistent RL and high-temperature maximum-entropy RL for the formal definitions and assumptions.
What is Q-value Expectile Disagreement?
QED is a practical, state-dependent entropy-temperature schedule. Directly comparing many independently trained policies during each learning run would be costly.
Instead, QED uses disagreement between two critics within a single run as a proxy for uncertainty that can lead to divergence across runs.
The schedule adapts how strongly the algorithm encourages entropy across states. Its expectile-based construction and treatment of off-policy error matter: simply turning up a fixed entropy coefficient can damage learning.
The algorithm section explains the critic-disagreement proxy and why extrapolation outside the replay distribution can become unstable.
What improved, and what are the tradeoffs?
On the 18-task dm_control suite, applying QED to SAC-LN and MAD-SAC reduced pairwise policy divergence by about two orders of magnitude at appropriate settings while preserving comparable normalized return. The analysis uses independently trained runs and explicitly measures behavioral differences, rather than inferring consistency from reward curves alone.
The consistency–performance tradeoff is important.
Very strong constraints can lower returns and slow learning. The paper reports that behavior depends on the base algorithm and the schedule parameter; settings that work well for MAD-SAC are not automatically appropriate for a less stable learner. Reduced return variance is also algorithm-dependent.
| Quantity | What to inspect |
|---|---|
| Return | Does the agent still solve the control task well? |
| Inter-run policy divergence | Do independent training runs produce similar action distributions? |
| Return variability | How stable is performance across independent runs? |
| Sample efficiency | How much interaction is needed to reach the desired behavior and performance? |
The evaluation protocol measures policies on each method’s induced states. That choice is part of the interpretation: differences between methods can reflect both policy differences and the states they visit.
When is behavioral consistency useful?
This objective is relevant when repeated training should yield comparable policies, when debugging a change that alters behavior without much change in return, or when deployment decisions depend on how an agent acts rather than only its aggregate score.
Behavioral consistency is a measurement objective, not a general safety certificate.
Two policies can behave similarly while both failing at a task, and diverse strategies may be desirable in other settings. Our RLVR mode-collapse work asks a different question: whether one model retains multiple correct solution modes. Consistency across training runs and diversity among a model’s valid solutions should not be conflated.
Scope and limitations
The experiments concern continuous control, the tested algorithms, and the chosen evaluation distribution. The critic-disagreement signal is a practical proxy rather than direct access to all cross-run variation. The Boltzmann-policy bound and the empirical QED method have different assumptions and should be interpreted separately.
Read the paper’s limitations for the limits of the evaluation. The earlier similar-behavior repository contains project experiments, but its README predates the current QED paper; use the paper’s algorithm, settings, and appendices as the authoritative specification.
Paper abstract
Reinforcement learning (RL) often exhibits high variance across training runs, leading to unreliable performance and posing a major challenge to deployment in real-world domains. In this work, we address the challenge of cross-run policy divergence by formalizing the problem of behavior-consistent RL , where the objective is to obtain policies that are both high-performing and distributionally similar across training runs. Our key observation is that maximum-entropy RL provides a direct mechanism for controlling behavioral divergence by anchoring runs to a common (uniform) prior. We prove that, for Boltzmann policies, choosing the temperature proportional to Q -function disagreement bounds the pairwise KL divergence between the induced policies. However, we also show that naïvely increasing entropy might impair policy optimization while amplifying off-policy error. Building upon these observations, we propose Q -value Expectile Disagreement (QED), a state-dependent temperature schedule that uses double-critic disagreement as a single-run proxy for cross-run disagreement. Empirically, we demonstrate that across 18 continuous-control tasks, QED reduces across-run divergence by two orders of magnitude without sacrificing performance, resulting in a considerable reduction in return variance at modest sample-efficiency costs.
Cite this work
Please cite the research paper. This page is an accompanying explanation.
Show BibTeX citation
@inproceedings{behaviorconsistentrl2026,
title = {Behavior-Consistent Deep Reinforcement Learning},
author = {Marcel Hussing and Liv G. d'Aliberti and Claas Voelcker and Benjamin Eysenbach and Eric Eaton},
booktitle = {Workshop on Automated Reinforcement Learning at RLC 2026},
year = {2026},
url = {https://arxiv.org/abs/2605.21214}
}Explanation updated . Publication details and experimental scope are stated above.