Résumé
Cooperative multi-agent reinforcement learning (MARL) for team-level tactical problems requires agents that can both discover reusable behavioral abstractions (skills) and execute each skill with primitive actions. We present \emph{AZ-HMARL}, a hierarchical framework that assigns discrete skills via QMIX and executes them with a per-agent AlphaZero planner operating on real environment dynamics. To prevent skill collapse, we derive an intrinsic reward that maximizes a variational lower bound on the mutual information $\mathrm{MI}(z, \tau)$ between skills and trajectories. A segment-level decoder $p_\phi(z|\tau)$ is trained to predict the assigned skill $z$ from the observed trajectory $\tau$, and its log-probability serves as the intrinsic reward.
Integrating this segment-level reward with AlphaZero introduces a structural reward inconsistency: the decoder reward is unavailable during MCTS tree search, creating a mismatch between the tree backup and the value-head training target. We formalize this inconsistency through a value consistency gap metric and propose a step-level entropy proxy that is fully MCTS-consistent. Experiments on several SMAX scenarios demonstrate that the entropy proxy achieves superior win rates and lower value inconsistency compared to the decoder baseline, though with reduced skill discriminability. In heterogeneous teams, we observe emergent role specialization where different unit types execute different skills. This arises purely from the intrinsic reward signal without explicit role supervision.
Integrating this segment-level reward with AlphaZero introduces a structural reward inconsistency: the decoder reward is unavailable during MCTS tree search, creating a mismatch between the tree backup and the value-head training target. We formalize this inconsistency through a value consistency gap metric and propose a step-level entropy proxy that is fully MCTS-consistent. Experiments on several SMAX scenarios demonstrate that the entropy proxy achieves superior win rates and lower value inconsistency compared to the decoder baseline, though with reduced skill discriminability. In heterogeneous teams, we observe emergent role specialization where different unit types execute different skills. This arises purely from the intrinsic reward signal without explicit role supervision.
| langue originale | Anglais |
|---|---|
| titre | Multi-Agent Systems - 23rd European Conference, EUMAS 2026, Proceedings |
| Sous-titre | 23rd European Conference, EUMAS 2026, Malmö, Sweden, September 23-25, 2026, Proceedings |
| rédacteurs en chef | Fabian Lorig, Paul Davidsson, Franziska Klügl, Valérie Camps, Juan Carlos Nieves Sanchez, Elsy Kaddoum |
| Editeur | Springer Science and Business Media Deutschland GmbH |
| Pages | 54-72 |
| Nombre de pages | 19 |
| Volume | LNCS Volume 16829 |
| ISBN (Electronique) | 978-3-032-39395-1 |
| ISBN (imprimé) | 978-3-032-39394-4 |
| Les DOIs | |
| état | Publié - 23 sept. 2026 |
| Evénement | 23rd European Conference on Multi-Agent Systems, EUMAS 2026 - Malmo, Sucde Durée: 23 sept. 2026 → 25 sept. 2026 |
Série de publications
| Nom | Lecture Notes in Computer Science |
|---|---|
| Volume | 16829 LNCS |
| ISSN (imprimé) | 0302-9743 |
| ISSN (Electronique) | 1611-3349 |
Une conférence
| Une conférence | 23rd European Conference on Multi-Agent Systems, EUMAS 2026 |
|---|---|
| Pays/Territoire | Sucde |
| La ville | Malmo |
| période | 23/09/26 → 25/09/26 |
Empreinte digitale
Examiner les sujets de recherche de « Hierarchical Cooperative Multi-Agent Reinforcement Learning with AlphaZero Skill Execution ». Ensemble, ils forment une empreinte digitale unique.Contient cette citation
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver