Zur Hauptnavigation wechseln Zur Suche wechseln Zum Hauptinhalt wechseln

Hierarchical Cooperative Multi-Agent Reinforcement Learning with AlphaZero Skill Execution

  • Université de Liège

Publikation: Beitrag in Buch/Bericht/Konferenzband › Konferenzbeitrag › Begutachtung

Abstract

Cooperative multi-agent reinforcement learning (MARL) for team-level tactical problems requires agents that can both discover reusable behavioral abstractions (skills) and execute each skill with primitive actions. We present \emph{AZ-HMARL}, a hierarchical framework that assigns discrete skills via QMIX and executes them with a per-agent AlphaZero planner operating on real environment dynamics. To prevent skill collapse, we derive an intrinsic reward that maximizes a variational lower bound on the mutual information $\mathrm{MI}(z, \tau)$ between skills and trajectories. A segment-level decoder $p_\phi(z|\tau)$ is trained to predict the assigned skill $z$ from the observed trajectory $\tau$, and its log-probability serves as the intrinsic reward.

Integrating this segment-level reward with AlphaZero introduces a structural reward inconsistency: the decoder reward is unavailable during MCTS tree search, creating a mismatch between the tree backup and the value-head training target. We formalize this inconsistency through a value consistency gap metric and propose a step-level entropy proxy that is fully MCTS-consistent. Experiments on several SMAX scenarios demonstrate that the entropy proxy achieves superior win rates and lower value inconsistency compared to the decoder baseline, though with reduced skill discriminability. In heterogeneous teams, we observe emergent role specialization where different unit types execute different skills. This arises purely from the intrinsic reward signal without explicit role supervision.
OriginalspracheEnglisch
TitelMulti-Agent Systems - 23rd European Conference, EUMAS 2026, Proceedings
Untertitel23rd European Conference, EUMAS 2026, Malmö, Sweden, September 23-25, 2026, Proceedings
Redakteure/-innenFabian Lorig, Paul Davidsson, Franziska Klügl, Valérie Camps, Juan Carlos Nieves Sanchez, Elsy Kaddoum
Herausgeber (Verlag)Springer Science and Business Media Deutschland GmbH
Seiten54-72
Seitenumfang19
BandLNCS Volume 16829
ISBN (elektronisch)978-3-032-39395-1
ISBN (Print)978-3-032-39394-4
DOIs
PublikationsstatusVeröffentlicht - 23 Sept. 2026
Veranstaltung23rd European Conference on Multi-Agent Systems, EUMAS 2026 - Malmo, Schweden
Dauer: 23 Sept. 2026 → 25 Sept. 2026

Publikationsreihe

NameLecture Notes in Computer Science
Band16829 LNCS
ISSN (Print)0302-9743
ISSN (elektronisch)1611-3349

Konferenz

Konferenz23rd European Conference on Multi-Agent Systems, EUMAS 2026
Land/GebietSchweden
OrtMalmo
Zeitraum23/09/26 → 25/09/26

Fingerprint

Untersuchen Sie die Forschungsthemen von „Hierarchical Cooperative Multi-Agent Reinforcement Learning with AlphaZero Skill Execution“. Zusammen bilden sie einen einzigartigen Fingerprint.

Dieses zitieren