Multi-agent training for Pommerman: Curriculum learning and population-based self-play approach
,
,
and
*
Article Sidebar
Full Text: 
PDF
Main Article Content
Abstract
Pommerman is a multi-agent environment that has received considerable attention from researchers in recent years. This environment is an ideal benchmark for multi-agent training, providing a battleground for two teams with communication capabilities among allied agents. Pommerman poses significant challenges for model-free reinforcement learning due to delayed action effects, sparse rewards, and false positives, where opponents can lose because of their own mistakes. This study introduces a system designed to train multi-agent systems to play Pommerman using a combination of curriculum learning and population-based self-play. We also tackle two challenging problems when deploying the multi-agent training system for competitive games: sparse reward and a suitable matchmaking mechanism. Specifically, we propose an adaptive annealing factor, based on agents’ performance, to dynamically adjust the dense exploration reward during training. Additionally, we implement a matchmaking mechanism utilizing the Elo rating system to pair agents effectively. Our experimental results demonstrate that our trained agent can outperform top learning agents without requiring communication among allied agents.
Article Details
Conflict of Interest
© 2026 The authors. This is an open access article distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License.
References
Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I. (2020). Emergent tool use from multi-agent autocurricula. In Proceedings of the International Conference on Learning Representations (ICLR 2020).
Bansal, T., Pachocki, J., Sidor, S., Sutskever, I., & Mordatch, I. (2017). Emergent complexity via multi-agent competition. arXiv preprint arXiv:1710.03748.
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009, June). Curriculum learning. In Proceedings of the 26th annual international conference on machine learning (pp. 41-48). https://doi.org/10.1145/1553374.1553380
Bhati, R., Gottipati, S. K., Mars, C., & Taylor, M. E. (2023). Curriculum learning for cooperation in multi-agent reinforcement learning. arXiv preprint arXiv:2312.11768.
Elo, A. E. (1978). The rating of chessplayers, past and present. Arco Publishing.
Gao, C., Hernandez-Leal, P., Kartal, B., & Taylor, M. E. (2019a). Skynet: A top deep RL agent in the inaugural pommerman team competition. arXiv preprint arXiv:1905.01360.
Gao, C., Kartal, B., Hernandez-Leal, P., & Taylor, M. E. (2019b). On hard exploration for reinforcement learning: A case study in pommerman. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (Vol. 15, No. 1, pp. 24-30).
Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735-1780.
Huh, D., & Mohapatra, P. (2023). Multi-agent reinforcement learning: A comprehensive survey. arXiv preprint arXiv:2312.10256.
Huynh, N.-M. (2020). The strength improvement of computer game Pommerman (Master’s thesis). National Chiao Tung University.
Jiang, Y., Liu, Q., Ma, X., Li, C., Yang, Y., Yang, J., Liang, B., & Zhao, Q. (2023). Learning diverse risk preferences in population-based self-play. arXiv preprint arXiv:2305.11476.
Liu, S., Lever, G., Merel, J., Tunyasuvunakool, S., Heess, N., & Graepel, T. (2019). Emergent coordination through competition. arXiv preprint arXiv:1902.07151.
Matiisen, T., Oliver, A., Cohen, T., & Schulman, J. (2020). Teacher–student curriculum learning. IEEE transactions on neural networks and learning systems, 31(9), 3732-3740.
Osogami, T., & Takahashi, T. (2019). Real-time tree search with pessimistic scenarios. arXiv preprint arXiv:1902.10870.
Panwar, H., Chatterjee, S., & Dube, W. (2021). A fast evolutionary adaptation for MCTS in Pommerman. arXiv preprint arXiv:2111.13770.
Peng, P., Pang, L., Yuan, Y., & Gao, C. (2018). Continual match based training in Pommerman: Technical report. arXiv preprint arXiv:1812.07297.
Resnick, C., Eldridge, W., Ha, D., Britz, D., Foerster, J., Togelius, J., ... & Bruna, J. (2018). Pommerman: A multi-agent playground. arXiv preprint arXiv:1809.07124.
Resnick, C., Gao, C., Márton, G., Osogami, T., Pang, L., & Takahashi, T. (2019). Pommerman & NeurIPS 2018: Multi-Agent Competition. In The NeurIPS'18 Competition: From Machine Learning to Intelligent Conversations (pp. 11-36). Cham: Springer International Publishing.
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., ... & Silver, D. (2020). Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839), 604-609.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
Shelke, O., Meisheri, H., & Khadilkar, H. (2021). School of hard knocks: Curriculum analysis for Pommerman with a fixed computational budget. arXiv preprint arXiv:2102.11762.
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., ... & Hassabis, D. (2017). Mastering the game of go without human knowledge. Nature, 550(7676), 354-359.
Soviany, P., Ionescu, R. T., Rota, P., & Sebe, N. (2021). Curriculum learning: A survey. arXiv preprint arXiv:2101.10382.