skip to main content
survey

Explainability in Deep Reinforcement Learning: A Review into Current Methods and Applications

Published: 08 December 2023 Publication History
  • Get Citation Alerts
  • Abstract

    The use of Deep Reinforcement Learning (DRL) schemes has increased dramatically since their first introduction in 2015. Though uses in many different applications are being found, they still have a problem with the lack of interpretability. This has bread a lack of understanding and trust in the use of DRL solutions from researchers and the general public. To solve this problem, the field of Explainable Artificial Intelligence has emerged. This entails a variety of different methods that look to open the DRL black boxes, ranging from the use of interpretable symbolic Decision Trees to numerical methods like Shapley Values. This review looks at which methods are being used and for which applications. This is done to identify which models are the best suited to each application or if a method is being underutilised.

    References

    [1]
    Andrew Anderson, Jonathan Dodge, Amrita Sadarangani, Zoe Juozapaitis, Evan Newman, Jed Irvine, Souti Chattopadhyay, Alan Fern, and Margaret Burnett. 2019. Explaining reinforcement learning to mere mortals: an empirical study. In Proceedings of the 28th International Joint Conference on Artificial Intelligence. 1328–1334.
    [2]
    Wentao Bao, Qi Yu, and Yu Kong. 2021. DRIVE: Deep reinforced accident anticipation with visual explanation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV’21). IEEE, 7599–7608.
    [3]
    Pablo Barros, Ana Tanevska, Francisco Cruz, and Alessandra Sciutti. 2020. Moody learners-explaining competitive behaviour of reinforcement learning agents. In Proceedings of the Joint IEEE 10th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob’20). IEEE, 1–8.
    [4]
    Nicolas Blystad Carbone. 2020. Explainable AI for Path Following with Model Trees. Master’s thesis. NTNU.
    [5]
    Wang Chang, Wu Lizhen, Yan Chao, Wang Zhichao, Long Han, and Yu Chao. 2020. Coactive design of explainable agent-based task planning and deep reinforcement learning for human-UAVs teamwork. Chinese J. Aeronaut. 33, 11 (2020), 2930–2945.
    [6]
    Antoni Climent, Dmitry Gnatyshak, and Sergio Alvarez-Napagao. 2021. Applying and verifying an explainability method based on policy graphs in the context of reinforcement learning. In Artificial Intelligence Research and Development. IOS Press, 455–464.
    [7]
    Francisco Cruz, Richard Dazeley, and Peter Vamplew. 2019. Memory-based explainable reinforcement learning. In Proceedings of the Australasian Joint Conference on Artificial Intelligence. Springer, 66–77.
    [8]
    Francisco Cruz, Richard Dazeley, Peter Vamplew, and Ithan Moreira. 2023. Explainable robotic systems: Understanding goal-driven actions in a reinforcement learning scenario. Neural Comput. Appl. 35, 25 (2023), 18113–18130.
    [9]
    Giang Dao, Wesley Houston Huff, and Minwoo Lee. 2021. Learning sparse evidence-driven interpretation to understand deep reinforcement learning agents. In Proceedings of the IEEE Symposium Series on Computational Intelligence (SSCI’21). IEEE, 1–7.
    [10]
    Arun Das and Paul Rad. 2020. Opportunities and challenges in explainable artificial intelligence (xai): A survey. Retrieved from https://arXiv:2006.11371
    [11]
    Yashesh Dhebar, Kalyanmoy Deb, Subramanya Nageshrao, Ling Zhu, and Dimitar Filev. 2022. Toward interpretable-ai policies using evolutionary nonlinear decision trees for discrete-action systems. IEEE Trans. Cybernet. (2022). DOI:
    [12]
    Zihan Ding, Pablo Hernandez-Leal, Gavin Weiguang Ding, Changjian Li, and Ruitong Huang. 2020. Cdt: Cascading decision trees for explainable reinforcement learning. Retrieved from https://arXiv:2011.07553
    [13]
    Nathan Douglas, Dianna Yim, Bilal Kartal, Pablo Hernandez-Leal, Frank Maurer, and Matthew E. Taylor. 2019. Towers of saliency: A reinforcement learning visualization using immersive environments. In Proceedings of the ACM International Conference on Interactive Surfaces and Spaces. 339–342.
    [14]
    Jeff Druce, Michael Harradon, and James Tittle. 2021. Explainable artificial intelligence (XAI) for increasing user trust in deep reinforcement learning driven autonomous systems. Retrieved from https://arXiv:2106.03775
    [15]
    Jeff Druce, James Niehaus, Vanessa Moody, David Jensen, and Michael L. Littman. 2021. Brittle AI, causal confusion, and bad mental models: Challenges and successes in the XAI program. Retrieved from https://arXiv:2106.05506
    [16]
    Mira Finkelstein, Nitsan Levy Schlot, Lucy Liu, Yoav Kolumbus, Jeffrey Rosenschein, David C. Parkes, and Sarah Keren. 2021. Deep reinforcement learning explanation via model transforms. In Proceedings of the Deep RL Workshop NeurIPS.
    [17]
    Jasmina Gajcin, Rahul Nair, Tejaswini Pedapati, Radu Marinescu, Elizabeth Daly, and Ivana Dusparic. 2022. Contrastive explanations for comparing preferences of reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence.
    [18]
    Suna Sihang Guo, Ruohan Zhang, Bo Liu, Yifeng Zhu, Dana Ballard, Mary Hayhoe, and Peter Stone. 2021. Machine versus human attention in deep reinforcement learning tasks. Adv. Neural Info. Process. Syst. 34 (2021), 25370–25385.
    [19]
    Weisi Guo. 2020. Partially explainable big data driven deep reinforcement learning for green 5G UAV. In Proceedings of the IEEE International Conference on Communications (ICC’20). IEEE, 1–7.
    [20]
    Wenbo Guo, Xian Wu, Usmann Khan, and Xinyu Xing. 2021. Edge: Explaining deep reinforcement learning policies. Adv. Neural Info. Process. Syst. 34 (2021), 12222–12236.
    [21]
    Lei He, Nabil Aouf, and Bifeng Song. 2021. Explainable deep reinforcement learning for UAV autonomous path planning. Aerospace Sci. Technol. 118 (2021), 107052.
    [22]
    Alexandre Heuillet, Fabien Couthouis, and Natalia Díaz-Rodríguez. 2022. Collective explainable AI: Explaining cooperative strategies and agent contribution in multiagent reinforcement learning with shapley values. IEEE Comput. Intell. Mag. 17, 1 (2022), 59–71.
    [23]
    Tobias Huber, Dominik Schiller, and Elisabeth André. 2019. Enhancing explainability of deep reinforcement learning through selective layer-wise relevance propagation. In Proceedings of the Joint German/Austrian Conference on Artificial Intelligence (Künstliche Intelligenz). Springer, 188–202.
    [24]
    Alessandro Iucci, Alberto Hata, Ahmad Terra, Rafia Inam, and Iolanda Leite. 2021. Explainable reinforcement learning for human-robot collaboration. In Proceedings of the 20th International Conference on Advanced Robotics (ICAR’21). IEEE, 927–934.
    [25]
    Theo Jaunet, Romain Vuillemot, and Christian Wolf. 2020. DRLViz: Understanding decisions and memory in deep reinforcement learning. In Proceedings of the Computer Graphics Forum, Vol. 39. Wiley Online Library, 49–61.
    [26]
    Ho-Taek Joo and Kyung-Joong Kim. 2019. Visualization of deep reinforcement learning using grad-CAM: how AI plays atari games? In Proceedings of the IEEE Conference on Games (CoG’19). IEEE, 1–2.
    [27]
    Shirel Josef and Amir Degani. 2020. Deep reinforcement learning for safe local planning of a ground vehicle in unknown rough terrain. IEEE Robot. Autom. Lett. 5, 4 (2020), 6748–6755.
    [28]
    Jakob Karalus and Felix Lindner. 2021. Accelerating the convergence of human-in-the-loop reinforcement learning with counterfactual explanations. Retrieved from https://arXiv:2108.01358
    [29]
    Olivera Kotevska, Jeffrey Munk, Kuldeep Kurte, Yan Du, Kadir Amasyali, Robert W. Smith, and Helia Zandi. 2020. Methodology for interpretable reinforcement learning model for HVAC energy control. In Proceedings of the IEEE International Conference on Big Data (BigData’20). IEEE, 1555–1564.
    [30]
    Mikel Landajuela, Brenden K. Petersen, Sookyung Kim, Claudio P. Santiago, Ruben Glatt, Nathan Mundhenk, Jacob F. Pettit, and Daniel Faissol. 2021. Discovering symbolic policies with deep reinforcement learning. In Proceedings of the International Conference on Machine Learning. PMLR, 5979–5989.
    [31]
    Huiling Li, Jun Wu, Hansong Xu, Gaolei Li, and Mohsen Guizani. 2021. Explainable intelligence-driven defense mechanism against advanced persistent threats: A joint edge game and AI approach. IEEE Trans. Depend. Secure Comput. 19, 2 (2021), 757–775.
    [32]
    Wentian Li, Xidong Feng, Haotian An, Xiang Yao Ng, and Yu-Jin Zhang. 2020. Mri reconstruction with interpretable pixel-wise operations using reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 792–799.
    [33]
    Roman Liessner, Jan Dohmen, and Marco A. Wiering. 2021. Explainable reinforcement learning for longitudinal control. In Proceedings of the International Conference on Agents and Artificial Intelligence (ICAART’21). 874–881.
    [34]
    Guiliang Liu, Oliver Schulte, Wang Zhu, and Qingcan Li. 2019. Toward interpretable deep reinforcement learning with linear model u-trees. In Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD’18). Springer, 414–429.
    [35]
    Guiliang Liu, Xiangyu Sun, Oliver Schulte, and Pascal Poupart. 2021. Learning tree interpretation from object representation for deep reinforcement learning. Adv. Neural Info. Process. Syst. 34 (2021), 19622–19636.
    [36]
    Yang Liu, Xinzhi Wang, Yudong Chang, and Chao Jiang. 2022. Towards explainable reinforcement learning using scoring mechanism augmented agents. In Proceedings of the 15th International Conference on Knowledge Science, Engineering and Management (KSEM’22). Springer, 547–558.
    [37]
    Jakob Løver, Vilde B. Gjærum, and Anastasios M. Lekkas. 2021. Explainable AI methods on a deep reinforcement learning agent for automatic docking. IFAC PapersOnLine 54, 16 (2021), 146–152.
    [38]
    Xiaozhen Lu, Liang Xiao, Pengmin Li, Xiangyang Ji, Chenren Xu, Shui Yu, and Weihua Zhuang. 2023. Reinforcement learning-based physical cross-layer security and privacy in 6G. IEEE Commun. Surveys Tutor. 25, 1 (2023), 425–466. DOI:
    [39]
    Siyu Luan, Zonghua Gu, Rui Xu, Qingling Zhao, and Gang Chen. 2022. LRP-based network pruning and policy distillation of robust and non-robust DRL agents for embedded systems. Concurr. Comput.: Pract. Exper. (2022), e7351.
    [40]
    Daoming Lyu, Fangkai Yang, Bo Liu, and Steven Gustafson. 2019. SDRL: Interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 2970–2977.
    [41]
    Aditi Mishra, Utkarsh Soni, Jinbin Huang, and Chris Bryan. 2022. Why? why not? when? visual explanations of agent behaviour in reinforcement learning. In Proceedings of the IEEE 15th Pacific Visualization Symposium (PacificVis’22). IEEE, 111–120.
    [42]
    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski et al. 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (2015), 529–533.
    [43]
    Xiaotong Nie, Motoaki Hiraga, and Kazuhiro Ohkura. 2019. Visualizing deep q-learning to understanding behavior of swarm robotic system. In Proceedings of the Symposium on Intelligent and Evolutionary Systems. Springer, 118–129.
    [44]
    Tiago Miguel Monteiro Nunes, Clark Borst, Erik-Jan van Kampen, Brian Hilburn, and Carl Westin. 2021. Human-interpretable input for machine learning in tactical air traffic control. http://mahaloproject.eu/wp-content/uploads/2022/06/SID_Paper_2021_MAHALO-2.pdf
    [45]
    Juan Marcelo Parra-Ullauri, Antonio García-Domínguez, Nelly Bencomo, Changgang Zheng, Chen Zhen, Juan Boubeta-Puig, Guadalupe Ortiz, and Shufan Yang. 2022. Event-driven temporal models for explanations-ETeMoX: Explaining reinforcement learning. Softw. Syst. Model. 21, 3 (2022), 1091–1113.
    [46]
    Sindre Benjamin Remman and Anastasios M. Lekkas. 2021. Robotic lever manipulation using hindsight experience replay and shapley additive explanations. In Proceedings of the European Control Conference (ECC’21). IEEE, 586–593.
    [47]
    Sindre Benjamin Remman, Inga Strümke, and Anastasios M. Lekkas. 2022. Causal versus marginal shapley values for robotic lever manipulation controlled using deep reinforcement learning. In Proceedings of the American Control Conference (ACC’22). IEEE, 2683–2690.
    [48]
    Stefano Giovanni Rizzo, Giovanna Vantini, and Sanjay Chawla. 2019. Reinforcement learning with explainability for traffic signal control. In Proceedings of the IEEE Intelligent Transportation Systems Conference (ITSC’19). IEEE, 3567–3572.
    [49]
    Aaron M. Roth, Jing Liang, and Dinesh Manocha. 2021. XAI-N: Sensor-based robot navigation using expert policies and decision trees. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’21). IEEE, 2053–2060.
    [50]
    Lincoln V. Schreiber, Gabriel de O. Ramos, and Ana L. C. Bazzan. 2021. Towards explainable deep reinforcement learning for traffic signal control. Proc. of LatinX in AI Workshop @ ICML 2021 LXAI. LXIA.
    [51]
    Simon Schwaiger, Mohamed Aburaia, Ali Aburaia, and Wilfried Woeber. 2021. Explainable artificial intelligence for robot arm control. Ann. DAAAM Proc. (2021), 640–648.
    [52]
    Alexander Sieusahai and Matthew Guzdial. 2021. Explaining deep reinforcement learning agents in the Atari domain through a surrogate model. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 17. 82–90.
    [53]
    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel et al. 2018. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362, 6419 (2018), 1140–1144.
    [54]
    Eduardo Soares, Plamen P. Angelov, Bruno Costa, Marcos P. Gerardo Castro, Subramanya Nageshrao, and Dimitar Filev. 2020. Explaining deep learning models through rule-based approximation and visualization. IEEE Trans. Fuzzy Syst. 29, 8 (2020), 2399–2407.
    [55]
    John Stamper and Steven Moore. 2019. Exploring teachable humans and teachable agents: Human strategies versus agent policies and the basis of expertise. In Proceedings of the 20th International Conference on Artificial Intelligence in Education (AIED’19). Springer, 269–274.
    [56]
    Mark Stefik, Michael Youngblood, Peter Pirolli, Christian Lebiere, Robert Thomson, Robert Price, Lester D. Nelson, Robert Krivacic, Jacob Le, Konstantinos Mitsopoulos et al. 2021. Explaining autonomous drones: An XAI journey. Appl. AI Lett. 2, 4 (2021), e54.
    [57]
    Philipp Theumer, Florian Edenhofner, Roland Zimmermann, and Alexander Zipfel. 2022. Explainable deep reinforcement learning for production control. In Proceedings of the Conference on Production Systems and Logistics (CPSL’22). Hannover, 809–818.
    [58]
    Connor van Rossum, Candice Feinberg, Adam Abu Shumays, Kyle Baxter, and Benedek Bartha. 2021. A novel approach to curiosity and explainable reinforcement learning via interpretable sub-goals. Retrieved from https://arXiv:2104.06630
    [59]
    Marko Vasić, Andrija Petrović, Kaiyuan Wang, Mladen Nikolić, Rishabh Singh, and Sarfraz Khurshid. 2022. MoËT: Mixture of expert trees and its application to verifiable reinforcement learning. Neural Netw. 151 (2022), 34–47.
    [60]
    A. Vijay and K. Umadevi. 2019. Secured AI guided architecture for D2D systems of massive MIMO deployed in 5G networks. In Proceedings of the 3rd International Conference on Trends in Electronics and Informatics (ICOEI’19). IEEE, 468–472.
    [61]
    George A. Vouros. 2022. Explainable deep reinforcement learning: State of the art and challenges. ACM Comput. Surveys 55, 5 (2022), 1–39.
    [62]
    Yuyao Wang, Masayoshi Mase, and Masashi Egi. 2020. Attribution-based salience method towards interpretable reinforcement learning. In Proceedings of the AAAI Spring Symposium: Combining Machine Learning with Knowledge Engineering.
    [63]
    Zheng Wang and Shen Wang. 2022. XRouting: Explainable vehicle rerouting for urban road congestion avoidance using deep reinforcement learning. In Proceedings of the IEEE International Smart Cities Conference (ISC’22). 1–7. DOI:
    [64]
    Lindsay Wells and Tomasz Bednarz. 2021. Explainable ai and reinforcement learning—a systematic review of current approaches and trends. Front. Artificial Intell. 4 (2021), 550030.
    [65]
    Salomón Wollenstein-Betech, Christian Muise, Christos G. Cassandras, Ioannis Ch Paschalidis, and Yasaman Khazaeni. 2020. Explainability of intelligent transportation systems using knowledge compilation: A traffic light controller case. In Proceedings of the IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC’20). IEEE, 1–6.
    [66]
    Duo Xu and Faramarz Fekri. 2021. Interpretable model-based hierarchical reinforcement learning using inductive logic programming. Retrieved from https://arXiv:2106.11417
    [67]
    Ke Zhang, Jun Zhang, Pei-Dong Xu, Tianlu Gao, and David Wenzhong Gao. 2022. Explainable AI in deep reinforcement learning models for power system emergency control. IEEE Trans. Comput. Soc. Syst. 9, 2 (2022), 419–427.
    [68]
    S. Kevin Zhou, Hoang Ngan Le, Khoa Luu, Hien V. Nguyen, and Nicholas Ayache. 2021. Deep reinforcement learning in medical imaging: A literature review. Med. Image Anal. 73 (2021), 102193.

    Cited By

    View all
    • (2024)A Review of Symbolic, Subsymbolic and Hybrid Methods for Sequential Decision MakingACM Computing Surveys10.1145/366336656:11(1-36)Online publication date: 28-Jun-2024
    • (2024)Advancements in Deep Reinforcement Learning and Inverse Reinforcement Learning for Robotic Manipulation: Toward Trustworthy, Interpretable, and Explainable Artificial IntelligenceIEEE Access10.1109/ACCESS.2024.338542612(51840-51858)Online publication date: 2024
    • (2024)Blocklength Allocation and Power Control in UAV-Assisted URLLC System via Multi-agent Deep Reinforcement LearningInternational Journal of Computational Intelligence Systems10.1007/s44196-024-00530-817:1Online publication date: 3-Jun-2024
    • Show More Cited By

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Computing Surveys
    ACM Computing Surveys  Volume 56, Issue 5
    May 2024
    1019 pages
    ISSN:0360-0300
    EISSN:1557-7341
    DOI:10.1145/3613598
    Issue’s Table of Contents

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 08 December 2023
    Online AM: 01 November 2023
    Accepted: 11 October 2023
    Revised: 19 June 2023
    Received: 13 July 2022
    Published in CSUR Volume 56, Issue 5

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Deep reinforcement learning
    2. DRL
    3. Explainable AI
    4. XAI
    5. neural networks
    6. Survey
    7. Review

    Qualifiers

    • Survey

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)1,126
    • Downloads (Last 6 weeks)102

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)A Review of Symbolic, Subsymbolic and Hybrid Methods for Sequential Decision MakingACM Computing Surveys10.1145/366336656:11(1-36)Online publication date: 28-Jun-2024
    • (2024)Advancements in Deep Reinforcement Learning and Inverse Reinforcement Learning for Robotic Manipulation: Toward Trustworthy, Interpretable, and Explainable Artificial IntelligenceIEEE Access10.1109/ACCESS.2024.338542612(51840-51858)Online publication date: 2024
    • (2024)Blocklength Allocation and Power Control in UAV-Assisted URLLC System via Multi-agent Deep Reinforcement LearningInternational Journal of Computational Intelligence Systems10.1007/s44196-024-00530-817:1Online publication date: 3-Jun-2024
    • (2024)Exploring the Reliability of SHAP Values in Reinforcement LearningExplainable Artificial Intelligence10.1007/978-3-031-63800-8_9(165-184)Online publication date: 10-Jul-2024
    • (2023)QUANTITATIVE STUDIES OF DEEP REINFORCEMENT LEARNING IN GAMING, ROBOTICS AND REAL-WORLD CONTROL SYSTEMSBulletin of Business and Economics (BBE)10.61506/01.0001912:2(389-395)Online publication date: 29-Aug-2023
    • (2023)Explainability of Deep Reinforcement Learning Method with Drones2023 IEEE/AIAA 42nd Digital Avionics Systems Conference (DASC)10.1109/DASC58513.2023.10311156(1-9)Online publication date: 1-Oct-2023

    View Options

    Get Access

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    Full Text

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media