Towards Usable and Useful Explainable AI

Overview

Explainable AI (XAI) plays a pivotal role in enabling understanding and trust in AI systems, especially as their complexity continues to grow. In the era of classical deep learning, research efforts have focused on providing human-understandable explanations for model outputs and behaviors, often delivered through explanation-as-a-service, i.e., usable XAI. More recently, with the advent of large language models, researchers have paid additional attention to understanding the internal representations and mechanisms of these models, leveraging this knowledge to address broader challenges such as safety issues, forming the foundation of useful XAI.

In this talk, I will present my recent contributions to both directions. First, I will discuss the faithfulness issues of attention-based interpretation techniques, a hallmark of usable XAI in the deep learning era, which are unstable against randomness and perturbations during training or testing. I will then introduce editable concept bottleneck models that support removing or inserting some training data or new concepts to address issues like privacy concerns, data mislabelling, spurious concepts, or concept annotation errors. Next, I will discuss Med-MICN, a multi-modal interpretability alignment framework that aligns for various angles, including neural symbolic reasoning, concept semantics, and saliency maps, achieving strong results in the medical domain. Finally, I will share my vision and recent work on usable and useful XAI for LLMs and MLLMs. These works underscore the practical and theoretical advancements required to build XAI systems that are both usable and useful. They highlight the transformative potential of XAI in real-world applications, offering tools to enhance trust, performance, and safety in AI-driven systems.