Toward Explainable AI (Part 4): Bridging Theory and Practice—Beyond Explainability, What Else Is Needed

Series reminder: This series explores how explainability in AI helps build trust, ensure accountability, and align with real-world needs, from foundational principles to practical use cases.

Previously, in Part III: The Two Major Categories of Explainable AI Techniques.  How XAI methods help open the black box.

In this Part: We examine how explainable AI is applied in critical sectors (healthcare, finance, and public service) and why it has become a legal and ethical imperative.

6. Explaining Explainability: A Recap

Explainable artificial intelligence has become a necessity to ensure transparency, trust, and the legitimacy of algorithmic decisions. In highly sensitive domains such as healthcare, finance, and public governance, understanding how an AI system reaches a decision is no longer a luxury, it is an ethical, operational, and regulatory requirement.

We’ve seen that model opacity can generate distrust, slow down adoption, or even worsen existing biases in the data. In contrast, explainability mechanisms help detect errors more effectively, correct problematic behaviors, and align system decisions with the expectations of both users and regulators.

The legal framework (most notably the European AI Act) now sets high standards for traceability and auditability. As a result, companies must find a balance between performance and explainability, or risk facing legal, ethical, or reputational consequences.

Toward High-Performance, Explainable AI

The idea that we must choose between power and transparency is becoming increasingly outdated. Explainability and performance can coexist. In fact, many studies show that a more understandable model is often more robust when faced with the uncertainties of real-world data.

Hybrid approaches are emerging: constrained neural networks, mixed-architecture algorithms, and integrated explanation methods. These efforts reflect a deeper shift. Reliable AI is no longer just AI that works, it’s AI that can be explained.

This raises a legitimate question:

Can an AI system that performs well, but remains incomprehensible, truly be considered trustworthy?

Operational Frameworks and Governance of Explainability

Beyond the technical and methodological aspects discussed in this first part, it is important to emphasize that the effective implementation of XAI also requires appropriate operational frameworks and governance structures. 

The mere adoption of explainability tools is not enough: organizations must clearly define roles and responsibilities related to explainability, establish integrated processes throughout the AI lifecycle (from design to deployment and maintenance) and implement regular audit strategies for both models and their explanations. 

Furthermore, systematic documentation of AI decisions and their underlying rationale must be established to meet legal requirements and ensure traceability.

Finally, specific management approaches are needed for particularly complex models that may remain difficult to interpret for end users or regulators despite XAI techniques. While these organizational and governance challenges are indispensable, they are beyond the scope of this document, which focuses on the conceptual foundations and practical illustrations of explainability. They deserve dedicated in-depth study.

7. A Final Word Before We Get Practical

Explainable AI is neither a passing trend nor a regulatory checkbox. It is a cornerstone for building algorithmic systems that are responsible, auditable, and governed. Most importantly, it makes them acceptable to humans.

In this first part, Frédéric Jacquet laid the groundwork: why explainability is essential, which approaches make it possible, and how they apply in real-world contexts.

But true understanding begins with implementation.

That is what Marc-Antoine Iehl offers in the second part of this document. Through two detailed case studies, he demonstrates how to apply explainability methods (LIME and SHAP) to models used in healthcare and finance.

This is not just about visualizing results, but about understanding the hidden logic behind the predictions of complex models. The approach is rigorous, illustrated with real datasets, scripts, and visualizations that show the algorithm "in action".

Wrap-up and what’s next: Real-world impact makes explainability indispensable.

Next, in part V, we’ll explore the limits of current methods and what else is needed to make AI truly trustworthy.

Links to the previous articles published in this series:

• Toward Explainable AI (Part I): Bridging Theory and Practice—Why AI Needs to Be Explainable
• Toward Explainable AI (Part 2): Bridging Theory and Practice—The Two Major Categories of Explainable AI Techniques
• Toward Explainable AI (Part 3): Bridging Theory and Practice—When Explaining AI Is No Longer a Choice

Glossary

Algorithmic Bias: Systematic and unfair discrimination in AI outcomes caused by prejudices embedded in training data, model design, or deployment processes, which can lead to disparate impacts on certain population groups. Detecting and mitigating algorithmic bias is a key objective of explainable AI.

Bias Detection (via XAI): Use of explainability methods to identify biases or disproportionate effects in algorithmic decisions.

Contrastive and Counterfactual Explanations: Explanations that compare the decision made to what could have happened by changing certain variables (e.g., “Why this outcome instead of another?”).

Decision Plot: A graphical representation tracing the successive impact of variables on an individual prediction.

Evaluation Metrics for Explainability: Criteria used to assess the quality of an explanation (fidelity, robustness, consistency, etc.).

Feature Importance (Variable Contribution): Measurement or attribution of the relative impact of each variable on the model’s final decision.

Force Chart (Force Plot): An interactive visualization illustrating positive or negative forces exerted by each variable on a prediction.

Fidelity: A measure of how faithfully the explanation reflects the true logic of the model.

Global Explanation: An overview of the model’s behavior across the entire dataset.

Human Interpretability: The quality of an explanation to be understood and useful to a human, non-expert user.

Intrinsically Interpretable Models: Models whose very structure allows direct understanding (e.g., decision trees, linear regressions).

LIME (Local Interpretable Model-agnostic Explanations): A local explanation method that generates simple approximations around a given prediction to reveal the influential factors.

Local Explanation: A detailed explanation regarding a single prediction or individual case.

Model Transparency: The quality of a model in making its decision-making processes accessible and understandable.

Post-hoc Explainability: Explainability techniques applied after model training, without altering its internal functioning.

Robustness (Stability): The ability of an explainability method to provide consistent explanations despite small variations in input data.

SHAP (SHapley Additive exPlanations): An approach based on game theory that assigns each variable a quantitative contribution to the prediction, providing both global and local explanations.

Summary Chart (Summary Plot): A visualization ranking variables according to their average influence on predictions.

Waterfall Chart: A static visualization showing step by step how each variable contributes to the final prediction.

References

1. LIME: [link]

2. Chest X-Ray Images (Pneumonia): [link]

3. German Credit Dataset: [link]

4. Complete notebook for the LIME case study: [link] 

5. Complete notebook for the SHAP case study: [link] 

6. Leslie, D., Rincón, C., Briggs, M., Perini, A., Jayadeva, S., Borda, A., et al. (2024). AI Explainability in Practice, The Alan Turing Institute, AI Ethics and Governance in Practice Programme. Participant & Facilitator Workbook. https://www.turing.ac.uk/sites/default/files/2024-06/aieg-ati-7-explainabilityv1.2.pdf

7. AI explainability in practice. (2024). The Alan Turing Institute. https://www.turing.ac.uk/sites/default/files/2024-06/aieg-ati-7-explainabilityv1.2.pdf

8. What is AI (artificial intelligence)? (2024, April 3). McKinsey & Company. https://www.mckinsey.com/featured-insights/mckinsey-explainers/what-is-ai

9. Kite-Powell, J. (2022, July 28). Explainable AI is trending and here’s why. Forbes. https://www.forbes.com/sites/jenniferhicks/2022/07/28/explainable-ai-is--trending-and-heres-why

10. Excellence and Trust in AI — Brochure. (n.d.). Shaping Europe’s Digital Future. https://digital-strategy.ec.europa.eu/en/library/excellence-and-trust-ai-brochure

11. Heikkilä, M. (2024, March 6). Nobody knows how AI works. MIT Technology Review. https://www.technologyreview.com/2024/03/05/1089449/nobody-knows-how-ai-works/

12. Heaven, W. D. (2025, July 30). Large language models can do jaw-dropping things. But nobody knows exactly why. MIT Technology Review. https://www.technologyreview.com/2024/03/04/1089403/large-language-models-amazing-but-nobody-knows-why/

13. Francis, J. (2024, November 15). Council Post: Why 85% of your AI models may fail. Forbes. https://www.forbes.com/councils/forbestechcouncil/2024/11/15/why-85-of-your-ai-models-may-fail

14. Bao, A., & Zeng, Y. (2024). Understanding the dilemma of explainable artificial intelligence: a proposal for a ritual dialog framework. Humanities and Social Sciences Communications, 11(1). https://doi.org/10.1057/s41599-024-02759-2

15. Gallese, C. (2023). The AI Act proposal: a new right to technical interpretability? arXiv (Cornell University). https://doi.org/10.48550/arxiv.2303.17558

16. GALDON CLAVELL, G. (n.d.). AI Auditing - Checklist for AI auditing. In SUPPORT POOL OF EXPERTS PROGRAMME. https://www.edpb.europa.eu/system/files/2024-06/ai-auditing_checklist-for-ai-auditing-scores_edpb-spe-programme_en.pdf

17. White, B., & Guha, K. (2025, June 14). Component based Quantum Machine Learning explainability. arXiv.org. https://arxiv.org/abs/2506.12378