Toward Explainable AI (Part 3): Bridging Theory and Practice—When Explaining AI Is No Longer a Choice

Series reminder: This series explores how explainability in AI helps build trust, ensure accountability, and align with real-world needs, from foundational principles to practical use cases.

Previously, in Part II: The Two Major Categories of Explainable AI Techniques. How XAI methods help open the black box

In this Part: We examine how explainable AI is applied in critical sectors (healthcare, finance, and public services) and why it has become a legal and ethical imperative.

4. Real-World Applications

Healthcare and Medicine

AI is frequently discussed in the medical field, especially when it comes to making a diagnosis or assessing risk. Today, algorithms are capable of analyzing medical images, detecting anomalies in X-rays, and even predicting the likelihood of developing certain pathologies based on clinical data. These tools prove fully effective only when both physicians and patients are able to understand the recommendations provided by the AI. In such cases, explainable AI makes it possible to identify the factors that led to a given prediction. This prevents both the practitioner and the patient from being placed in a position of blindly accepting the results. Such a situation could often lead to rejection of the diagnosis due to a lack of understanding. 

For example, an AI-based diagnostic system might indicate that the size of a tumor, the patient’s medical history, and certain biological markers were the basis for its analysis. This level of applicability provides healthcare professionals with greater transparency in their decision-making and helps prevent medical errors.

Finance and Insurance

In the financial sector, artificial intelligence is widely used for credit application assessment and insurance pricing. Banks and insurance companies rely on predictive models to evaluate customer creditworthiness and adjust premiums based on perceived risk

In these use cases, an opaque algorithm can lead to decisions that are perceived as unfair or arbitrary, such as denying a loan. With explainable AI, it becomes possible to show clients the specific reasons behind a decision, for instance by highlighting factors like repayment history or debt level.
This transparency can help applicants better understand their rating and, if needed, take appropriate steps to improve their profile.

Moreover, as discussed earlier, explainability helps reduce discriminatory bias. It can reveal, for example, whether certain population groups are systematically disadvantaged by a model, and make it possible to apply corrections when needed.

Public Sector and Governance

In the administrative and governmental sphere, AI is part of the toolbox used to automate decision-making in areas such as the allocation of social benefits, tax processing, or fraud detection. While these AI algorithms help speed up processes and improve efficiency, they also raise critical questions around transparency and fairness.

When an AI system makes a decision that impacts a citizen, it is essential that the decision can be explained in a way that is understandable. The goal is to prevent any perception of arbitrariness.

For example, an automated system that allocates grants must be able to explain why an application was accepted or rejected. It should clearly state the specific criteria considered, such as income, family situation, or eligibility for certain programs. 

From a legal standpoint, the use of explainable AI helps ensure compliance with democratic principles. It guarantees that decisions remain subject to control and challenge, which helps maintain public trust in these systems.

"By 2028, 80% of CIOs are expected to implement organizational changes to utilize AI, automation, and analytics effectively, fostering agile and insight-driven digital enterprises. This evolution will create both new opportunities and notable risks, compelling every organization to navigate a unique path to impactful and responsible outcomes." - IDC (2024). The Path to AI Everywhere: Exploring the Human Challenge. IDC Research.

While these approaches help make AI more interpretable, they also bring significant challenges, from technical complexity to the delicate balance between transparency and performance.

5. Challenges and Limitations of Explainable AI

Technical Complexity

One of the major challenges of explainable artificial intelligence lies in the complexity of the underlying models, especially deep learning algorithms. These models are built on neural networks with multiple layers of processing. They operate through nonlinear relationships and millions of automatically adjusted parameters. For example, GPT-3, the model behind ChatGPT-3.5, contains 175 billion parameters. 

The more complex a model is, the harder it becomes to extract a clear and understandable explanation for humans. Unlike simpler methods such as decision trees or linear regressions, neural networks do not offer an explicit logic between input and output, which makes their interpretation extremely challenging.

Post-hoc explainability methods, such as LIME or SHAP, attempt to provide explanations by analyzing the impact of each variable on the outcome, but they do not necessarily reveal the model’s internal workings. This complexity also makes it difficult to validate and audit algorithms, especially in the most critical domains.

Bias and Limitations of Explanations

Even when an AI system provides an explanation for its decisions, that explanation can still be biased. Tools like SHAP and LIME, while powerful, do not guarantee absolute transparency. They rely on local approximations, which means that the explanation may be valid for a specific case but not reflect the model’s overall behavior.

It's also important to understand that some explanations can mislead users by presenting a simplified or partial version of the algorithm’s reasoning. For example, a credit scoring model might explain a loan rejection by highlighting specific financial criteria, while hiding more complex correlations or systemic biases embedded in the training data.

This phenomenon can be amplified when explanations are designed to be acceptable rather than accurately reflect reality. This raises important questions about the trustworthiness and objectivity of explainability tools.

When it comes to biases in training data, it’s important to understand that making AI explainable does not make those biases disappear. An AI system can still produce discriminatory decisions, even if it is able to justify its choices. It’s clear that explainability alone is not enough. It must be supported by a broader reflection on data quality and the management of algorithmic bias.

Regulatory and Ethical Constraints

In response to the risks posed by algorithmic opacity, regulators are working to impose requirements around explainability, particularly in Europe with the AI Act. This legislative framework aims to ensure that decisions made by AI remain understandable, traceable, and justifiable, especially in so-called “high-risk” applications such as medical diagnostics or public surveillance systems.

But it’s important to keep in mind that complying with these new requirements presents a real challenge for companies. On the one hand, they must adapt their models to ensure greater transparency. This may involve costly and complex technical changes. On the other hand, they need to find a balance between explainability and performance: the most accurate and powerful models are often the most opaque, and imposing too many constraints could slow down technological innovation.

Another key issue concerns legal responsibility: who should be held accountable if an AI model causes harm? The AI Act emphasizes the need for human oversight and recourse mechanisms in the case of questionable automated decisions. This requires companies to implement monitoring and documentation systems that can provide robust explanations in the event of a dispute.

Wrap-up and what’s next: Real-world impact makes explainability indispensable.

Next, in part IV, we’ll explore the limits of current methods and what else is needed to make AI truly trustworthy.

Links to the previous articles published in this series:

• Toward Explainable AI (Part I): Bridging Theory and Practice—Why AI Needs to Be Explainable
• Toward Explainable AI (Part 2): Bridging Theory and Practice—The Two Major Categories of Explainable AI Techniques

Glossary

Algorithmic Bias: Systematic and unfair discrimination in AI outcomes caused by prejudices embedded in training data, model design, or deployment processes, which can lead to disparate impacts on certain population groups. Detecting and mitigating algorithmic bias is a key objective of explainable AI.

Bias Detection (via XAI): Use of explainability methods to identify biases or disproportionate effects in algorithmic decisions.

Contrastive and Counterfactual Explanations: Explanations that compare the decision made to what could have happened by changing certain variables (e.g., “Why this outcome instead of another?”).

Decision Plot: A graphical representation tracing the successive impact of variables on an individual prediction.

Evaluation Metrics for Explainability: Criteria used to assess the quality of an explanation (fidelity, robustness, consistency, etc.).

Feature Importance (Variable Contribution): Measurement or attribution of the relative impact of each variable on the model’s final decision.

Force Chart (Force Plot): An interactive visualization illustrating positive or negative forces exerted by each variable on a prediction.

Fidelity: A measure of how faithfully the explanation reflects the true logic of the model.

Global Explanation: An overview of the model’s behavior across the entire dataset.

Human Interpretability: The quality of an explanation to be understood and useful to a human, non-expert user.

Intrinsically Interpretable Models: Models whose very structure allows direct understanding (e.g., decision trees, linear regressions).

LIME (Local Interpretable Model-agnostic Explanations): A local explanation method that generates simple approximations around a given prediction to reveal the influential factors.

Local Explanation: A detailed explanation regarding a single prediction or individual case.

Model Transparency: The quality of a model in making its decision-making processes accessible and understandable.

Post-hoc Explainability: Explainability techniques applied after model training, without altering its internal functioning.

Robustness (Stability): The ability of an explainability method to provide consistent explanations despite small variations in input data.

SHAP (SHapley Additive exPlanations): An approach based on game theory that assigns each variable a quantitative contribution to the prediction, providing both global and local explanations.

Summary Chart (Summary Plot): A visualization ranking variables according to their average influence on predictions.

Waterfall Chart: A static visualization showing step by step how each variable contributes to the final prediction.

References

1. LIME: [link]

2. Chest X-Ray Images (Pneumonia): [link]

3. German Credit Dataset: [link]

4. Complete notebook for the LIME case study: [link] 

5. Complete notebook for the SHAP case study: [link] 

6. Leslie, D., Rincón, C., Briggs, M., Perini, A., Jayadeva, S., Borda, A., et al. (2024). AI Explainability in Practice, The Alan Turing Institute, AI Ethics and Governance in Practice Programme. Participant & Facilitator Workbook. https://www.turing.ac.uk/sites/default/files/2024-06/aieg-ati-7-explainabilityv1.2.pdf

7. AI explainability in practice. (2024). The Alan Turing Institute. https://www.turing.ac.uk/sites/default/files/2024-06/aieg-ati-7-explainabilityv1.2.pdf

8. What is AI (artificial intelligence)? (2024, April 3). McKinsey & Company. https://www.mckinsey.com/featured-insights/mckinsey-explainers/what-is-ai

9. Kite-Powell, J. (2022, July 28). Explainable AI is trending and here’s why. Forbes. https://www.forbes.com/sites/jenniferhicks/2022/07/28/explainable-ai-is--trending-and-heres-why

10. Excellence and Trust in AI — Brochure. (n.d.). Shaping Europe’s Digital Future. https://digital-strategy.ec.europa.eu/en/library/excellence-and-trust-ai-brochure

11. Heikkilä, M. (2024, March 6). Nobody knows how AI works. MIT Technology Review. https://www.technologyreview.com/2024/03/05/1089449/nobody-knows-how-ai-works/

12. Heaven, W. D. (2025, July 30). Large language models can do jaw-dropping things. But nobody knows exactly why. MIT Technology Review. https://www.technologyreview.com/2024/03/04/1089403/large-language-models-amazing-but-nobody-knows-why/

13. Francis, J. (2024, November 15). Council Post: Why 85% of your AI models may fail. Forbes. https://www.forbes.com/councils/forbestechcouncil/2024/11/15/why-85-of-your-ai-models-may-fail

14. Bao, A., & Zeng, Y. (2024). Understanding the dilemma of explainable artificial intelligence: a proposal for a ritual dialog framework. Humanities and Social Sciences Communications, 11(1). https://doi.org/10.1057/s41599-024-02759-2

15. Gallese, C. (2023). The AI Act proposal: a new right to technical interpretability? arXiv (Cornell University). https://doi.org/10.48550/arxiv.2303.17558

16. GALDON CLAVELL, G. (n.d.). AI Auditing - Checklist for AI auditing. In SUPPORT POOL OF EXPERTS PROGRAMME. https://www.edpb.europa.eu/system/files/2024-06/ai-auditing_checklist-for-ai-auditing-scores_edpb-spe-programme_en.pdf

17. White, B., & Guha, K. (2025, June 14). Component based Quantum Machine Learning explainability. arXiv.org. https://arxiv.org/abs/2506.12378