Making AI Faster: A Deep Dive Across Users, Developers, and Businesses

AI isn’t just about building smarter models—it's about making them practical, performant, and scalable. This means solving for three interdependent axes: speed, quality, and cost. Let’s break down why these matter across three critical stakeholder perspectives:

• End Users expect seamless, trustworthy, and responsive AI experiences.
• AI Developers need faster iteration loops, debuggable pipelines, and scalable training.
• Business Stakeholders demand ROI, cost efficiency, and regulatory compliance.

Think of AI powering a voice assistant or a self-driving car or any other AI use-case. Speed determines usability, accuracy builds trust, and cost dictates feasibility. I am writing these articles as a three part series to help discover practical strategies to accelerate AI development, boost performance, and optimize costs without compromising innovation. Drawing from real-world experiences, we will discuss Making AI Faster, Better, Cheaper. 

In this article, I will focus on the first pillar - "Faster", dive deep into why Faster AI matters, what the main challenges are and strategies to make AI Faster. I will also bring out three key perspectives: End Users, AI Developers, and Businesses.

Why "Faster" Matters

Speed is now a necessity in AI development. Whether you're building the next-generation voice assistant, fraud detection engine, or personalized learning platform, latency and efficiency directly impact user experience, developer velocity, and business competitiveness. 

1. Users are oblivious to the systemic complexities behind an AI product they use. They demand real-time and smooth experiences that they have been used to working with non-AI surfaces. In this age of limited attention span, users expect fast responses from AI systems like chatbots, recommendation engines, and smart assistants. Delayed responses can break the user's trust, decrease satisfaction and cause a drop in engagement.
2. Developers need shorter training and deployment cycles to ship products quickly and reliably. Long build and test loops kill momentum, increase burnout, and hamper innovation. Reducing iteration time and delivering incremental improvements drives quicker feedback cycles, which directly improve product quality.
3. Businesses compete on time-to-market and must innovate rapidly while managing costs. Being first to market with a new AI capability can offer a temporary monopoly, drive customer growth, and improve brand loyalty. A slow AI lifecycle, on the other hand, can erode competitive advantage and revenue opportunities.

I. End-User Perspective: Instant Gratification

Challenges

1. High latency ruins user experience: especially in real-time interactions with voice assistants. In use cases like autonomous driving or fraud detection, high latency can even cause critical errors. “Even a 100-millisecond delay can be critical, potentially being the difference between life and death for a pedestrian or car passenger” (Addressing Data Processing Challenges in Autonomous Vehicles | IoT for All, n.d.). 
2. Low personalization due to lag in real-time inference: High latency prevents timely adjustments based on user context, making the product feel generic. “minimizing the time your system takes to generate and serve a recommendation improves conversion” (Amazon Personalize Improvements Reduce Model Training Time by up to 40% and Latency for Generating Recommendations by up to 30% | Amazon Web Services, 2022). 
3. Inconsistent performance across platforms: Users expect uniform behavior whether they’re on a smartphone, tablet, or desktop. Latency or responsiveness that varies by platform can reduce trust. “In an analysis of 120 reviews across six smart health apps, users repeatedly complained that the same AI feature felt “snappy on iPad but sluggish on my Android phone,” and flagged “inconsistent performance” as a top reason for mistrust and churn” (Mohajeri & Cheng, 2022).

Strategies

1. Edge & On-Device Inference: Running models on local devices reduces the time spent communicating with cloud servers. This strategy is particularly effective in latency-sensitive applications such as voice typing or smart cameras. It also helps ensure availability in low-connectivity areas, a common scenario with moving self-driving vehicles. Moreover not sending the data over to the cloud for inference helps preserve user-privacy.
2. Asynchronous & Streaming Pipelines: Asynchronous processing allows the system to return partial or preliminary results immediately, while continuing to process the full request in the background. This is ideal for search engines, autocomplete systems, or streaming summarization applications, where a "fast enough" answer now is better than a perfect one later.
3. Model Compression: Techniques like pruning and quantization shrink the size of the model, enabling them to run faster without significant drops in accuracy. This enables real-time AI features even in hardware-constrained environments, making AI more accessible to all users.

II. Developer Perspective: Unblocking Velocity

Challenges

1. Data bottlenecks: Data is the backbone of training and evaluating AI models. Often projects are delayed due to missing train/eval datasets. Throwing money at the problem is not a solution. Large human teams have also not been able to deliver at the scale to satisfy AI needs. Tencent’s HD-map division with 1000+ labelers called it “very time-consuming and costly,” prompting them to build the auto-labeling system (Tang et al., 2022).
2. Hardware bottlenecks: GPU/TPU shortages delay training. With limited compute capacity and long queue times, developers often spend more time waiting than building. These delays extend the product development lifecycle. A survey of 1,400 AI professionals found 85% delayed projects through GPU scarcity; 39% slipped schedules by 3-6 months (Digitalisation World, 2025).
3. Long training cycles: Some models take weeks to train. These long cycles not only slow down iteration but also increase the risk that the market moves on or the underlying data distribution shifts by the time a model is ready. A single GPT-4 ran weeks on thousands of GPUs at $41M (Buchholz, 2024). 
4. Debugging friction: CUDA version mismatches, NCCL timeouts, and inconsistent environment setups create long debugging cycles and consume valuable engineering time that should be spent on innovation (Macheng, 2022).
5. Compliance slowdowns: Weeks lost in audits and sign-offs, especially in regulated domains like finance or healthcare. Risk reviews, documentation requirements, and model validation often occur too late in the development cycle, causing launch delays. Banks spend a median 7 weeks per model on validation reviews before approval (Kumar et al., 2022). 

Strategies

1. Hardware acceleration: Determining the right hardware for your use-case shall accelerate performance of your products. Choosing between the more generalizable GPUs or the more specific ASICs can make a world of difference to your product’s performance. Even while choosing GPUs, ensuring that you use GPUs tuned for inference vs training at the right time, would accelerate performance. e.g. (Ai, 2025).
2. Elastic Multi-Cloud GPU Scheduling: By dynamically routing jobs to available GPUs across multiple cloud providers, teams can minimize wait times and optimize for cost and availability (Bhardwaj, 2024).
3. Transfer Learning on Domain-Specific Datasets: Instead of training large models from scratch, developers can fine-tune pre-trained models on their own data. This cuts training time by orders of magnitude and allows teams to leverage state-of-the-art architectures with relatively small compute budgets (P, 2025). 
4. Sparse Mixture-of-Experts (MoE): MoE architectures activate only a subset of model parameters per inference, reducing the compute cost without sacrificing accuracy. When combined with expert-trimming, developers can further reduce the size and latency of production models (D, 2025). 
5. Automated Labeling QA & Weak Supervision: LLM as a judge is becoming very popular to deliver high accuracy labels to unblock developers from receiving train/eval data. Automating labeling processes through weak supervision frameworks or model-based annotations reduces manual effort (What Is Snorkel Flow? | Snorkel AI, n.d.).
6. Early Risk & Compliance Audits: Building compliance workflows into the development pipeline can help avoid last-minute surprises. Incorporating model cards, audit logs, and transparent documentation early on accelerates approval processes and builds trust with legal and regulatory teams (Mitchell et al., 2019). 

III. Business Perspective: Time-to-Value

Challenges

1. Slow launches miss market opportunities: When AI development takes too long, competitors can seize market share or user behavior may shift, rendering the delayed product obsolete. Apple Intelligence delays let rivals like Google and OpenAI dominate the AI assistant market (Chowdhury, 2025). 
2. Idle infrastructure drains costs without output: Unused GPU capacity or overprovisioned compute resources can inflate budgets without delivering proportional value. Slow feedback loops also waste engineering time and reduce morale. 100 GPUs running at 40% utilization can waste over $1.5 million annually on idle compute (Cabrera-Naranjo, 2025).
3. Regulatory overhead slows production: Without proactive governance, AI products may fail compliance checks at launch. This can result in costly remediations or legal penalties, and in some cases, lead to product cancellations. South Korea banned new downloads of DeepSeek until it addressed data transfer concerns (Wikipedia contributors, 2025).

Strategies

1. Track ROI per Optimization: Piloting optimizations with measurable success criteria (e.g., training time saved, costs reduced, or conversion rates improved) enables data-driven decisions. It also builds internal case studies to justify future investments in AI acceleration. e.g. Vannevar Labs cut ML inference costs by 45% (Hamrick, 2024).
2. Cross-Functional team setup: Creating cross functional pods across ML teams, infra, and operations personnel spreads responsibility and creates shared ownership of cost and performance metrics. These teams can identify bottlenecks, recommend tradeoffs, and manage infrastructure budgets more effectively. Setting service level agreements around latency, accuracy, and cost, teams are incentivized to optimize holistically (Yau, 2025). 

Achieving faster AI outcomes doesn’t mean rushing or sacrificing quality. It involves setting measurable goals, iterating continuously, and fostering collaboration across functions.

Conclusion

Making AI faster is not about brute-forcing bigger hardware or throwing more engineers at the problem. It’s about strategic thinking, thoughtful design, and organizational alignment. Balancing speed with accuracy and responsibility is what distinguishes scalable AI systems from fragile prototypes. By adopting these approaches across the user, developer, and business dimensions, AI teams can ship better products faster—without cutting corners.

Want to dive deeper into "Better" and "Cheaper" strategies? Stay tuned for the next post in this series.

Disclaimer: Views expressed are my own and do not represent those of Meta or its affiliates.

References

1. Addressing data processing challenges in autonomous vehicles | IoT for all. (n.d.). IoT for All. https://www.iotforall.com/addressing-data-processing-challenges-in-autonomous-vehicles

2. Amazon Personalize improvements reduce model training time by up to 40% and latency for generating recommendations by up to 30% | Amazon Web Services. (2022, April 29). Amazon Web Services. https://aws.amazon.com/blogs/machine-learning/amazon-personalize-improvements-reduce-model-training-time-by-up-to-40-and-latency-for-generating-recommendations-by-up-to-30/

3. Mohajeri, B., & Cheng, J. (2022). “Inconsistent Performance”: Understanding Concerns of Real-World Users on Smart Mobile Health Applications Through Analyzing App Reviews. UIST ’22: The 35th Annual ACM Symposium on User Interface Software and Technology, 1–4. https://doi.org/10.1145/3526114.3558698

4. Tang, K., Cao, X., Cao, Z., Zhou, T., Li, E., Liu, A., Zou, S., Liu, C., Mei, S., Sizikova, E., & Zheng, C. (2022, December 14). THMA: Tencent HD Map AI system for creating HD map annotations. arXiv.org. https://arxiv.org/abs/2212.11123

5. Digitalisation World. (2025, July 7). Delays in AI and ML projects due to GPU availability. Digitalisation World. https://digitalisationworld.com/news/69243/delays-in-ai-and-ml-projects-due-to-gpu-availability

6. Buchholz, K. (2024, August 23). The extreme cost of training AI models. Forbes. https://www.forbes.com/sites/katharinabuchholz/2024/08/23/the-extreme-cost-of-training-ai-models/

7. Macheng. (2022, April 12). About Timeout when use Multi-gpu training [Online forum post]. GitHub. https://github.com/huggingface/accelerate/issues/314

8. Kumar, P., Laurent, M., Rougeaux, C., & Tejada, M. (2022, March 9). Model risk management 2.0 evolves to address continued uncertainty of risk-related events. McKinsey & Company. https://www.mckinsey.com/capabilities/risk-and-resilience/our-insights/model-risk-management-2-point-0-evolves-to-address-continued-uncertainty-of-risk-related-events

9. Ai, N. (2025, February 25). Choosing the best GPU for machine learning in 2025: A complete guide. Medium. https://medium.com/%40marketing_novita.ai/choosing-the-best-gpu-for-machine-learning-in-2025-a-complete-guide-1d3e5aa8560a

10. Bhardwaj, R. (2024, August 19). AI on Kubernetes Without the Pain. SkyPilot Blog. https://blog.skypilot.co/ai-on-kubernetes/

11. P, P. (2025, June 16). What is transfer learning in Computer Vision? Beginner guide. Roboflow Blog. https://blog.roboflow.com/what-is-transfer-learning/

12. D, J. (2025, June 8). Mixture of Experts (MOE): How AI models train faster and cheaper. Medium. https://jwaran78.medium.com/mixture-of-experts-moe-how-ai-models-train-faster-and-cheaper-6a74feaad78c

13. What is Snorkel Flow? | Snorkel AI. (n.d.). https://docs.snorkel.ai/docs/25.4/user-guide/intro/what-is-snorkel-flow

14. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019). Model Cards for Model Reporting. FAT* ’19: Conference on Fairness, Accountability, and Transparency, 220–229. https://doi.org/10.1145/3287560.3287596

15. Chowdhury, H. (2025, March 11). Apple is scrambling to catch up in a race it had a headstart in. Business Insider. https://www.businessinsider.com/apple-intelligence-siri-delay-ai-race-assistant-2025-3

16. Cabrera-Naranjo, S. (2025, April 17). The Massive & Hidden Cost of AI: Why GPU Underutilization is Costing Enterprises Millions. https://www.linkedin.com/pulse/massive-hidden-cost-ai-why-gpu-underutilization-cabrera-naranjo-dbyie/

17. Wikipedia contributors. (2025, July 6). DeepSeek (chatbot). Wikipedia. https://en.wikipedia.org/wiki/DeepSeek_%28chatbot%29

18. Hamrick, N. (2024, November 20). How Vannevar Labs cut ML inference costs by 45%. Vannevar Labs. https://vannevarlabs.com/blog/2024/11/20/aws-ray-eks/

19. Yau, L. W. (2025, May 6). Study: Cross-functional collaboration in data Teams. Dialpad. https://www.dialpad.com/blog/cross-functional-collaboration/

Disclaimer: Views expressed are my own and do not represent those of Meta or its affiliates.