How Agile Development Principles Accelerate AI Initiatives
How Agile Development Principles Accelerate AI Initiatives
This article looks at how some organizations can often underestimate the data effort involved in AI projects and how Agile AI can help data science teams.
Join the DZone community and get the full member experience.Join For Free
Insight for I&O leaders on deploying AIOps platforms to enhance performance monitoring today. Read the Guide.
Data scientist teams are becoming increasingly frustrated with their AI initiatives. Organizations often underestimate the data effort involved in AI projects and place data scientists under tremendous time pressure and oftentimes unrealistic expectations. These strategic data projects are exploratory, high-visibility, and high-risk, and it’s unlikely there’s a perfect solution at the end. There is a strong desire to accelerate delivery and reduce risk, or even get AI projects back on track.
A primary reason for this frustration is the largely linear, sequential approach applied to AI projects. This is a familiar pattern in the software development world called “waterfall development” where progress flows in one sequential direction, like a waterfall, through the phases of the development cycle. The waterfall approach requires an enormous amount of up-front planning with all of the requirements and anticipated issues defined up-front. The process flows downstream from stage to stage and team to team until all stages are complete. The downside to this method is that it is restrictive and rigid in nature. Teams may waste time and resources, experience long delays before seeing value, and risk building the wrong thing.
Similarly, applying a waterfall approach to an AI initiative increases risk as project teams are attempting to anticipate and solve for all needs up front rather than taking a step-by-step, iterative approach that enables the discovery of issues early allowing for course correct along the way. With the waterfall approach, AI teams may find at the end of a project that they’ve built the wrong model, that the primary value comes from only a particular part of the model, or discover that there is more business value in applying AI differently altogether.
Similar challenges drove the significant shift in software development over the past decade to “Agile Development.” In this approach, the larger effort is broken down into smaller components that can be completed and launched quickly, and provide incremental value. Subsequent launches will continually improve and add new components. Agile is designed to give teams more flexibility and allow them to ship software on time. It emphasizes iterative development by building software in pieces, and launching and learning quickly.
The most successful AI teams are applying the same agile principles to their projects and while it’s not new to the software development world, it’s rarely being applied to the development of AI projects. Using an Agile AI approach allows for iterative development, enabling AI initiatives to be broken down into manageable bite-sized pieces. As a result, teams learn continuously and can re-prioritize where necessary, which delivers value faster and builds the right solution more efficiently.
To give an example of what each of these approaches look like in the context of an AI project, imagine a company that needs to review hundreds of thousands of user-generated photos for approval before they can be displayed on their website. Each image must be checked for several types of violations: no explicit material, no copyrighted images, and no products or brands that conflict with advertising partners.
Currently, an internal quality control team handles the review process, in addition to other responsibilities, but it is time-consuming, expensive, and doesn’t scale as volume increases. The company decides to automate this process with AI and initiates a new project. A data science team is established and they start work on the AI model.
In the traditional waterfall approach, the team starts to design the AI solution as the end-goal of a comprehensive model that addresses the three core criteria under review for each image. To train the model, the data science team must generate large amounts of example data for each violation type and the various combinations. Once the model design is complete with appropriate training data, the testing process starts with model scoring and validation, accounting for variations of all three violation types so the model can be improved. Only at the conclusion of that phase can the solution be deployed and start to add value.
With Agile AI, the first step would be to identify which of the three violation types is the most impactful to the business and the most in need of automation. Let's suppose the business team decides to prioritize the identification of advertising conflicts because it is the most time-consuming for humans and has direct revenue implications. The data science team will focus on an AI model and the associated training data for just the advertising violation while the internal quality control team continues to handle the others.
Now the data science team only has to generate training data for one set of imagery and can test, score, and validate a simpler model. In true Agile fashion, the first deployment of the model would be through a suggestion engine that attempts to determine advertising conflicts as the human quality control team confirms or corrects when they review each image. This approach reduces the time to deploy the initial solution and more quickly increases efficiency. The AI model continually improves with human feedback and eventually, roles are flipped so that humans only review the exceptions and low-confidence results.
As the initial model matures, the data science team focuses on the model and data for the next violation type using the same iterative and incremental approach. The process is repeated until the entire solution is complete, or even changed, based on actual business value and real-world observations. The combination of breaking down the AI solution into smaller components of incremental value and repeating the process through each stage of the AI train+test+exception lifecycle accelerates value and reduces the risk for data science teams.
While commonplace in software development, the Agile approach is rare in AI. However, those teams adopting Agile AI are deploying more accurate and more valuable solutions faster. Agile AI is an immense help to data science teams and it expands the opportunity to integrate AI into other business functions more quickly.
Opinions expressed by DZone contributors are their own.