AI Engineering Development Process

Motivation for AI Engineering Development Process

Artificial intelligence (AI) applications often involve not only classical application engineering but also elements of research. Sometimes it is not clear from the start which approach will be better and one needs to conduct experiments to evaluate multiple approaches. For example, if we are building a machine learning model we would need to evaluate and experiment with different features until we find an optimal feature set. 

Furthermore, if we are building machine learning models, usually debugging is not an easy task. Also in many cases, it is not trivial to evaluate the performance of statistical models and how this performance will translate to business value. All these factors can add an additional layer of complexity that the engineering teams need to cope with.

Development Process for Engineering AI Systems

One way to manage the inherent complexity in AI projects could be to introduce a clear methodology in the development process. An easy-to-understand process usually helps members of the engineering team and also business stakeholders to be on the same page where the project is at the moment and also what the next steps are. A well-defined process should help with the construction of a technical roadmap and also help in raising transparency in general.

Since AI projects can be seen as located at the intersection between application development and scientific research, we argue that these projects can benefit from some practices commonly used in research and academia. In this post, we will present an AI Engineering Cycle. The cycle is motivated by the work presented in this paper and it is very much related to the idea of the falsifiable hypothesis introduced by Karl Popper.

The AI Engineering Cycle can be seen in Fig. 2. Some steps probably could be seen as one logical step, however, we have split them here for more clarity. Additionally, the scope of this process is related to experimentation. It is not capturing all activities – e.g., gathering of test data is not depicted, however still needed.

In my experience, one of the key points to have clarity in AI projects is to always start an experiment with a clear falsifiable hypothesis, i.e. hypotheses which can be proven to be wrong. If the engineering team does not have a clear hypothesis in mind to start with, there is the risk that the team loses itself in experimentation – a problem that can be compared with chasing one’s tail.

After starting with a clear falsifiable hypothesis and going through the experimentation cycle one can evaluate the results and prove or disprove the hypothesis. With the new insights, the engineering team can then create a new hypothesis and start the next experimentation cycle.

Example

For example, if we want to evaluate whether a new feature will improve the performance of a machine learning model, one run through the AI engineering cycle could look as:

1. Falsifiable Hypothesis: The new feature X will not improve the performance of the model with the data set T.
2. Design an Experiment: In this case, the design of the experiment is quite simple. We just need to add the new feature in the training process, train the model and evaluate with the test data set.
3. Implement the Experiment: Add the new feature in the Training/Validation/Test data set and in the training process.
4. Run the experiment: Run the training to produce a model.
5. Evaluate results: Evaluate the model, one needs to evaluate with the same evaluation metrics as the baseline model in order to have comparability of course.
6. Draw conclusions about the hypothesis: If the evaluation metrics indicate a better performance we have disproved the negative hypothesis. The feature does improve the model with the given data set T. In the next experiment, we could for example test if the feature is still beneficial with a larger data set.

Conclusion

This short article proposes a simple development process for engineering AI systems. The presented process is hopefully generic enough to make sense for different applications.