Model Drift in Computer Vision Models: Understanding the Problem

What Is a Computer Vision Model?

Computer vision (CV) is a field within artificial intelligence and machine learning (ML) that involves computer programs interpreting the meaning of image and video inputs. A computer vision algorithm can analyze specific criteria in visual data and use its interpretations to implement predictive and decision-making tasks.

Modern computer vision models are highly sophisticated and cover many use cases to automate critical decision-making systems and enhance business efficiency. As a type of machine learning algorithm, computer vision models are challenging to deploy.

Of the many algorithms developed, only a handful of models continue in production. The development life cycle of a computer vision model begins with the collection of high-quality visual data. The next stages include data preparation, model training and evaluation, software deployment, monitoring, and re-training.

What Is Model Drift?

In computer vision and other types of machine learning, model drift refers to performance degradation that negatively affects the model’s predictive abilities over time. This drift results from changes in real-world environments and various other factors, such as changes in digital environments and relationships between variables. 

The two kinds of model drift are concept and data drift. Concept drift occurs when the properties of dependent variables change. The function that models the relationship between each feature and a given variable becomes unsuitable for the new environment. An example of this is the changing definition of spam emails over time.

Data drift occurs when the underlying distributions of a model’s features change over time. It may result from many types of changes — for example, behavioral changes in different conditions, seasonal changes, or changes in underlying populations. Another example of data drift is the changed values of features resulting from unusual activity during the pandemic. 

How to Detect Model Drift

The best way to accurately detect computer vision model drift is to compare the ML model’s predicted values with the true values. The model’s accuracy deteriorates when the predicted values drift further and further from the true values. When the relevant metric drops below a specified threshold, it indicates the model has drifted.

The F1 score is a popular metric for data scientists evaluating the accuracy of machine learning models — it encompasses a model’s recall and precision. However, other metrics may be more relevant depending on the use case. Type two errors, for example, are crucial for image recognition models that analyze tumors. 

Monitoring a model’s accuracy is not always possible — sometimes, obtaining the true and predicted data is infeasible. For example, suppose the model predicts a public company’s net income. In that case, it is only possible to measure the predictions’ accuracy when the company produces a quarterly earnings report (four times per year). 

Here are some alternatives for situations where comparing predicted and actual values are impossible: 

• Kolmogorov-Smirnov test (K-S) — this nonparametric test compares the cumulative predictions of two data sets, a training data set and a post-training data set. The test’s null hypothesis is that the two data sets have identical distributions. If the model is drifting, the null will be rejected. 
• Population Stability Index (PSI) — this metric is useful for measuring how the distribution of variables changes over time. It is commonly used to monitor changes in a population’s characteristics to detect model degradation.
• Z-score — this metric compares training and live data feature distributions. For instance, if several live data points for a specific variable have a score of +/- 2, it suggests the variable’s distribution has shifted.

Overcoming Model Drift

Here are some techniques to help minimize model drift.

Accurate Data Labeling

Preventing data drift requires retraining the model with all the available data, including data from before the drift and newer data. It is important to accurately label all new data before adding it to the training data set.

Data labeling involves attaching labels to specific variables to convey meaning. It is crucial for supervised ML projects where the target data might be an image or video snippet. Data labeling helps counter data drift, maintaining the machine learning model’s performance in production.

Data labeling is key to supervised machine learning because the model uses the input data based on the attached labels. For instance, a computer vision model can learn to identify product placement in video footage by training on videos with labeled products.

Data labeling allows the model training team to incorporate data from shifted distributions into the main training data, helping reduce data drift. It could be an improvised effort. Human annotators can leverage intuitive labeling software to process and annotate data quickly.

Regular Data Updates and Weighting

It is often possible to update a machine learning model. This approach to updating the model is more efficient than older approaches like periodic re-fitting. It does not discard the static model entirely. Instead, the model’s existing state serves as a starting point for the fitting process, updating the model fit with samples of the latest historical data.

An example of a use case that can benefit from this approach is a machine learning algorithm that uses coefficients or weights (e.g., neural networks, regression algorithms). Some ML algorithms enable the weighting of input data (identifying its importance).

For example, the weighting used could be inversely proportional to the data’s age, prioritizing newer data with higher weights and de-emphasizing older data with lower weights. 

Model Tuning or Retraining

When the ML project team detects data or other model drift, it can retrain the model using newer data. The approach will be different depending on the type of drift:

• Outdated data — if the older data is no longer relevant and could impact model accuracy, the retraining process should only use newer data. 
• Insufficient data — if the older data won’t make the predictions inaccurate, the retraining should use all the data available.
• Models that allow weighting — the retraining process can use all available data, but the team, where possible, should assign higher weights to the new data to ensure the model pays more attention to it.

Another approach is online machine learning, with the model continuously learning from real-time data feeds. It allows the model to stay up to date with changing datasets.

In some cases, it is not enough to retrain the model, so it might be necessary to rebuild the model. If the model was built based on outdated training data, it might need to be rebuilt with newer data. The team can run experiments using different model architectures, features, or hyperparameters to update the model and keep it in line with more recent data.

Conclusion

In this article, I introduced the concept of model drift and discussed three strategies for overcoming it in computer vision models:

• Accurate data labeling — correctly labeling new data flows can reduce model drift over time. 
• Regular data updates and weighting — the model’s existing state can be a starting point, and it should be continuously updated with samples of the latest historical data.
• Model tuning or retraining — retraining the data with a fresh dataset is often the best way to prevent model drift.

I hope this is useful as you improve the performance of your computer vision models.