Is It Worth It? ROI of Recommender Systems

Recommender systems promise to reduce churn and increase sales. But how do you measure their actual success? What is it that you should measure? And what challenges should you look out for when you’re building your recommendation engine? In this article, I’ll discuss some challenges of recommendation engines, the ROI, and standard metrics to help evaluate their performance.

You might also like:  Building a Recommendation System Using Deep Learning Models

Challenges of Recommender Systems

Most articles about recommendation engines focus on all the bright sides of recommendations: personalized customer experience, lower churn, increase in sales, and more revenue. While all of that is true, as we can see looking at the examples of numerous companies including Amazon, adopting a new technology requires a strategic approach — so you should be realistic and well-prepared and not only optimistic about the future outcomes. There are some challenges that you have to be aware of.

Lack of Data

Data is the key element of any predictive model, and recommendation systems are no exception. They generate accurate recommendations based on available information. It only makes sense that the recommender systems thought to be the best ones are those from companies who are in possession of vast amounts of data, such as Google, Amazon, Netflix, or Spotify. Good recommender systems analyze item data and customer behavioral data to find similarities and suggest items. Artificial intelligence thrives on data: the more data the system has to work with, the better the outcomes.

Everything Is Changing

Data is changing, user preferences are changing, your business is changing all the time. That’s a whole lot of updates. How well will your algorithm keep up with the changes? Of course, you can have real-time recommendations that take into account the most recent data, but they are also more difficult to maintain. On the other hand, batch processing is easier to maintain but does not reflect the recent changes in data.

People 

Believe it or not, but people are the source of a large share of the problems with recommender systems. They may be unpredictable and yet expect technology to assist them no matter what. I may be browsing Amazon for a smartwatch today and leave the site, but tomorrow I won’t be interested in it anymore, now I need a present for my sister. If I want to buy a TV, and the system knows it, should it still recommend TVs or influence my behavior and recommend other items? There are a lot of difficult questions that may come to your mind when you focus on people – and focusing on people is still important because it’s them who buy your products.

Oh, and just one more small thing. As Steve Jobs said:

A lot of times, people don’t know what they want until you show it to them.

That’s why the “discovery” factor plays a significant role in recommendations. People may tend to watch similar shows and read similar books, let’s say sci-fi. Does that mean that all they want to see is sci-fi? They need diversity so they can discover things outside of what they already know too. 

Additionally, sometimes ratings don’t reflect reality. When people watch a stupid comedy, they may rate it lower than an Oscar movie. Let’s say 2 stars vs. 5 stars. This may teach the system that the Oscar movie is what should be suggested. However, the reality is not that simple. A stupid comedy may be worth 2 stars and still be a preferred choice on a Friday evening. 

Optimization

The recommender system should be getting better and better all the time. It’s not enough to just start it and let it run. Machine learning algorithms help the system “learn” the patterns, but the system still needs some guidance to provide relevant results. You need to improve it and make sure that whatever changes are introduced, you’re still going towards your business objective.

Have you heard of the Netflix prize competition? It was an open competition for the best collaborative filtering algorithm. The goal of the competition was to improve member retention. The winners won 1 million dollars. Netflix knows well that they have to improve, and the better they get, the more money they make. Recommendations contribute to lower churn, which in turn means higher monthly revenue.

ROI of Recommender Systems

Recommender systems are used to help companies make more money by reducing churn and increasing sales. Let’s look at some numbers to prove it’s true.

As mentioned above, Netflix is constantly improving its recommendation engine. They’ve invested a lot of money and time in it, but it pays off — recommendations influence 80% of content watched on Netflix. The algorithms contribute to lower churn and help the company save 1 billion dollars a year in value from customer retention. Netflix is also mastering the art on recommending as fast as possible — a typical member will usually lose interest after 60 to 90 seconds, having browsed 10 to 20 titles. That’s not a lot, and yet Netflix manages to retain customers and attract new ones. In the second quarter of 2017, Netflix had a 32.3% year-over-year growth, adding 5.2 million subscribers to the existing 99 million members in the previous quarter.

Now, let’s look at Amazon. Amazon’s revenue for the twelve months ending June 30, 2019, was $252.063B, a 21.11% increase year-over-year. Amazon is undoubtedly the king of online retail. The company introduced item-to-item collaborative filtering as early as 1998 and has been improving the recommendations ever since.

In many services, including e-commerce platforms, recommendations are only present at one stage of the customer journey, but Amazon has integrated recommendations at every step of the way to maximize order value. Amazon has never revealed how much revenue is generated thanks to recommendations but McKinsey estimates that 35 percent of what consumers purchase on Amazon comes from product recommendations.

Measuring ROI of Recommender Systems

Evaluation metrics for recommendation engines are necessary if you want to see your return on investment but also to help you improve your model. Below, I’ll present a list of some standard metrics used to evaluate recommender systems. However, you should keep in mind that your recommendation engine is there to meet your specific goals, so you can add different metrics to the list.

Interaction

It’s the most important metric group that evaluates the engagement of users with the recommendations. 

• Click-through rate (CTR)
• Conversion rate overall
• Conversion rate per user

Business Impact

The last group of metrics is the one that shows business results. 

• Proportion of orders with recommendations
• Recommended items per order
• Increase in average order value
• Increase in revenue
• Increase in customer retention
• Recommendations vs sold items

The chart below shows the predicted sales of recommended items compared to the actual number of sales. The more similar the bars are, the better recommendations are generated.

Again, these are just some of the metrics that you can use, and it’s crucial that you stick to the ones that measure the things that matter to your business. It’s always about the business first.

Is It Worth It?

I can’t answer this question for you. One thing I know for sure: the early adopters of AI technologies benefit from increased revenue and higher customer satisfaction, having unlocked the potential hidden in all the data; however, those who fall behind risk never catching up. It’s hard to compete with data-driven companies, especially when you’re not a data-driven organization yourself.

Further Reading

How to Develop a Simple Recommendations Engine Using Redis