How to Use Machine Learning to Improve Search

Great talking with Grant Ingersoll, founder and CTO of Lucidworks, about the state of artificial intelligence (AI) and machine learning (ML). Grant and his team use machine learning to build and improve, search applications.

Q: What are the keys to a successful AI/ML strategy?

A: In a market like search, there is a significant opportunity for monetization with high-quality results in e-commerce and other customer-facing industries that result in demand creation. It’s critical to build in functionality that can be put into production. We help by taking care of the plumbing around the lack of AI/ML talent.

Q: How can companies benefit from AI/ML?

A: Since we focus on returning relevant results to end users we focus on the precise application of machine learning in the discovery space. We tune relevant information to build better recommendation and prediction engines that deliver what the customer is looking for. We use signal capture of user feedback with searchable data. Most data scientists don’t understand how search works. We capture data and leverage using Spark and Solr for clustering, anomaly detection, and joining data in the search engine. The same data that is used for query predictions is used for all other applications. This leads to massive efficiency and productivity gains so the product pays for itself in three weeks.

Q: How has AI/ML changed in the past year?

A: There has been interest and hype from the deep learning (DL) side of the house. It takes a lot of hardware and data to get started but it’s finally getting real for people beyond Google and Facebook. There’s a groundswell of interest from engineers that have not played in this space before. There’s also greater industry interest due to TensorFlow and the tremendous revenue generation and cost savings opportunities around preventive and predictive maintenance.

Q: What are some real-world problems you are helping your clients solve?

A: Monetizing search in e-commerce and interactions with users. The biggest impact is learning from customers and leveraging from their clickstream to improve relevance. We have a large e-commerce client that was hand-tuning search that is now using a learned relevance model that’s more accurate and relevant. In traditional enterprises, they’re using AI?ML to have a 360-degree view of their customer and for their CSRs to know the next best action for a particular customer. All interactions with an investment bank are entered into our engine so we can let the reps know what the customer will be most interested in while also empowering the reps to know more about the customer and provide a more personal touch. We also enable reps to look up a specific event and then drill down on the relevant elements of the event.

Q: What are the most common issues you see preventing companies from realizing the benefits of AI/ML?

A: Clients expect magic and do not realize the amount of work involved in the initial engineering of the data. We try to manage expectations about what’s real and practical and that AI/ML is not a silver bullet. AI/ML is a long-term investment and a long-term play. You need to identify the specific use case and the problem you are trying to solve. A more personal product is able to take feedback from the customer to provide a better customer experience.

Q: Where do you see the greatest opportunities in the implementation of AI/ML?

A: Bringing relevant information to the forefront. Deep learning is unlocking new content on the search side like images and audio so you can query by image or voice.

Q: What skills do developers need to be proficient with AI/ML projects?

A: Have a learning mindset. Dig into open source and work on projects that are inherently interesting to you. Do tutorials. Be skeptical. If the results look too good to be true, they probably are – take another look at them. Brush up on your math skills – statistics, linear algebra, and calculus. AI/ML isn’t just for developers. Companies are in desperate need for product managers. That’s another way you can be involved without writing code.