Over a million developers have joined DZone.

Measuring Search Quality: Retrieval and Relevance Metrics

· Java Zone

What every Java engineer should know about microservices: Reactive Microservices Architecture.  Brought to you in partnership with Lightbend.

I am trying to put together a framework for search quality evaluation for a specialist information provider.

At the moment quality is measured by counting the number of hits for certain key docs across various queries, and monitoring changes on a regular schedule. I’d like to broaden this out into something more scalable and robust, from which a more extensive range of metrics can be calculated. (As an aside, I know there are many ways of evaluating the overall search experience, but I’m focusing solely on ranked retrieval and relevance here).

We are in the fortunate position of being able to acquire binary relevance judgements from SMEs, so can aspire to something like the TREC approach:


But of course we are running just a single site search engine here, so can’t pool results across runs to produce a consolidated ‘gold standard’ result set as you would in the TREC framework.

I am sure this scenario repeats the world over. One solution I can think of is to run your existing search engine with various alternative configurations, e.g. precision oriented, recall oriented, freshness oriented, etc. and aggregate the top N results from each to emulate the pooling approach. Can anyone suggest any others? Or perhaps an alternative method entirely?

Jason Hull one of DZone's MVBs recommended looking at this article for some answers.

Microservices for Java, explained. Revitalize your legacy systems (and your career) with Reactive Microservices Architecture, a free O'Reilly book. Brought to you in partnership with Lightbend.


Published at DZone with permission of Tony Russell-rose, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}