In yesterday’s post, I discussed Hamlet Batista’s seven reasons why search results aren’t more relevant. Today, I want to explore the heart of the question a bit further, namely:

What is relevance in search?

This is an important question, particularly for us here at VortexDNA, since the purpose of our technology is to improve relevance. If we’re not totally clear on what relevance is, though, how can we do our job?

In addition, although the definition of relevance might seem obvious, there’s some intrigue there if we dig a bit deeper. I commented yesterday that I agreed with Batista’s first and most important point:

Relevance is subjective.

If we’re in agreement on that (and if any of you disagree with the above statement, I’d love to hear from you), then how can we define it? It’s like saying you have a company that can increase beauty in the eyes of the beholder.

If relevance is subjective, then how can you claim that the ideal would be 100% precision (efficiency in returning only relevant results) and 100% recall (inclusion of all relevant results)?

In fact, how could you ever begin to measure recall? It would be crazy. First, you’d have to get all possible results in the world for a particular search, whether or not they’re included in the results generated by a particular search engine. Then you’d have to find out which of those results are relevant—and because relevance is subjective, that answer will be different for every user. Finally, you’d have to check all relevant results for a particular user against the complete set of results returned by a given search engine. That comparison would only give the recall for that query/user combination.

No wonder Batista says users care more about precision than recall.

So what is relevance? A good old Google ‘define:’ search (sorry, Charles) yielded a stack of results, all of which had one thing in common:

Relevance is only relevance in relation to something or someone.

Personally, I believe that you can edit that sentence down further: relevance is only relevance in relation to someone. It always comes back to people. I don’t care that a given answer is relevant to my search string; in fact, it’s entirely likely that my search string isn’t the most appropriate one for the information I’m after. Are the results relevant to me? That’s what I’m after.

A while back I wrote about discovery in search. (Thanks again, David Berkowitz, for prompting the post.) Discovery is essentially that quality of being able to find what you didn’t know you were looking for, and it’s one of the driving forces behind the evolution of the Web.

Think back, and try to remember how much you had been able to anticipate what you looked at online in the past week. Was any of your Web activity happenstance? A link sent by a friend to a site you didn’t know you’d visit? A sideways reference from one page to another? A search result you didn’t expect yet thoroughly enjoyed?

Discovery has another impact on increasing relevance. If people don’t know what they’re after, how can search engines know whether or not they’re delivering it?

Now let’s combine thes two concepts: people don’t always know what they’re looking for, and what we want is different for each of us. So what is relevant? If we limit relevance to answers directly related to the specific query we’ve entered, we eliminate discovery. Limiting relevance to discovery would make search a free-for-all. I suggest that relevance could be defined as a set of qualities:

Qualities of relevance

1. The human connection: relevant results connect to the searcher.
2. The discovery angle: relevance can be expected or unexpected.
3. The subjective nature: the degree of relevance changes from person to person and moment to moment.
4. The measurement conundrum: the degree of relevance occurs along a spectrum that makes it impossible to achieve 100%.

What does the above mean for companies who work in the relevance space? It means that any technology that aims to improve relevance must be able to address its various facets: it must be able to deliver results that connect with the individual user, at that very moment the user is looking, and whether the user expects to find the results or not.

It also means that, like so much in life, perfection is unattainable. Our aim is to move incrementally and continually along the spectrum.

What do you think is the most important aspect of relevance? And do you think I’ve missed some of its qualities? I’d be delighted to hear from you, agreement and disagreement alike.

This entry was posted on Tuesday, July 24th, 2007 at 11:21 am and is filed under Relevance, Search. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.