Ladies and gentlemen, children of all ages…

It’s here!

Seven participants, seven wise people, seven sages stepped up to the plate to respond to my challenge and join the Rising Star Dream Team. Their brief was simple and unconstrained: write your thoughts on The Future of Search.

I am in awe of the contributions I received for our collaborative report. From multi-page treatises to paragraphs of stream of consciousness, the submissions were thoughtful, provoking, and informed.

When I began this exercise, I thought I’d get a bit of information here and there, and I’d put it all together into one post, maybe two. Instead, I find myself introducing here a series of guest posts from our authors, culminating in the posting of a complete PDF report comprising all of the entries.

The schedule will be as follows (I’ll use US days to keep things simple—it’s tomorrow in New Zealand):

• This post: Introduction (above), schedule (here), synopsis (below)
• Later today: Andrew Matthews from The Wandering Glitch 2
• Thursday: Ran Geva from Omgili and Charles Knight from AltSearchEngines
• Friday: Branton Kenton-Dau from VortexDNA and Nitin Karandikar from Software Abstractions
• Monday: Raf Manji from VortexDNA / Sustento and The Master Report, downloadable for your enjoyment!

Synopsis
Henry Ford, who was known for his effective turn of phrase, once said, ‘If I’d have asked people what they wanted, they would have said a faster horse.’ One of the really interesting things about the contributions on the future of search was that very few people suggested a faster horse.

I’ve broken this summary down into some of the recurring themes:

• Searchless search
• Collaborative search
• Search that understands
• Contextual search
• User interface

The bottom line response, nearly universal across everyone who participated, is this:

Search has the potential to look dramatically different from how it looks today.

The fact that the current standard hasn’t changed significantly over the past ten years doesn’t mean it won’t. From search that goes with you to search that understands you to search that you co-create, we’re not done innovating.

Never bet against the infinite power of human creativity.

Searchless search

In a dramatic example of not offering a faster horse, more than one contributor talked about the tables of search turning from seek to receive. Essentially, they said, search as we know it will either cease to exist or cease being the dominant paradigm. Just as RSS feeds transformed the way we absorb news and blogs, there’s the potential for search to become a more passive process, where relevant content comes to the user.

Raf Manji had this to say:

We will no longer search for stuff we will simply receive what we need. Search is an external process: I am looking for something. Receive is an internal process: I have everything I need inside me.

Nitin concurred, offering the possibility of ‘notification’:

This can be best explained as a Reverse search, where it is the content that finds the user - thus turning the concept of search on its head.

Don’t think there’s any way to obliterate search as we know it? Perhaps not… but perhaps.

Collaborative search

The report itself is an example of how collaboration is exponentially greater than individual effort. Trust me, I could not have come up with it on my own! By distributing the effort we ended up with something complex, intriguing and thought-provoking.

At least three people pointed out that the concept of collaboration can and should be applied to search. Branton’s entire piece was on empowering people—everyone—to participate.

However brilliant the engineers are at Google, or Hadar Shemtov and his Natural Language team at Yahoo! are (and they ARE BRILLIANT), there is no way of outsurpassing the creative brilliance of millions of people collaborating to create a better search.

Charles was equally passionate, naming his piece ‘Collaborate or Perish’. From his perspective, though, the collaboration must occur among the many alternative search engines that are out there: those who are already innovating, disrupting and shifting paradigms as if their lives depended on it.

This probably won’t surprise anybody, but the Future of Search rests with the 100 or so best Alternative search engines.

Visit my blog (”Visit my blog, please!”) and you can scroll to your hearts content until you are convinced that they already have all of the innovation that they need to go after the Big G. But, and this should be the “but” heard ’round the blogosphere, they have been, they are, and they will be just so many gnats on an elephant’s back as long as they remain separate.

What would worry you most: 100 little BBs, 10 bullets, or one cannonball coming at you? Which would you prefer on your arm, the soft rays of the beach sun, or the same rays focused to a super-hot point with a magnifing glass? (There’s that new UI again!) How about a drag race between a Corvette and thirty 10-hp mopeds, or one 300-hp Camaro? There is strength in numbers, but not when they are individually wrapped. Collaborate or Perish. If the Alts don’t begin their collaboration now, the Google cycle will just continue unchecked.

Ran also touched on the concept, but in his piece it was in the context of distributed crawling:

The future of the crawlers should be in diverting resources from hundreds of centralized crawlers to the millions of users surfing the web. Let the user’s computers act as crawlers. This will save bandwidth and be much more efficient in finding the “dark parts” of the web. I actually think Google is already utilizing this method when we use some of its free software. They of course ask our permission to do so as they should.

The trend is clear, and it looks like we all recognize it: the more we can tap into the power of people, the greater chance we have for success.

Search that understands

AKA semantic search. Search that derives meaning from your query and delivers results based on the meaning, not the words.

Andrew had a lot to say about this one:

Most users will not ever use the advanced features of an engine like Google. To reach them, we need a way to produce semantically meaningful queries to match the semantically meaningful data that will be out there on the web. How can we do that? I think the answer lies in context. Assuming that the search engine doesn’t understand a word that you say when you type in a sequence of search keywords, it makes sense that the relevance of the results will decline as the number of possible misinterpretations increases. The English language, or any other natural language for that matter, is full of so many nuances and alternate meanings for each significant word that the space of possible interpretations for each word increases dramatically. As a consequence, the chances of a search engine being able to deliver relevant results are bound to decline at a similar rate. In fact, the problem is worse than that. The rate of decline of relevance is going to be the product of the probabilities of getting the sense of each word right for each word in the query. If there are five ‘senses’ to each of two words in a search string, then the chances of you getting the right result are going to be one in twenty five. A long shot. There are various heuristics that can be used to shorten the odds. But in the end, you’re never going to consistently get decent results out of a machine that doesn’t understand you or the documents that it has indexed. Enter the semantic web.

There’s more, but I’m putting his complete piece up later today, so you can read it then.

He wasn’t alone, either. Ephraim touched on it:

In ten years search will be in the form of an electronic assistant, most likely activated by voice. Users will talk to their assistant and describe what they are looking for. The assistant will use a rules engine in order to “understand” (i.e., place in its proper context) a query. When necessary the assistant will be able to ask questions in order to narrow the search down.

The search assistant will also learn from previous search queries in order to understand individual phraseology quirks of the user. Much like speech recognition can understand a word spoken in English by an Australian, British or American speaker.

As did Ran:

Currently, we as users need to rewrite the way we speak in order to interact with a search engine. Many times we need to completely rephrase a sentence so a search engine will “understand” it correctly (hopefully). No doubt, there is a need to solve this problem. Again, this is not the case when the search query is simple and contains only few and straightforward keywords. Unfortunately, many times this is not the case and a good search engine should be able to analyze the meaning of our search query. Semantic search engines are on the right path.

Contextual search

Andrew also went into a thorough exploration of contextual search, what he calls embedded search:

In my case, I spend most of my day inside a programmer’s IDE (integrated development environment) – like a glorified text editor for source code. When I’m in there I’m constantly faced with little puzzles that crop up to do with the program I’m writing. When I need to get a piece of reference information I have to come out of the IDE and call up a browser and perform the search in that before switching back to what I was doing. That context switching is distracting and I’d prefer the whole experience to be smoother. In addition to that, there is a whole bunch of contextual information that exists inside of the application that is (potentially) germane to the search I’m doing. Embedding the internet search facilities inside of general purpose applications such as my IDE provides a wealth of information to the search engine with which to automatically filter the results or to target the search.

User interface

Finally (for now), more than one participant discussed the user interface. Ran was firm on this one:

If a search engine conquered all the technological barriers but the presentation layer isn’t good it failed. The presentation is the place where the user sees the results returned in response to a query. If the results weren’t good enough, the search engine should supply an easy way to refine them. The user has very little tolerance, the interface should be highly intuitive and fast.

Giving too much information around the results is a big mistake, since the user will get lost. Too much flash and animation is cool and fun but not in the long run. When a user is looking for information it should be to the point.

Nitin also spoke about the importance of the interface, concurring that it has a dramatic impact on the effectiveness of a given search technology.

Search engines of the future will likely implement completely new paradigms for users to navigate and view search results. Often, meta-results - representing information about the results - are as important as the results themselves: users can figure out where a given result fits into the overall universe of results, and find the related results to an item of interest that has been found.

This is only a glimpse of the full discussion; please tune in for all of the individual pieces and the full report.

And please let us hear from you! If you’ve got a response to something you’ve read here or your own vision for the future of search, leave a comment or shoot me an email. This project is a group effort, and you’re always invited.

This entry was posted on Wednesday, September 5th, 2007 at 11:09 am and is filed under People matter, Search. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.