Deep Web Tech Blog

Why Doesn’t Explorit Everywhere! Bring Back All Results?

One of the more common questions that I get from prospects and customers alike is why don’t we bring back all results from each of the sources that we federate? Just earlier in the week one of the librarians at one of our newest customers asked this question. I went back to our blog archive and dug up this wonderful blog article that Darcy wrote in 2015 – Getting the Best Results vs. Getting all of the Results and sent it on to our customer. I love it when I can answer a customer or prospect question by sending them a link to a blog article that answers their question.

So this afternoon I decided to expand a bit on Darcy’s original blog article.

In an effort at transparency and to inform our users of the status of searching, the user can look at Search Status popup that displays the list of sources searched with the number of results returned and the number of results found at the source (when the source provides this information). This Search Status popup is a link under the progress bar in the upper left hand corner of the Results page – the text of the link indicates the count of all sources involved in the search, e.g., “54 of 54 sources complete.”

Viewing the Search Status popup, the user can see, for a broad query, e.g., security, that collectively the sources may have available several hundred to thousands of results while we only retrieved up to the first 100 results. It does beg the question of why we can’t bring back all the results.

So let us for a moment go directly to one of the more popular sources that we federate — PubMed, a very large database of 20 million medical articles (some full-text but mostly just meta-data).

Doing the following PubMed searches:

     “myocardial infarction”  —  returns 213,186 results

     “myocardial infarction” AND aspirin  — returns 7,395 results

     “myocardial infarction” AND aspirin AND statins  — returns 542 results

Even with the most specific of the above queries, PubMed still returned 542 results, more results than most users will review, and certainly more than we would like to return from a source. However, we could retrieve the 542 result if we wanted to.

The above example illustrates one of my main responses to the question – Why do we not bring back all results? – What I say is that instead of focusing on Explorit Everywhere! bringing back more results, the focus should be on users realizing that the issue is to be more precise in their queries so that they are getting the most relevant results. It is not very useful to get all the results if they do not help the user find the answer they are looking for. Doing a broad search like “myocardial infarction” that found 213,186 results is not as helpful as doing a more precise search like “myocardial infarction” AND aspirin AND statins” with its 542 results. In the more precise search, the user is more likely to find a relevant answer.

In conclusion, when users issue more precise queries, they will find that Explorit Everywhere! returns most or all of the available results at each source, with the results ranked using our secret sauce so the user can quickly and easily find what they were looking for across all available sources. For the case where more results are available at the source and the user needs to examine all results (perhaps they are doing some legal due diligence) then the user can go directly to the source and conduct the search there.