Deep Web Tech Blog

Getting the Best Results vs. Getting All of the Results

A question we hear regularly is, “Why doesn’t Explorit Everywhere! return all of the results from every source that is searched?” For example, if a user goes directly to a source to search, they may find thousands of results for their query.  But, performing the same search on an Explorit Everywhere! application may only return 100 results from that source.  Why aren’t we returning the thousands of results like the source does?

DWT specifically returns up to 100 of the top results (unless our customer specifies that we return more) from each source to ultimately avoid overloading the user with information that may not be relevant to their search. Because the majority of Explorit Everywhere! applications have at least 10 sources of information, if each of those 10 sources returns 100 results, then the user will see 1000 results for their query, divided into a default of 20 results per page, or 50 pages of results total. Each of those results has been ranked as relevant to the user’s query with the most relevant results across all of the sources on page 1. Of course, we hope that the gold nugget is right at the top, front and center.  (You can read more about how DWT ranks results on this post –  Ranking: The Secret Sauce for Searching the Deep Web.) But, if we returned all of the results from all of the sources, then the total number of results and total number of pages increases to a dizzying number.

We know from the countless SEO studies on Google’s results placement that the majority of consumers using Google rarely click to the second page of results. In fact, page 2 and 3 may get only around 6% of the clicks on any given search. (Marketing Land).  While the likelihood of next page clicks does go up with age and education, we’ve found that it is unlikely even for our erudite Explorit Everywhere! users to click through 50 pages of results.  Rather, most researchers will perform a new search or refine their search if they don’t find what they are looking for within the first couple of pages.

repeat the search with the omitted results included.”  Try clicking that link to see how many results you actually get – probably around 900 results, or 9 pages. So much for the millions that are available! Even Google tries to avoid overloading users by limiting the vast number of results for broad queries.

For those researchers preferring to narrow their research to just one or two relevant sources on Explorit Everywhere!, it’s possible that 100 results from each source may not be enough. If a source has particularly relevant results, then we suggest capitalizing on that information by going directly to that source to continue searching. Use Explorit Everywhere! as a tool to not only find relevant results, but to find relevant sources of information to further your information discovery.