Numerous myths and misunderstandings exist about federated search; For example, how does real-time search differ from an indexed search and why is this important? At Deep Web Technologies, we welcome your questions and we feel confident that once you recognize the value of the technology and understand our offerings you’ll choose Deep Web Technologies.
We invite you to read the questions and answers below and to contact us with questions we didn’t cover.
| Federated Search and My Business |
|---|
| How does federated search fit in with enterprise search? How does federated search improve my bottom line? How do you connect me to my knowledge? |
The Deep Web and Federated Search
The Deep Web is the set of web sites and their documents that cannot be accessed via crawler-type search engines such as Google. Deep web content typically lives inside of databases, and is accessed through search forms. Wikipedia has a good article about the Deep Web.
Federated search is the technology of simultaneously searching multiple content sources from one search form and aggregating the results into a single results page. Federated search engines sometimes perform additional functions such as removing duplicates from the results lists and ranking documents against one another. Wikipedia has a good article about federated search.
How does federated search work?
Federated search engines use software “connectors” to access information sources. The federated search engine takes the user’s search query, transforms the search terms to match each content source’s requirements, and submits the query to each of the sources simultaneously. When the search results come back from each of the sources, the federated search engine merges them together into a single set of results pages with a unified look and feel.
What is the relationship between the Deep Web and federated search?
Although federated search technically refers to the simultaneous search of multiple content sources regardless of how the content is accessed, the reality is that federated search is often performed on deep web content sources.
Why is it so difficult to search the Deep Web?
Searching the deep web is difficult because each source searched has a unique method of access. The federated search software must be configured to properly search each source and to process its results. Additionally, sources change their access methods from time to time, requiring modification or rewriting of the interface to that source. Some sources are very difficult to search because they require use of cookies, sessions, and authentication mechanisms.
How is federated search different from metasearch?
Meta search typically refers to a search engine that searches other search engines. However, some people use the term “metasearch” synonymously with the term “federated search.”
Why do I need federated search? Can’t I harvest / crawl / index everything like Google does?
There is a large amount of content that is not available to crawl-type search engines like Google. Federated search engines, in particular ones that perform deep web searches, are required to access this additional content.
What type of content can I find in the Deep Web via federated search that Google won’t provide?
There are many scientific, technical, and business databases whose contents are not available to Google. Many but not all of these are subscription databases.
Federated Search and My Business
How does federated search fit in with enterprise search?
As users demand applications that can search more sources of content from fewer search pages it will only be a matter of time before the distinction between federated search and enterprise search disappears. Deep Web Technologies’ Explorit Research Accelerator federated search application can be configured with custom connectors to search a number of repositories normally accessed via enterprise search providing seamless access to more content than federated search or enterprise search alone provides.
How does federated search improve my bottom line?
Federated search solutions reduce the time it takes to find relevant information, decreases the chance of missing relevant content and improves the utilization of paid content. Additionally, the time needed for researchers to learn the quirks of numerous search interfaces is eliminated. The time saved from not needing to search multiple sources plus the improved quality of documents found translates to labor and cost savings.
How does federated search connect me to my knowledge?
Federated search connects directly to your knowledge sources, whether they be subscription, internal or public. Information that you would normally search to by going to the source, federated search uses connectors for, which retrieve the results and returns them to you. In this manner, searching ten, twenty or more of your knowledge sources in real time is simply a matter of typing in your query. You are assured that the information returned is the same information that is on the source.
Technical Questions
A connector is a piece of software that is written to access a content source. A connector must know the URL of the source, how to send search commands, what the search syntax is, and how to process the search results that are returned from a source. Connectors can be challenging to write if access to a source requires handling multiple steps, URL redirection, cookies, sessions, or authentication methods.
Duplicate results from multiple sources should be removed to improve the user search experience. Deep Web Technologies’ Explorit Research Accelerator application is flexible in its approach to de-duplication. Results with identical URL’s can be considered duplicates as can results that have the same title and author. The configuration of the de-duplication algorithm should be customized to the particular databases being searched based on how duplicates manifest themselves in the results, i.e. what fields are being duplicated. No one solution fits all deployments.
Why is Explorit faster than other federated searches?
Because Explorit searches in real-time, it is dependent on the speed of the knowledge sources to complete user queries. Sometimes, that can take 10, 20 or even 30 seconds to complete. Explorit federated search speeds this up with “incremental search” by displaying the results from the fastest sources immediately while the search continues in the background on the slower sources. In this way, real-time quality results are not sacrificed for speed.
How do you access subscription content that requires authentication?
Deep Web Technologies has developed tremendous expertise in developing connectors to content requiring cookies, sessions, username/password authentication, and IP-based authentication. Our connectors perform the authentication steps just as they occur when a user accesses an authenticated database using a browser.
Beyond the ability of simple sorting to organize search results by author, date or publication, clustering allows users to organize search results by topic. Our smart clustering software is able to organize the results into topics and subtopics at the time of search, allowing researchers to drill down into details of a topic in an intuitive way.
How do you handle advanced searches if a particular source doesn’t support all the fields filled in?
A Deep Web Technologies’ Research Accelerator(TM) application has sophisticated support for mapping user search fields to fields supported by the remote source. The customer can decide which field or fields to search on the remote host on a per source basis in a number of flexible ways. Fields not available on the remote source can be ignored or the search engine can search different fields instead.
How do you perform relevance ranking?
Deep Web Technologies has developed a number of sophisticated relevance ranking algorithms. Our algorithms take into account the document source, the frequency of user search terms in various fields of the search results, and other factors. We compare search results from different sources against one another to determine ranks of individual results against the result sets. Users find our relevance ranking to be quite good, often better than that provided by the content provider.
Where would my application run? Would you host it? Would I host it? Who monitors and maintains it?
Deep Web Technologies provides flexible hosting and maintenance options. For some customers we host their applications in our state-of-the-art data center, providing backups as well as application, operating system, hardware, and connector monitoring and support. With a hybrid installation, customers provide the hardware and operating system and we install, monitor and maintain the application on their behalf. Other customers license the application and host and maintain it themselves.
What are alerts and how do they work?
Alerts are simply search terms that you have defined, that run on a schedule and are delivered directly to your email inbox or RSS feed. Explorit alerts can be set up for daily, weekly or monthly delivery and always deliver the freshest results from your knowledge sources, regardless of whether or not your sources have an alerts system.
