|
About Deep Web Technologies
| Who is Deep Web Technologies? |
 |
| Deep Web Technologies was founded in 2002 by federated search pioneer Abe Lederman who serves as President and Chief Technology Officer. Deep Web Technologies sprang from Abe's three year relationship and successful track record of work completed with the U.S. Department of Energy's Office of Scientific and Technical Information (OSTI). From this affiliation, the first federated search for the federal government was born with the creation of Distributed Explorit™. Abe's 17-year background in the field of knowledge management, coupled with a network of connections from Verity and Los Alamos National Laboratory, and his pioneering work on the Explorit™ platform laid the groundwork for the company's rapid growth. |
| What is your vision? What is your mission? |
|
| Our vision is to provide a software platform that permits integration of "best of class" software tools, incorporates the wisdom of discipline-focused specialists, and delivers the best and most complete results that translate into better decision-making. To realize our vision, our mission is to continually upgrade our current proprietary software tools and add new ones to bring leading edge knowledge management technology to our clients. We view our customers as our partners and strive to learn about their businesses, including the types of intelligence they require, the top sources of content they use, and the ways they use the information, so that we can craft the best possible solution. |
| What expertise do you have in the federated search industry? |
|
| Deep Web Technologies has been in existence since 2002, focusing solely on federated search technologies. Deep Web Technologies' founder Abe Lederman has been involved in all aspects of the industry, from software development, to sales, marketing, and consulting for a number of years before that. Since its inception, Deep Web Technologies has developed two generations of federated search applications, the most current generation using leading edge technologies. |
back to top >
The Deep Web and Federated Search
| What is the Deep Web? |
|
| The Deep Web is the set of web-sites and their documents that cannot be accessed via crawler-type search engines such as Google. Deep web content typically lives inside of databases, and is accessed through search forms. Wikipedia has a good article about the Deep Web. |
| What is federated search? |
|
| Federated search is the technology of simultaneously searching multiple content sources from one search form and aggregating the results into a single results page. Federated search engines sometimes perform additional functions such as removing duplicates from the results lists and ranking documents against one another. Wikipedia has a good article about federated search. |
| What is a connector? |
|
| A connector is a piece of software that is written to access a content source. A connector must know the URL of the source, how to send search commands, what the search syntax is, and how to process the search results that are returned from a source. Connectors can be challenging to write if access to a source requires handling multiple steps, URL redirection, cookies, sessions, or authentication methods. |
| How does federated search work? |
|
| Federated search engines use software "connectors" to access information sources. The federated search engine takes the user's search query, transforms the search terms to match each content source's requirements, and submits the query to each of the sources simultaneously. When the search results come back from each of the sources, the federated search engine merges them together, modifying the look and feel of each of the result pages to have a single look and feel. |
| What is the relationship between the Deep Web and federated search? |
|
| Although federated search technically refers to the simultaneously search of multiple content sources regardless of how the content is accessed, the reality is that federated search is often performed on deep web content sources. |
| Why is it so difficult to search the Deep Web? |
|
| Searching the deep web is difficult because each source searched has a unique method of access. The federated search software must be configured to properly search each source and to process its results. Additionally, sources change their access methods from time to time, requiring modification or rewriting of the interface to that source. Some sources are very difficult to search because they require use of cookies, sessions, and authentication mechanisms. |
| How is federated search different from metasearch? |
|
| Meta search typically refers to a search engine that searches other search engines. However, some people use the term "metasearch" synonymously with the term "federated search." |
| Why do I need federated search? Can't I harvest / crawl / index everything? |
|
| There is a large amount of content that is not available to crawl-type search engines like Google. Federated search engines, in particular ones that perform deep web searches, are required to access this additional content. |
| What type of content can I find in the Deep Web via federated search that Google won't provide? |
|
| There are many scientific, technical, and business databases whose contents are not available to Google. Many but not all of these are subscription databases. |
back to top >
Products, Services, Support
| What products do you sell? |
|
| Deep Web Technologies' flagship product is its Explorit Research Acceleratorfederated search application. The product can be customized for specialized customer needs, both in terms of look-and-feel and to add functionality. Deep Web Technologies develops connectors for a wide range of content databases and will create custom connectors to meet your needs. |
| What services do you provide? |
|
| Deep Web Technologies can host your federated search application in our data center, or we can deploy it in your hosting environment. Additionally, we can maintain your application, including monitoring and updating of connectors as needed. We can also provide a needs assessment, consulting services, deployment and maintenance training, custom software development and look-and-feel design service. |
| What is unique about your software? |
|
| Deep Web Technologies prides itself on developing robust and scalable applications built on leading edge and open standards. Our application connectors are the most sophisticated in the industry, which means that they will search remote content more effectively than many other applications. Our relevance ranking algorithms are sophisticated and return highly relevant results from across a number of different sources. We offer rapid development and deployment of connectors and applications and we monitor connectors daily to quickly detect and correct problems. We gladly customize applications to meet customer needs. |
| What are alerts? How do they work? |
|
| The alert service allows for users to save their favorite searches in their private Explorit Research Acceleratoraccounts. These searches are typically performed on the users' behalf every week and new results are emailed to the user or can be viewed through an RSS feed. The alert service can be hosted at Deep Web Technologies' data center or performed at the customer's site. |
| How long does it take to deploy a solution? |
|
| Deployment time varies depending on a number of factors. An application built with a small number of connectors or with connectors that have already been developed and that has simple look-and-feel requirements may be deployed in a small number of days. If a custom solution is to be developed, involving complex layout work, numerous connectors, and custom development work the deployment schedule will be longer. |
back to top >
Federated Search and My Business
| How does federated search fit in with enterprise search? |
|
| As users demand applications that can search more source of content from fewer search pages it will only be a matter of time before the distinction between federated search and enterprise search disappears. Deep Web Technologies' Explorit Research Accelerator federated search application can be configured with custom connectors to search a number of repositories normally accessed via enterprise search providing seamless access to more content than federated search or enterprise search alone provides. |
| How does federated search improve my bottom line? |
|
| Federated search solutions reduce the time it takes to find relevant information, decreases the chance of missing relevant content and improves the utilization of paid content. Additionally, the time needed for researchers to learn the quirks of numerous search interfaces is eliminated. The time saved from not needing to search multiple sources plus the improved quality of documents found translates to labor and cost savings. |
| Where does the software run? Are you hosting it? Am I hosting it? Who is maintaining it? |
|
| Deep Web Technologies provides flexible hosting and maintenance options. For some customers we host their applications in our state-of-the-art data center, providing backups as well as application, operating system, hardware, and connector monitoring and support. With a hybrid installation, customers provide the hardware and operating system and we install, monitor and maintain the application on their behalf. Other customers license the application and host and maintain it themselves. |
back to top >
Technical Questions
| How do you handle duplicates? |
|
| Duplicate results from multiple sources should be removed to improve the user search experience. Deep Web Technologies' Explorit Research Accelerator application is flexible in its approach to de-duplication. Results with identical URL can be considered duplicates as can results that have the same title and author. The configuration of the de-duplication algorithm should be customized to the particular databases being searched based on how duplicates manifest themselves in the results, i.e. what fields are being duplicated. No one solution fits all deployments. |
| How do you access subscription content that requires authentication? |
|
| Deep Web Technologies has developed tremendous expertise in developing connectors to content requiring cookies, sessions, username/password authentication, and IP-based authentication. Our connectors perform the authentication steps just as they occur when a user accesses an authenticated database using a browser. |
| How do you handle advanced searches if a particular source doesn't support all the fields filled in? |
|
| Deep Web Technologies' Research Accelerator application (tm) has sophisticated support for mapping user search fields to fields supported by the remote source. The customer can decide which field or fields to search on the remote host on a per source basis in a number of flexible ways. Fields not available on the remote source can be ignored or the search engine can search different fields instead. |
| How do you perform relevance ranking? |
|
| Deep Web Technologies has developed a number of sophisticated relevance ranking algorithms. Our algorithms take into account the document source, the frequency of user search terms in various fields of the search results, and other factors. We compare search results from different sources against one another to determine ranks of individual results against the result sets. Users find our relevance ranking to be quite good, often better than that provided by the content provider. |
back to top > |
