Deep Web Tech Blog

  • Free Deep Web Search Portals

    Deep Web Technologies proudly powers the technology behind many public search portals, some buiBinary code and multiple screenslt for clients to surface their information, others built by DWT to showcase our technology. While most public portals don’t have the subscription or premium resources available to private clients, many of them have deep web resources otherwise unavailable through popular search engines.

    Try one of these public portals: searches over 60 databases and over 2200 selected websites from 15 federal agencies, offering 200 million pages of authoritative U.S. government science information including research and development results. is governed by the interagency Alliance.
  is a global science gateway comprised of national and international scientific databases and accelerates scientific discovery and progress by providing one-stop searching of databases from around the world (Architecture: What is under the Hood). Multilingual provides real-time searching and translation of globally-dispersed multilingual scientific literature.

    National Library of EnergyThe Department of Energy (DOE) National Library of EnergyBeta (NLEBeta) is a national resource to advance energy literacy, innovation and security. The NLEBeta is a new search tool designed to make it easier for American citizens to find and access information from across the DOE complex nationwide, without knowing DOE’s organizational structure. Such information is offered by DOE to advance its four broad mission areas:

    • Science and R&D
    • Energy and Technology for Industry and Homeowners
    • Energy Market Information and Analysis
    • Nuclear Security and Environmental Management
    National Library of Energy

    Missouri Digital HeritageThrough the Missouri Digital Heritage Initiative, the Missouri State Archives and the Missouri State Library, in partnership with the State Historical Society of Missouri, are assisting institutions across the state in digitizing their records and placing them online for easy access.

    Missouri Digital Heritage Federated Search

    UNECA ASKIAThe Access to Scientific and Socio-economic Information in Africa (ASKIA) Initiative is under the Public Information & Knowledge Management Division of the United Nations Economic Commission for Africa. It defines a framework for bringing together scientific and socio-economic information for the African community, including scientists, researchers, academics, students, economists and, policy-makers, over an interactive online portal acting as a one-stop shop to such knowledge and associated information from/on Africa. The overall goal of the initiative is to strengthen knowledge discovery and access by tapping into global scientific and socio-economic knowledge on and from Africa.


    Biznar – 107 business sources available for market and industry research. Includes search of patents and social media.


    Mednar – A deep web medical search of over 60 public medical resources.


    Environar – Environar searches scientific and government databases in support of environmental, chemical and biological research.


  • DWT and Wellspring Partner to Find and Evaluate Emerging Technologies

    WS Logo_BlueForecasting and scouting of new technologies is integral to innovative companies. The need to expertly and systematically identify emerging tech is increasing as more and more digital records are added to the ever-growing pool of knowledge assets.

    With our newly integrated service, DWT and Wellspring offer a simple approach to find emerging technologies. Rather than the hunter-gatherer method of looking at each source serially, we’ve created a solution to simultaneously search patents, publications, scientific journals and invention databases for fast tech discovery by innovation-driven companies. Within seconds of a query, organizations will have an aggregated list of relevant results gleaned from over 100 million records, which can then be routed for evaluation and action.

    Dr. Robert Lowe, Wellspring’s CEO, said, “DWT’s tools and content integration are logical strategic steps of Wellspring’s product evolution. DWT represents another industry-leading data service we have integrated into our products this year, along with CPA Global and DocuSign. The seamless integration enables our clients to find and import critical information without any data entry. Ultimately, the DWT partnership results in better informed decisions for our clients.”

    Abe Lederman, Deep Web Technologies’ CEO and CTO, added, “We are very excited to work with Wellspring in the technology scouting and corporate venturing market. We worked closely with Wellspring to provide their customers with access to a curated set of high-quality science and technology information sources, including Wellspring’s Flintbox, a leading technology marketplace for technology transfer and technology scout professionals.”

    Read the press release from September 23, 2015.

  • Deep Web for Enterprises – what you can learn about competitors and customers (and vice-versa)

    Editor’s Note: This is a guest article by Andreea-roxana Obreja. Andreea graduated from the University of Portsmouth, AndreeaUnited Kingdom with a First Class Honours Degree in Business Information Systems. Her personal interest for covert data and online research have inspired her to author a comprehensive review of the potential of the Deep Web as a business tool for her final year project. The project has been awarded the Clever Touch Prize for the most Original Business Systems Project by the University of Portsmouth. The conclusions of her project will be presented at the 12th Conference of the Italian Chapter of AIS at the Sapienza University of Rome.

    We might be using the “Deep Web” every day without calling it this way or even being aware of its existence. Simply filling in a web form enables us to access the Deep Web and retrieve data from a variety of databases, some of which are free, subscription-based or have major access costs attached. Any online data used for business purposes (not necessarily the same purposes for which it has been collected) can be risky, but not knowing what data there is out there about you and your company represents a significantly higher threat. On the other hand, a thorough, Deep Web search can greatly benefit companies researching competitors, potential employees, customers and business trends.

    There are various types of data that can be accessed using intermediate technical skills and a few Deep Web Figure 1 - Layers of the Webresources: information customers share about the organisation and its products, information employees share about their jobs, products they are working on and company strategy/policies. More importantly, data aggregated from publicly-available databases can reveal costly, confidential information.

    In terms of resources, an initial Deep Web exploration does not imply major investment or require a team of highly skilled IT developers. Freely available tools such as DWT’s Biznar represent an excellent starting point to explore a variety of authoritative business databases for a real-time search. Other subject-specific publicly available search portals include Mednar for medical researchers or for scientific information. This kind of exploration can be learnt and done in-house with minimum resources and can save your company many hours of online searching using traditional search engines. For on-demand searches, constant monitoring of specific databases and alerts, commercial applications such as those powered by Explorit Everywhere! can facilitate the use of a targeted Deep Web search strategy, advise on the content that needs monitoring and provide a unified access point to all the necessary data sources.

    Going back to the types of data that might be made visible through Deep Web resources without its owner being aware, currently, intellectual property on the Deep Web is a matter under scrutiny. While traditional search engines might only take into account the big picture, trying to match your search terms in the title, abstract and key words; Deep Web tools can perform fully comprehensive searches. Apart from monitoring your own patents, inventions and discoveries online, this could save your company money by preventing you from becoming a litigation target after mistakenly infringing on other company’s intellectual property rights.

    The ubiquitous availability of social media applications and people’s urge to share data have led to extensive concerns in terms of how much data about your company are your employees and customers disclosing. Social media enables the creation of enormous amounts of data which is not easily to search and interpret. Most of this data is stored in dynamical databases which are not indexed by traditional Surface Web search engines. This means that they are part of the Deep Web and sometimes only protected by the individual’s privacy settings. With the right Deep Web tools, anyone can monitor the details that customers share about the products, purchasing experience and the customer’s general attitude towards the organisation. More than monitoring various data sources in isolation, aggregating them can reveal new information or give a renewed meaning to the existing (most of the time, publicly-available) one.  Cautiousness is advised when aggregating data that has been collected by another organisation as its processing might breach data protection regulation.

    On the negative side of things, sheltered by a fake username and encouraged by a number of followers, anyone can express an opinion about the organisation on social media which is going to demand a sum of resources to trace, challenge or prove wrong.  More dangerously, the ease of creating and sharing content challenges the employees’ obligation to comply with the company’s non-disclosure policies, making social media sites an ideal source of data about company difficulties, new products or future strategy. Constant monitoring and awareness of these breaches can help the company reinforce policies and put in place contingency plans in order to contain the damages.

    Even if you feel that traditional online research tools provide you with all the data necessary to your business activities, Deep Web datasources can no longer be ignored. The Deep Web, and its renowned subset, the Dark Web, is significantly larger compared to the Surface Web and due to its vastness, its content cannot always be monitored or regulated. Being aware of its existence and acquiring technology to monitor your presence and the data about you on it, or to monitor your competitors, might prove beneficial in a market where competitive intelligence is a critical component to success.

    Summary based on ‘The business potential of the Deep Web for SMEs’ published as a final year undergraduate project for the University of Portsmouth, UK


  • DWT Trip to the UK – Resource Meeting Follow-up

    NHS-presWe’re happy to report that Abe Lederman’s trip to Coventry, UK for the first ever Resource Discovery Tools for Health Libraries meeting last week went well! Abe demonstrated the Explorit Everywhere! Medical solution running at Queen Elizabeth Hospital in Birmingham to many lovely people, and listened to innovative librarians sharing their needs, problems and experiences. Take a look at his presentation online.

    Abe emphasized some of the advantages of Federated Search over Discovery Services, particularly being able to search all of the sources that are important to the user. This resonated with the librarians Abe spoke with.

    Finally, Nick Dimant, the Managing Director of our new partner in the UK, PTFS-Europe, was introducing us to one of their customers and described us as having “breathed new life” into federated search. We like it!

    If you are interested in the NHS product offering by DWT, please open the Explorit Everywhere! Medical flier for more information.

  • INIS Sees Impressive Searches Through

    The International Nuclear Information System, INIS, operated by the International Atomic EnergyAgency (IAEA) in Vienna, Austria, INIS cooperation with the   IAEA INIShouses the INIS Collection which offers access to over 3.6 million bibliographic records and over 350,000 full-text documents on topics such as Radioactive Waste Management, Nuclear Safety and the Fukushima and Chernobyl accidents. This set of non-conventional literature is available directly through the INIS online search, and is also included in the federated search portal.  

    The portal, governed by the WorldWideScience Alliance, is a global gateway to scientific databases and portals. The portal accelerates information discovery through a one-stop search of databases around the world, including the INIS Collection.

    Cooperation between the International Nuclear Information System (INIS) and WWS dates back to June 2009 when the first contacts were made during the Summer Public Conference, Managing Data for Science, organized by the International Council for Scientific and Technical Information (ICSTI) in Ottawa, Canada. Following the conference, a WWS Alliance meeting was organized where inclusion of the INIS database as a new information resource into the WWS was discussed.

    In 2012, was the top referrer to the INIS Collection, with search numbers predicted to grow.

    Current statistics of searches coming to the INIS Collection Search from WWS are impressive. There were almost 70 000 unique searches during the first 7 months of 2015 and this number is constantly increasing. The first 7 months of 2015 alone generated the same amount of searches as the whole of 2014. By the end of 2015, the number of searches coming from WWS to INIS is expected to reach 100 000.

    Deep Web Technologies proudly provides the federated search technology that powers the portal. In June, DWT upgraded the portal to include a mobile, responsive design and localization of page text. Other small improvements ensure that WorldWideScience will continue to advance information discovery for its researchers and Alliance members.

  • DWT Partners with Microsoft Translator

    We are now a Microsoft Translator Alliance Partner. DWT was invited to join the brand new WWS-FlowMicrosoft Translator Partner program which is designed to create opportunities for businesses to deliver best-in-class translation solutions.

    DWT uses the Microsoft Translator API to create multilingual search solutions for global customers. These solutions tap into the available Microsoft Translator languages and automatic translation capability so researchers can query multilingual databases in their own language, and then translate results from the sources back into their query language. Behind the scenes, DWT and Microsoft Translator shift the query from the user’s language into the language of the information sources, search the sources for the most relevant results, retrieve, aggregate and rank the results, and then translate the results back into the user’s selected language. It’s a seamless, powerful process that melts walls between researchers of different languages by allowing information discovery and flow without the need for personal translators.

    This unique, multilingual search feature originally launched with in 2010. is a global science gateway comprised of national and international scientific databases and portals. With the DWT-Microsoft Translator partnership, users can search in ten different languages and translate results from sources in multiple languages back into their selected search language.

    In early 2015, Microsoft sponsored a “Customer Story” page entitled WorldWideScience Alliance and Deep Web Technologies, revealing the history, technology and benefits of the multilingual solution. In April, DWT followed the customer story with a guest article in Multilingual Magazine, expanding on the nature of multilingual applications, the development and benefits of multilingual searching for government agencies and global companies.

    Deep Web Technologies’ is the technology behind other notable public multilingual search portals such as and UNECA’s Each of these custom portals uses DWT’s Explorit Everywhere! federated search technology and the Microsoft Translator API to search and translate.

    DWT’s partnership with Microsoft Translator is vital to the steady pulse of our multilingual applications. We believe that each multilingual application we build furthers the cross-pollination of global ideas and scientific advancement. Share our vision: In the not-too-distant-future, language barriers for science researchers and global business searchers no longer exists, and informed research will access critical information in multilingual databases with one search.

  • Biznar – A Deep Web Google Alternative

    Marcy Phelps recently posted her “Favorite Alternatives to Google” on her blog, Phelps Research. Her list of four top Google alternatives included DuckDuckGo, TinEye, WolframAlpha and Biznar.

    Biznar – When I need to cut through the clutter and focus on high-quality sources, I head to this deep web business research portal. Right now they draw results from about 100 authoritative sources, and they’re in the process of expanding that list.

    Of course, we were pleasantly surprised to hear about this since Biznar is one of four free public portals created and maintained by DWT.

    In a separate post on the Somerville Public Library Blog, “Searching the Deep Web“, the post author, Kevin, relays his surprising results from a broad Biznar search:

    Another database described in this article was BizNar, a business research database created by Deep Web Technologies.  I did a search for “cookies” (just because) and got hits for (among many other things) dessert blogs, a CNN report on the black market for cookies in Hong Kong (yes, really), a pay-to-view analysis of the world market for cookies, and a bid from the Bureau of Prisons stating that the penitentiary in Marion, IL is looking to purchase 10,000 cookies, 25,532 bagels and/or muffins and/or cinnamon rolls, and 64,440 shelf-stable flour tortillas. There were also (of course) lots of articles on the virtual cookies that marketing companies leave on your computer.

    DWT is in the process of adding and updating sources in Biznar. If you have suggestions on how to make the portal more useful, new sources to add, or category changes, please let us know!

  • Accenture Predicts One-Stop Shopping for Procurement Offices

    Spend Matters July 29, 2015 post, Accenture and the Future of Procurement Technology: The Virtual Company Mall, discusses one of five shopping-cart-152462_640focus areas for advancement in e-procurement technologies mentioned in the Accenture Report: Procurement’s Next Frontier – The Future Will Give Rise to an Organization of One. According to Accenture, the five focus areas are the Virtual category room, the Virtual supplier room, the Supplier network, the Virtual company mall, and Supply analytics, and they are required for future procurement success. These solutions will unite: “cross-functional teams, internal stakeholders, business users, other buying companies, logistics providers, unchartered suppliers, transactional suppliers, strategic suppliers and innovators.”

    Accenture describes the Virtual Company Mall like this:

    Owned and managed by procurement, the Company Mall will feature a cloud-based set of pre-approved private and public ‘shops’ (i.e., including content from outside the company) ?from which internal customers can select goods and services, supported with business logic that guides their purchasing based on policies, preferred suppliers and contracts. The mall will include a robust service desk that directs customers to the right shops and provides spot-buy services as required, as well as virtual agents delivering consistent and automated buying support. These services will be enabled by a mix of digital disruptors, including cognitive systems through intelligence augmentation and, where possible, intelligent automation.

    What will it take to make the Virtual Company Mall a reality?  You guessed it: Federated Search.  Spend Matters states: “It will require a strong e-procurement capability inclusive of powerful federated search, including internal catalog, supplier hosted catalogs, traditional web storefronts and dynamic storefronts.”

    The existing Explorit Everywhere! federated search capabilities could easily be applied to future procurement technologies and the Virtual Company Mall vision. Integrating federated search technologies like Explorit Everywhere! will smooth the transition from current procurement practices to the single search, Virtual Mall. Consider:

    1. Searching disparate sources, such as internal catalogs, supplier catalogs, and multiple storefronts in real-time for up-to-date prices, descriptions and codes.
    2. Aggregating results for top relevant products and services to maximize efficiency.
    3. Flagging duplicate products and services for easy vendor/product comparison.
    4. Narrowing results to small selections to save to the supply chain or to purchase products or services outright.
    5. Retrieving information about products or services from obscure companies or little known vendors ensuring comprehensive source research, comparison and vendor discovery.
    6. Integrating best practices for search and retrieval of products, as well as e-commerce technologies, thereby creating a seamless experience for the procurement team.

    Federated search is Deep Web Technologies’ specialty. We are the experts in single search technology. Aggressive adopters of new procurement technologies should contact Deep Web Technologies to discuss partnership opportunities.

  • Keep It Simple, Searcher

    Explorit Everywhere! supports all kinds of fancy search logic, such as Boolean operators, nested parenthesis and advanced search fields. Keep_it_Simple_(3340381990)This is a big selling point for organizations with researchers who want to search for a precise bit of information, retrieve a small result set, and then narrow that set of results even further with filters and sorts.

    Serious researchers know where their information might be hiding, how to search for their information, and what search string may bump their information out of the source and into their lap. Using Explorit Everywhere! can save these researchers time and effort; one search across all of the resources they need to examine takes just a quick click of the search button.

    Researchers familiar with a wide range of resources also know that some database search engines are stuck in the stone age. These engines simply do not process advanced logic such as parenthesis, wildcards, many advanced search fields or even Boolean operators. When researchers search these directly, they must use basic search strings to even retrieve results at all. For these engines, searching with broad queries, and iteratively searching and reviewing results is just part of the package. In contrast, a few modern search engines can handle extremely complex queries, replete with parenthesis, quotations, and Booleans and wildcards, handing the user a golden platter of relevant results.

    Explorit Everywhere! usually includes both types of sources, with search engines supporting the very simple to the most complex queries. When a user submits a search string, Explorit Everywhere! must first evaluate the query. The resource connector, a bit of code that submits the query string from Explorit Everywhere! to the source, is programmed to “know” the source parameters and limitations. The connector acts as a proxy for the researcher by submitting the query to the resource, then retrieving the results for Explorit Everywhere! to rank against results from other sources. In many cases, the query string is submitted to the source exactly as it was entered by the user. In other cases, however, the query must be reshaped to make the string more acceptable to source idiosyncrasies. And, although Explorit Everywhere! connectors are very good at understanding how a source work and optimizing the query to submit to the source, there is only so much they can do when faced with a complex query and a Neanderthal source. It’s like trying to fit a square peg into a round hole.

    To ensure consistent results retrieval, complex queries may not be the best place to start a search. If a complex search through Explorit Everywhere! yields results from only a few sources, researchers should consider rephrasing a query into simpler terms. To ferret information out of selected sources, including the more primitive sources, start with a simpler search to retrieve a broad results set from most or all of the selected resources.

    Once the results are retrieved, researchers can view the whole playing field – all the results, across all of their resources. Starting from this vantage point, the informed researcher can narrow that field with laser precision, using filters, sorts, tabs and clustering, or iterate their search based on their initial findings.

    Of course, users can still search with complex Boolean strings, taping together a montage of brackets and parenthesis with a patchwork of wildcards and quotations. Explorit Everywhere! will dutifully perform a search and no doubt return results, often with great success. For those elusive results, consider broadening the search query, and narrowing the results set after all sources have returned their results successfully. While this process may seem backward and less efficient initially, it ultimately delves deeper into those entrenched databases containing pertinent information.

  • DWT Heading to the UK in September

    Deep Web Technologies’ fearless leader, Abe Lederman, will travel from New Mexico, USA to Coventry, coventryUK to attend the first ever Resource Discovery Tools for Health Libraries meeting on September 11. The growing use of discovery tools in the healthcare sector prompts discussion about suitable technologies and the nature of search for health librarians.

    The event is hosted by the University Hospitals of Coventry and Warwickshire NHS Trust.  Founded in 1948, the NHS includes four individual systems:

    The NHS aims to provide a wide range of free health services in response to the needs and requirements of the population.

    This event is free to all health librarians. DWT, as one of the event sponsors, will have an opportunity to present how Explorit Everywhere! applications further search and discovery in the healthcare industry.

    University Hospital, Coventry, United Kingdom
    Friday, 11 September 2015 from 10.00 to 16:00 (BST)
    More Information…

    If you plan on attending the conference and would like to meet with Abe, please let us know as soon as possible. His schedule is quickly filling up!

Page 1 of 812345»...Last »