Deep Web Tech Blog

  • The End of an Era Comes to OSTI

    Dr. Walter Warnick, Director of the U.S. Department of Energy Office of Scientific and Technical Information (OSTI), is retiring in early January 2014 after 17 years as Director. Dr. Warnick embraced the idea of knowledge dissemination, and championed aggressive efforts to capitalize on technological advances such as federated search.   Abe Lederman, DWT’s President and CTO is attending a retirement celebration today at OSTI in Oak Ridge, Tennessee. Abe’s remarks on his years working with Dr. Warnick are below.

    I recall vividly my first meeting with Dr. Warnick, Walt to me, in his office in Germantown in 1999. A few months back we had launched our first Explorit application at OSTI, the Environmental Science Network and were working on our second OSTI and other OSTI portals, Walt was always focused on making more and more content available to as he called them, the “science attentive citizen” and helping to accelerate scientific progress. Walt has been a visionary and a very uncommon Federal Government executive. He always challenged his staff, was always willing to take risks and was never patient with things that took too long to accomplish.

    The science portal that I am most proud of having developed is, a portal that now searches over 100 of the best science portals from all over the world, including databases in multiple languages. WorldWideScience got its start as a collaboration between the DOE Office of Science under Ray Orbach and the British Library now under Madame Lynn Brindley. I recall fondly attending a number of major events for in London, Seoul and Helsinki. Walt managed to establish a close relationship with Tony Hey, a senior Vice President at Microsoft who has become a strong supporter of OSTI.

    Walt has promoted WorldWideScience globally. At the highest levels of the State Department WorldWideScience is used as an example of scientific collaboration between the U.S. and the Chinese and Russian governments. In a presentation to the U.N. in Geneva in 2011 Walt spoke about how WorldWideScience had become an important resource to help bridge the digital divide between countries that have access to the latest scientific information and those that don’t.

    Three years ago I wrote a post for my company’s blog, Reminiscing on a 12 year partnership with OSTI, inspired by a wonderful article that Walt had written – Federated Search as a Transformational Technology Enabling Knowledge Discovery: The Role of In that blog post I had promised to highlight a few of the accomplishments of our partnership with OSTI in my next blog post. Although a bit late I now have the content for that follow-on blog article and promise to post this talk to our blog tomorrow.

    Walt, best of luck with whatever you do next. I’m sure that you will continue to promote the causes that are dear to you exemplified in the OSTI Corollary – Accelerating the sharing of scientific knowledge accelerates scientific discovery.


  • DWT Search Portal Instead of a Book? Why Not?

    While many schools of thought might consider this notion preposterous verging on sacrilegious, David Wojick suggests that (WWS) just happens to be THAT good.  In his interview on Scholarly Kitchen’s Chefs’ Selections:  The Best Books Read During 2013 (Part 2) he states:

    WWS integrates almost a hundred other science portals, most of which are national in scope, while some are international. The chief US contribution is, itself a federated portal which integrates numerous US federal agency research report portals. Both WWS and were built and are operated by the Department of Energy’s Office of Scientific Information (OSTI). Deep Web Technologies is the amazing developer.

    With a multilingual translation feature for results, WWS has captured scientists from across the globe. While Kindle may not support WorldWideScience for a bedtime story, the WWS mobile app can rock you softly to sleep in 10 different languages.  And no need for a bookmark.


  • There’s another chink in the Discovery Services’ armor

    Yesterday afternoon as I was going through some of the hundreds of emails messages that I get every day I came across the following Thomson Reuters Pulling Web of Science from Discovery Services.

    The title says it all! Thomson Reuters has decided that it is pulling Web of Science from the 3 major Discovery Services (Summon, EDS and Primo) as early as the end of this year.

    The article concludes with the following cautionary note:

    This could foreshadow a troubling development for all discovery service vendors if others follow.

    The Thomson Reuters announcement highlights what I have been warning readers of our blog, conference attendees, customers and prospects for years now. Discovery Services are only as good as the relationships that they establish with publishers and these relationships are fragile.

    Combine this Thomson Reuters announcement with an earlier announcement from EBSCO that it is no longer allowing its content to be indexed in Primo with the fact that EBSCO and ProQuest are bitter competitors who don’t allow their content to be indexed in each other’s Discovery Service and one starts to see life threatening chinks in the Discovery Services armor.

    Stay tuned for further developments and posts on this.

    UPDATE:  It seems that Thomson Reuters has changed their mind on discontinuing access.  InfoDOCKET has posted a letter from Thomson Reuters regarding Discovery Service Access to Web of Science that mentions their agreement with Ex Libris and ProQuest.  We’ll post additional news as we hear it.

  • Alternatives to Google in Science – DWT the MultiDisciplinary Winner!

    Every once in a while, a blog will post a list of what they view as the best science search engines.  DWT is always pleased when we see these, one such list was published on Le blog de, a French blog devoted to revealing search tools in the “visible Web, the invisible web, the social web and real-time web”. The author, Beatrice Foenix-Riou, enumerated 5 categories in her post:

    1. Search engines for scientific literature (Multidisciplinary / multi-use)
    2. Multidisciplinary scientific portals
    3. Engines for multidisciplinary research / open access resources
    4. Research tools dedicated to the field of Health
    5. Search Engine Facts

    (Note that text was translated from French into English so these may not be exact.)

    Deep Web Technologies is happy to see that out of the five categories, we have built 5 of the search engines included and hold the entire Multidisciplinary scientific portal category!  Our search engines are:

    So a big thank you to Beatrice for her dedication to finding robust search tools covering such a broad scope of Science-related material.  We are thrilled to see our search engines listed there!

  • The Deep Web isn’t all drugs, porn, and murder

    Time Magazine’s current issue (November 11, 2013) cover story, “The Secret Web: Where Drugs, Porn and Murder Live Online,” reveals the dark side of the Deep Web, where criminals can hide from surveillance efforts to commit nefarious deeds anonymously. The buzz about the evils of the Dark Web (as Time’s Secret Web is commonly referred to) started early last month when Ross Ulbright was arrested in San Francisco “on charges of alleged murder for hire and narcotics trafficking violation” and identified as the founder and chief operator of Silk Road. Ulbright, known as “Dread Pirate Roberts”, is accused of running what is described in Wikipedia as an underground website sometimes called the “ of illegal drugs” or the “eBay for drugs.” And, of course, the government shut down Silk Road.

    As founder and president of Deep Web Technologies, I take exception to the article’s referral of the dark regions of the Web broadly as the Deep Web. The term Deep Web, first coined in 2000, refers to huge areas of the Internet that serve legitimate organizations and the public. Not all of the Deep Web is dark. In fact, most of it isn’t. In fairness to the Time Magazine article authors, Grossman and Newton-Small, they do make this point early on:

    Technically the Deep Web refers to the collection of all the websites and databases that search engines like Google don’t or can’t index, which in terms of the sheer volume of information is many times larger than the Web as we know it.

    Understanding Deep Web Technologies that gives a hint as to what treasures lie in the Deep Web.

    The deep web is everywhere, and it has much more content than the surface web. Online TV guides, price comparison web-sites, services to find out of print books, those driving direction sites, services that track the value of your stocks and report news about companies within your holdings – these are just a few examples of valuable services built around searching deep web content.

    But, not only is the Deep Web of interest to consumers, it’s of particular value to academicians, scientists, researchers, and a whole slew of business people who rely on timely access to cutting edge Deep Web content to maintain a competitive edge.

    Here’s another snippet, this one from a series, “Federated Search Finds Content that Google Can’t Reach,” emphasizing the importance of Deep Web searching to research organizations.

    Federated search facilitates research by helping users find high-quality documents in more specialized or remote corners of the Internet. Federated search applications excel at finding scientific, technical, and legal documents whether they live in free public sites or in subscription sites. This makes federated search a vital technology for students and professional researchers. For this reason, many libraries and corporate research departments provide federated search applications to their students and staff.

    Hopefully you’re convinced that there’s valuable information in the Deep Web.  Now, no one knows exactly how big the Deep Web is compared to the Surface Web that Google, Bing, and the others crawl but it’s likely that the Deep Web is hundreds of times larger. This is great when you have access to tools like Deep Web Technologies’ Explorit search engine but it might  also make you nervous wondering how you can find that needle in the haystack in a web that is hundreds of times larger than the one you’re familiar with that is overwhelming you with too much information and too much junk mixed in with the good stuff.

    If what is in the Deep Web intrigues you, try a few of our Deep Web applications to see a bit of the richness that lies beneath the surface of the Web. 


    Update:  I have also written an email to Time Magazine which I’ve copied below.  I don’t know if they will publish it or not, but I certainly hope that they will recognize that the Deep Web is much more than a haven for criminals.

    Dear Editor:

    As someone who makes his living providing access to the legitimate parts of the Deep Web I am very concerned that your article paints a dark picture of the Deep Web as a whole. The company I founded, Deep Web Technologies, Inc., searches Deep Web sources on behalf of scientists, researchers, students and business people. My concern is that the public, and my potential customers, will equate all things related to the Deep Web with dark criminal activity. Please help me to correct this potential misperception to the reality that the Deep Web is about those areas of the Web that contain high quality content and that the Dark Web is just a fringe neighborhood within the Deep Web that most of us will never venture into.


    Abe Lederman

    Deep Web Technologies, Founder and President


  • Welcome to the Jungle

    Deep Web Technologies, based in the not-so-jungle-y desert of Santa Fe, NM was recently interviewed by Garrett Robinson on The Voice of Santa Fe.

    Listen to a 2 minute clip now or to the full broadcast on the Voice’s archives. Every Friday, Garrett interviews local businesses that have figured out how to be successful in today’s economy.  Our VP of Business Development, Frank Bilotto, called in to speak with Garrett and talk about how to access the Deep Web using our technology.


  • Taking My Library Back from Google

    Today our CEO, Abe Lederman, had the pleasure of working with our partner, EOS International, to broadcast his thoughts on Google, and how libraries can bring relevance back to their patrons.  Many researchers turn to Google as their first stop because it is “quick and easy”. However, Google is not always the highest quality, or the most reliable content.Webinar

    The webinar covered several points:

    • How patrons search and why
    • Limitations of Google Scholar
    • The deficiencies of Google and Google Scholar
    • How to make you, the librarian, the search authority
    • How to bring relevancy back to the library
    • How librarians can play a key role in bringing Federated Search to their patrons

    If you attended the webinar (on the 18th or the 25th), what did you hope to learn from it?  What has been your “Google” experience and do you have any advice  to share?  We look forward to hearing your thoughts!

    Want to see the presentation again?  View it here –  Taking My Library Back from Google.

  • Reprints Desk – Outscoring Competitors in Outsell Scorecard

    A big congratulations to Reprints Desk, Deep Web Technologies’ Document Delivery Partner who received top ratings in the 2012 Outsell. Document Delivery puts crucial content in information specialists hands allowing a backup model for cancelled subscriptions or incomplete collections. In light of the growing need for Document Delivery services, Outsell conducted a short survey in September 2012.

    In the Scorecard, 3 specific categories of Document Delivery were evaluated:

    • Depth and breadth of coverage: One of the critical elements of a document delivery supplier’s service is how comprehensive and deep its reach is across subjects. This attribute measures whether or not the content available from the vendor is what the buyer requires.
    • Fair pricing: It is typical for buyers to want lower prices. We asked buyers to rate how reasonable (fair) they believe providers’ pricing to be. 
    • Ease of doing business: Sales staff turnover, company restructuring or acquisition, customer service, and processes or methods that fit the needs of enterprise buyers are factors in the ease of working with vendors.

    In all areas, Reprints Desk was the highest rated Document Delivery vendor, scoring 4.4 out of 5 in Overall Satisfaction.  “Reprints Desk has penetrated the market since it opened up in 2006, and its reputation for working closely with its customers has won it many fans.”

    We’re pleased to be partnering  with Reprints Desk, providing our joint customers with a state-of-the-search capability integrated with the best Document Delivery service on the market.   Stay tuned for some exciting new enhancements to Article Galaxy Search coming soon.

  • Explorit now features visual clustering

    Our clustering engine has always been among our customers’ favorite features. We introduced the feature and its tremendous value in discovering documents in our “Clusters That Think” article.

    One of the most interesting features of our Explorit search product is our clustering engine, which analyzes results and produces “clusters” that represent a new and powerful way to navigate search results. The true power of these clusters is often overlooked, for they superficially resemble the output generated by the keyword-based systems and fixed taxonomies of other search engines. Our clustering technology, however, is more akin to a document-discovery engine, which provides a significant improvement over the alternatives in the library world.

    Clustering is particularly powerful because, as our original article on the subject explains, “users think in concepts, not keywords.” Extending this line of thinking, we believe that many people think in pictures and not in lists and sublists of text so we introduced visual clustering to our Explorit feature set.

    Here is a screenshot of a visual cluster of results for the search term “satellite communications.”
    Mednar, and at the science portals and as well.

  • mobile app among 10 best according to GCN

    Government Computer News (GCN) recently reviewed mobile apps developed by the federal government. Mobile was among the top 10 listed.

    GCN gave the Mobile app (which runs on the Android and on the Mobile Web) scores of 7 for usefulness, 8 for ease of use, and 8 for coolness factor.

    The website has this to say about the accolade:

    Coolness? Check. Usefulness? Check. Ease of Use? Check. The Mobile application has been named among the Top Ten in Best Federal Apps by Government Computer News (GCN). The recognition is timely, too. The Administration recently issued Digital Government: Building a 21st Century Platform to Better Serve the American People, the strategy which calls on all federal agencies to begin making mobile applications to better serve the American public. GCN called its Top Ten “ahead of the curve” with apps already in place.

    I downloaded the application to my Droid X. The install was effortless and the app has a very intuitive user interface, which allows for emailing of search results for later review.

    While we didn’t have any involvement in creating the mobile app we did develop the search technology that powers as well as the web services API that enables searches by Mobile.

    We’re quite delighted to see serve the mobile web.

Page 10 of 12« First...«89101112»