- Product Trial
Taylor and Francis just published an article by Brad Long, formerly of Central Michigan University (CMU): Addressing a Discovery Tool’s Shortcomings with a Supplemental Health Sciences-Specific Federated Search Engine. The article tells of CMU’s experience with Ex Libris’ Primo discovery tool, it explains what drove them to Deep Web Technologies (DWT) as the only federated search option, and shares their positive experience with us. Of special interest is the fact that CMU’s medical program is brand new. Any university with a new program like CMU’s can glean important insights from Mr. Long’s experience. And, new medical school or not, there’s a lot of meat in the Taylor and Francis article.
Here’s the article abstract:
Some discovery tools may not necessarily be the best resources for searching the health sciences literature. This is especially true when the discovery tool is not a vendor-neutral product or when it does not deep search full-text electronic resources. Therefore, supplementing a discovery tool with a health sciences-specific federated search engine could be a viable option. This is a review of one university’s decision-making and implementation process for adding a health sciences-specific supplemental federated search engine. Included is a 30-month usage analysis of the federated search engine, along with website usability testing results.
In this blog I’ll highlight the major takeaways from the CMU article.
We’ve written quite a bit about the good, the bad, and the ugly of discovery services so I won’t go into much detail here on its pros and cons. CMU noted some pluses (fast simultaneous search of resources, relevance ranking, and link resolver support) but their list of negatives was longer. The biggest shortcoming was that only nine of the university library’s key health sciences resources were available in Primo. Mr. Long noted that this deficiency is due to Primo (and discovery services in general) not being vendor neutral and also not performing real-time full-text searches.
We’ve heard this “not vendor neutral” complaint time and time again. When it comes to selecting deep web sources to include in their index, particularly medical sources, the discovery services come out lacking. We could not have developed our evidence-based medicine and chemical safety vertical search applications without the ability to select exactly the sources each of our customers wants to search. In the vertical portal market, which we believe to be the future of search, it’s the laser focus on the content most valuable to each customer, and the lack of distracting other content, that is the winning combination.
More is not better. We’re all steeped in a culture where we do that Google search and we somehow feel proud that Google found us 23 million hits for our search terms. And then (maybe) three things dawn upon us: (1) we’re only going to look at the first ten of those millions of hits, (2) we hope the relevance ranking is good or we’ll never find the other good stuff buried beyond the first page of results, (3) we have to waste time sifting through the irrelevant and misleading results and hope we find some good stuff. Mr. Long voiced CMU’s experience of the problem as “There is not an intuitive way to limit search results only to health sciences resources.”
We wrote recently about our evidence-based medical search application which is built on a foundation of “search what we need and only what we need.” Bedside clinicians need to diagnose patient conditions and prescribe the best evidence-based treatment, i.e. the one with the strongest evidence supporting the best possible outcome, and they need to do it quickly and efficiently. In collaboration with Dr. Sam Keim, Professor and Chair of the Department of Emergency Medicine at the College of Medicine Tucson, University of Arizona, we built a distraction-free evidence-based medicine (EBM) application, Explorit Everywhere! EBM. With Dr. Keim’s team taking the lead on selecting the few best medical resources for 28 specialties and organizing those resources into different layers of the evidence pyramid (categories) based on strength of evidence, we developed a tool that keeps clinicians focused. When they need strong evidence to support a treatment plan they keep their attention on the top layer of the pyramid. When they need background information they can consult the sources in the “textbooks and other resources” layer of the pyramid. And, because every single resource is hand-selected and expertly vetted there’s never a question about the quality of information.
CMU decided to address the major shortcomings of Primo with a federated search tool. It’s worth mentioning that CMU found itself in a situation we’ve seen in other universities – they buy a license for a discovery service that provides access to the entire university. So, the medical school gets that service for free. But, it’s lacking in some ways and the university needs to figure out what to do.
The biggest shortcoming of Primo for CMU was that, of the 25 connectors that CMU selected for the initial deployment of their federated search tool, only nine were available from Primo, or just slightly over 1/3. The article goes into a little bit of detail about sources that were added, dropped, or changed during its first two and a half years of existence. That discussion may be of interest to other health sciences organizations looking to understand a bit of the thinking behind one organization’s resource choices and how their online library collections evolved.
CMU did something we’ve not seen documented elsewhere; they performed an analysis of their query data and wrote up their results in the article. Here are the kinds of data they gathered, graphed, and reported on: number of queries, number of queries with at least one click-through to a resource (which Mr. Long refers to as “effective queries”), number of click-throughs per connector, cost per query, and cost per click-through. CMU concluded:
Usage statistics have demonstrated Medical Searcher is a cost-effective tool for locating medical literature. This is based on the overall cost per query and cost per effective query data.
The article concluded with a brief discussion on usability testing. Testing in that regard was performed on a very small scale and it was not specific to their federated search product so we have no comments to share on that piece of the article.
CMU was pleased with our search solution.
Through the first 30 months of implementation, end users are using Medical Searcher to retrieve both article citations and full-text resources. The implementation of Medical Searcher has successfully addressed Primo’s shortcomings with health sciences resources.
And, we’re pleased with their conclusion. Reviews of federated search tools are few and far between, especially with any level of detail. Finally, it’s worth noting that it’s not always an either-or situation. CMU was able to employ both discovery and federated search together.
If you’re pleased with your Explorit Everywhere! solution you could follow in Mr. Long’s footsteps and journal your experiences in various ways: a blog article, a presentation at a conference, or in our customer corner (something like this.) Aside from that, you can share the love, promoting Explorit Everywhere! in a number of ways.
Dr. Sam Keim is Professor and Chair of the Department of Emergency Medicine at the College of Medicine Tucson, University of Arizona. Dr Keim is a member of Deep Web Technologies’ advisory board and a strong advocate for developing and using technology to save lives.
Dr. Keim has collaborated with us to bring to fruition a technology that brings a laser focus to federated search in evidence-based medicine applications. This technology greatly streamlines the research process, helping busy clinicians to quickly make the right decision for treating their patients.
Dr. Keim was kind enough to answer some questions about his brainchild, Explorit Everywhere! EBM. You can read more about this technology in our two-pager.
Tell us about your background as a physician in emergency medicine.
I have practiced emergency medicine for thirty years. Most of this time at Banner University of Arizona Medical Center in Tucson, which is a busy level-1 trauma center. We actively train residents in emergency medicine and pediatric-emergency medicine as well as subspecialty fellows, nurses, clinical pharmacists and paramedics.
What is evidence-based medicine (EBM)? What problem does EBM solve? And, what is the evidence pyramid?
EBM is the mindful use of modern technology to acquire the best relevant medical information to the bedside to support clinical decision making. A clinical provider that utilizes EBM greatly benefits their patients because they are able to apply the best and latest proven scientific advances! The Explorit Everywhere! EBM Pyramid is our translation of well-described scientifically-proven levels of strength by which different types of information can be “graded.” Answers to clinical questions are not created equal! A clinician should strive to apply the STRONGEST evidence available for every patient. This frequently does not occur because the clinician doesn’t have a tool, like this one, to find the STRONGEST evidence available. Clinicians and patients no longer need to be frustrated by not having access to information about the best therapies and evidence.
What is behind your passion for EBM?
My passion is for optimizing patient care and lifelong learning simultaneously through 21st Century technology solutions. Patient care currently suffers because new advancements, even some that are many years old, have not been “found” by today’s clinicians. This must become unacceptable to providers, health systems and patients! The best and latest medical information must be delivered to busy clinicians without undue hardship. Explorit Everywhere! EBM is the tool that does this.
You’ve dedicated thirty years of research and teaching focused on EBM. And, you’ve co-authored several journal articles about EBM and technology. What are the major takeaways from those articles and from your experience?
The major takeaways are that 1) delivering patient care informed by the best evidence is not only better for patients but nourishes the clinician’s need for continuous lifelong learning, 2) continued research into improving search and delivery technology is critical, and 3) an effective collaboration between clinicians, software engineers, and library scientists is essential in creating tools, like Explorit Everywhere! EBM.
Can you expand on your previous answer and tell us how you teach medical students to become better EBM searchers
We have an innovative medical school curriculum at The University of Arizona College of Medicine that requires medical students to become inquisitive evidence explorers. A superb clinician must first be expert in asking relevant questions. Such as what is the most likely cause of this condition? What is the most likely high-risk alternative? What is the most effective therapy? What is the best estimate of prognosis? What are the potential harmful outcomes of the standard and alternative therapies? What does this all cost? Our medical students have been learning to use federated search for over a decade to drive their information searching for classroom and bedside clinical problems. It works extremely well for both answering the question at hand as well as driving home the differences between truth and fiction. All evidence is not created equal!
What do you say to the busy medical practitioner who doesn’t see a need for new technology. To them, DynaMed Plus or UpToDate are good enough. And, when you’re not looking, they might go to Google or Wikipedia.
Electronic textbooks are better than nothing. They’re a small step better than the old-school physicians that are still out there “winging it” with decades-old knowledge. But busy clinicians need to experience choices that are as convenient as the common electronic textbooks. Explorit Everywhere! EBM gives them those convenient choices.
Best quality evidence and efficiency are two critical goals for EBM researchers. How does Explorit Everywhere! EBM leverage federated search technology to meet these goals?
Explorit Everywhere! EBM marries state-of-the-art federated search technology with quality filters to deliver a “buffet table” of choices ranging from highest quality to lower quality. Because not all clinical questions have been rigorously answered by leading systematic review organizations, the clinician might need to use other sources. Explorit Everywhere! EBM allows this decision to be made after ONE single search execution. All of the results are there, teed up, without a need to execute another search. This delivers the best quality at a convenience not equaled by any other system.
How is Explorit Everywhere! EBM an evolution beyond prior technologies?
It is a profound leap forward because only the highest quality sources are included in the federation. The user, therefore, does not waste time reading and sorting through low impact information and misinformation. With Explorit Everywhere! EBM you are ONLY delivered the gold!
Who are the best users of Explorit Everywhere! EBM? Clinicians? Medical students? Residents? Medical researchers? Nurses?
The best users are all clinicians and anyone who needs the best quality medical information and clinical research delivered and understood now. Not an hour or two from now, but now. In a few short minutes of searching and reading. This includes all of my physician, nursing, pharmacy and veterinarian colleagues. Medical students and residents will find that Explorit Everywhere! EBM brings their performance up to top level. Best evidence always trumps “years of experience” so this is a tool that will allow young clinicians to practice medicine at the state-of-the-art!
Tell us about the collaboration between your University of Arizona Health Sciences Center and Deep Web Technologies? What roles are each party playing?
Deep Web Technologies is the engineering partner that we need for Explorit Everywhere! EBM to deliver on the dream – best evidence, anywhere, in seconds. The University of Arizona brings EBM and library sciences expertise. All three domains have complexity and are exploding in rapid growth. The dream team requires expertise in all of these domains. We have it right!
What is your vision of the perfect EBM system?
The perfect EBM system supports clinical decision making by bringing the best evidence to answer clinical questions, from novice to expert, instantaneously at the point of patient care. It will have advanced analytics and artificial intelligence to continuously learn and further support decision making. Finally, because patient care does occur everywhere, including on streets, sidewalks, jungles and extreme environments, the perfect system must be wireless, mobile and easy to use. I have no doubt that our team will be there soon!
Update: December 20, 2017. You can read more about the evidence-based pyramid and laser-focused search technology in our referisg article. And, we’ve put up a demo of EBM search at http://demo.ebm-search.com.