Business challenge

Quest Diagnostics captures data on 500 million clinical lab tests every day. How could it find an efficient way to process and analyze these data to provide meaningful, actionable health insights?

Transformation

By implementing IBM® BigInsights®, the company can easily move its data into a Hadoop cluster, enabling analytics at scale, and delivering new understanding of clinical outcomes for its clients.

Results

Harnesses

analytics to deliver diagnostic insights rapidly and cost-effectively

Eliminates

the majority of data quality issues

Cuts

the amount of work involved in the data staging processes significantly

Business challenge story

Understanding clinical lab results

Every day, Quest Diagnostics completes millions of clinical lab tests and related services, from routine biological tests to anatomic pathology and complex, specialized gene-based testing. Additionally, Quest Diagnostics provides services to employers and the government for workplace drug testing, wellness services, and risk assessment services for the insurance industry.

“IBM BigInsights and BigIntegrate can deliver diagnostic results easily, rapidly and cost-effectively.”

—Nilesh Saratkar, Big Data Lead, Quest Diagnostics

Results from every test are stored in Quest Diagnostics’ database, providing a statistical treasure-house of data. Quest Diagnostics analyzes the information and publishes diagnostic insights and original research, showing trends, treatments and outcomes based on 20 billion test results.

Additionally, for its own management purposes, Quest Diagnostics reviews usage: for example, which physicians and hospitals are using the service, by volume and test type? Based on these figures, Quest Diagnostics identifies patterns of use that may predict which client may be about to switch to another lab services provider, or highlight specific demand and opportunities for services that will help to grow the business.

Nilesh Saratkar, Big Data Lead at Quest Diagnostics, comments, “Quest Diagnostics analyzes huge quantities of data, and to do so cost-efficiently is a constant challenge. For example, although labs run the same tests, they collect different data-points, stored in different formats. Even with decades of experience in operating large data warehouses, we began to struggle with the variety and sheer volume of data.”

Transformation story

Looking for clear diagnosis

The Quest Diagnostics team wanted to capitalize on its existing skillsets and capabilities, while building out a scalable solution that would enable it to move to the next level of clinical analysis. Potentially this could involve heavy investment in data warehousing solutions, which Quest Diagnostics was keen to avoid.

Nilesh Saratkar explains, “We have a 15-year background in traditional data warehousing experience, and deep knowledge of the tools required for a big data environment. We soon realized that IBM’s big data analytics offerings could solve our technical and performance challenges, and enable new capabilities for our diagnostics services.”

Quest Diagnostics decided to extend its diagnostics analysis services by adopting some of the capabilities of IBM BigInsights. BigInsights supplements a standardized, Open Data Platform-compliant Hadoop cluster with a user-friendly data management tool that helps to extract, transform and load [ETL] the diverse clinical lab data-sets. It also provides an analysis tool that helps Quest Diagnostics gain insight from its data easily and efficiently.

“IBM is particularly good at providing services we need: grid computing, robust tools, and analysis of very large data-sets,” remarks Nilesh Saratkar. “By using these tools, which are available with IBM BigInsights, our data scientists and data engineers can spend less time writing complex Hadoop MapReduce jobs, which saves a lot of development and analysis time. To put it in another way: why write code manually in a complex language like Java, when you can use the IBM tools with minimum up-front investment and no additional training cost?

“The IBM BigInsights platform also gives us access to a host of new capabilities, including BigIntegrate for ETL processing. We know that around 70 percent of the time is spent on resolving basic data quality and data integration challenges, so building data quality and data governance into the solution gives us a huge advantage—not just operationally, but also to help us ensure compliance with HIPAA regulations and other data privacy standards on the use of confidential patient health information. As early adopters, we are working closely with IBM to help them develop and refine the BigIntegrate solution.”

Because the company’s data scientists often need to explore new data-sets totaling several terabytes of data, Quest Diagnostics can also make use of IBM BigSheets to provide a spreadsheet-like view of data while it resides on the Hadoop cluster.

“With BigSheets, our data scientists can understand the structure of a new data-set, follow the relationships between data quickly, and see the lineage without any help from the IT team or from developers,” says Nilesh Saratkar. “Due to the size of the data-sets, this would be impossible with traditional desktop-based spreadsheet tools.”

Results story

Planning healthy outcomes

Quest Diagnostics’ journey into big data analytics is ultimately designed to help physicians gain new evidence-based insight into treatments and results, improving patient care. By taking real-world inputs from multiple sources in varying formats, and using BigInsights to build a structured data-flow and harness powerful analytics, Quest Diagnostics is finding new ways to deliver results.

“We estimate that the integrated capabilities of BigIntegrate could perhaps take away the majority of the data integration work required to manage and match the data inputs that produce the ‘golden records’ we need for our diagnostic analyses,” says Nilesh Saratkar.

Alongside the medical diagnostics, Quest Diagnostics also explores general business analytics, using the same powerful tools to examine its engagement with physicians and hospitals. For example, by looking at the behavior of its clients, Quest Diagnostics can predict when a group of hospitals may be about to switch to another lab services provider, and take action to retain the customer relationship. Similarly, by trawling the data to identify changing levels of demand for different types of tests, Quest Diagnostics can identify new opportunities that help to grow the business.

Nilesh Saratkar concludes: “We are all about creating a scalable big data platform, using Hadoop as a data lake, completing analyses and pushing insight back to the business. IBM best practices for big data technologies are vital to Quest Diagnostics’ continuing progress, and we know that with IBM we have a partner that listens and provides answers quickly. Additionally, IBM has deep resource and support capabilities, a significant advantage to Quest Diagnostics as we continue to grow.”

“IBM BigInsights and BigIntegrate offer robust, reliable big data analytics components on a fully integrated platform, helping Quest Diagnostics to deliver diagnostic results easily, rapidly and cost-effectively, while meeting relevant standards for patient privacy and complying with HIPAA regulations. Any time we have new use cases or business opportunities, we reach out to IBM, look at our existing technology stack, and consider whether we have the components we need. We communicate our ideas and IBM comes up with possible solutions, and that’s how we grow together.”

10981561

About Quest Diagnostics

Based in Madison, N.J., Quest Diagnostics provides lab and clinical analytics services for around half the physicians and hospitals in the U.S., touching the lives of 30 percent of American adults. With some 700 MDs and PhDs among a staff of 45,000, Quest Diagnostics operates more than 2,200 patient service centers and in 2014 generated revenues of around USD 7.4 billion.

Take the next step

IBM Analytics offers one of the world's deepest and broadest analytics platform, domain and industry solutions that deliver new value to businesses, governments and individuals. For more information about how IBM Analytics helps to transform industries and professions with data, visit ibm.com/analytics. Follow us on Twitter at @IBMAnalytics, on our blog at ibmbigdatahub.com and join the conversation #IBMAnalytics.

View more client stories or learn more about IBM Analytics