New research shows disapproval of randomized controlled trials may impede ability to learn which policies and treatments work best

Geisinger and Microsoft research finds widespread aversion to randomized experiements in healthcare and beyond

News provided by

Jun 07, 2019, 16:51 ET

DANVILLE, Pa., June 7, 2019 /PRNewswire/ -- Randomized controlled trials (RCTs), commonly referred to as A/B tests, have long been the gold standard for scientific evidence in medicine and are increasingly used in other fields as diverse as education, government policy, and poverty reduction. A/B tests help to determine whether a new treatment or policy helps or hurts compared to an alternative. A new study by researchers at Geisinger and Microsoft Research published last month in the Proceedings of the National Academy of Sciences sheds light on previous controversies over A/B tests, finding that people often approve of an untested policy (A or B) being given to everybody but object to an A/B test of those same unobjectionable policies designed to determine which one works best.

The series of 16 studies with 5,873 participants from three diverse populations spanning nine domains in healthcare and beyond suggests an important obstacle to evidence-based policy and practice.

One study involved differing methods of communication aimed at educating providers on best practices for avoiding infections from catheterizations. One group of participants was told that the hospital director decided to print a checklist of safety steps for clinicians on the backs of their ID badges. Another group was told that the director instead decided to print the checklist on posters that will be hung on the wall. A final group of participants was told that the director thought of both these ideas and decided to conduct an experiment in which patients are randomly assigned to a doctor with a badge or a doctor who works in a room with a poster. They were also told that once the hospital director learns which policy is best at reducing infection rates, he or she will implement it for everyone.

Among the group who read about the director who implemented the badge policy, very few judged this decision to be inappropriate. So, too, with those who read about the director who implemented the poster policy. But many who read about the director who ran an A/B test to compare the badge and poster ideas thought this decision was inappropriate.

According to the study's lead author Michelle Meyer, PhD, JD, Assistant Professor and Associate Director, Research Ethics, in Geisinger's Center for Translational Bioethics and Health Care Policy, A/B tests necessarily treat people differently, and potentially unequally—at least temporarily. "But strikingly, we found that people still object to A/B tests even when the alternative was still that different people would randomly get different treatments," Meyer said. "Because of who they happen to have trained with, or what journal articles they happen to have read, some doctors prescribe one drug for things like high blood pressure—we called it Drug A—and others prescribe Drug B. Patients who are prescribed a blood pressure medication in the ER or in a walk-in clinic are basically being 'randomly assigned' to one drug or the other due to chance. Yet many of our participants find that unobjectionable but object to having their treatment formally assigned by an A/B test that would figure out which drug is best so it can be offered to everyone."

Follow-up studies found the same widespread disapproval of A/B tests despite approval of just giving everyone either A or B. Results were consistent in a variety of other domains, including consumer genetic testing products, autonomous vehicle design, and social policy programs. Doctors and participants with advanced degrees were no different than other participants.

In examining the reasons participants gave for objecting to the A/B tests, the researchers found that people often assume, incorrectly, that experts already know "what works." The researchers concluded that more research is needed to better understand why people object to experiments that compare unobjectionable policies and how to address those concerns.

About Geisinger
One of the nation's most innovative health services organizations, Geisinger serves more than 1.5 million patients in Pennsylvania and New Jersey. The system includes 13 hospital campuses, a nearly 600,000-member health plan, two research centers and the Geisinger Commonwealth School of Medicine. Geisinger is known for its focus on caring and innovative programs including the ProvenCare^® best-practice approach to maximize quality, safety and value; ProvenHealth Navigator^® advanced medical home; Springboard Health^® population health program to improve the health of an entire community; ProvenExperience™ to provide refunds to patients unhappy with their care experience; and Geisinger's MyCode^® Community Health Initiative, the largest healthcare system-based precision health project in the world. With more than 215,000 volunteer participants enrolled, MyCode is conducting extensive research and returning medically actionable results to participants. A physician-led organization, with approximately 32,000 employees and more than 1,800 employed physicians, Geisinger leverages an estimated $12.7 billion positive annual impact on the Pennsylvania and New Jersey economies. Repeatedly recognized nationally for integration, quality and service, Geisinger has a long-standing commitment to patient care, medical education, research and community service. For more information, visit geisinger.org or connect with us on Facebook, Instagram, LinkedIn and Twitter.