A Conversation with Alan Weil on Decision Making in a World of Uncertainty

States are counting on investments in the medical home model to help control costs. Those savings might very well prove elusive.  At least that’s the implication of recent research from Mathematica Policy Research and the Agency for Healthcare Research and Quality (AHRQ), which found a very thin evidence base for the claim that medical homes save money. According to the research, several commonly cited claims of savings—including those from medical home projects in Illinois and North Carolina—may be unreliable.
The State Refor(u)m community has already been talking about what this means. The conversation prompts the question: what sort of evidence do state health policymakers need in order to move prudently forward—whether with medical home projects or with others—in a world of uncertainty?  I spoke with NASHP Executive Director Alan Weil—a former state health official himself—to discuss the issue.

As a state health official, what sort of information do you want before investing in a new delivery system?
It’s always best if you can look at another state that has done something similar with success.  That’s a perfect world.  But the problem is that it doesn’t let anyone lead.  No one would ever go first if everyone waited until they could go second or third.  Sometimes there are issues that you have to move on, and there’s no one else to look to.  And then you have to make a decision based on what you do know.
In general, what is the “gold standard” for evidence?
The randomized control trial, as used in clinical tests, is often held up as the standard.  But it’s quite imperfect.  When you do a clinical trial, you have a very sterile environment.  You’re testing the intervention on a certain type of person with no complexities, no complicating factors.  It’s problematic in biomedical research (think Vioxx), and it’s also problematic in health policy.  You can’t have three or four or five states that are “clean,” where you drop in this intervention with no confounding factors.  It’s always messy.  Clinical trials purport to a cleanliness that may not exist in human beings and certainly doesn’t exist in public policy.
Do you think we have a tendency to overemphasize quantitative analyses at the expense of qualitative work?
Quantitative data is great because once you run your statistical tests and finish your tabulations, you have a chart that shows “this is different than that.”  There’s a definitiveness that is very compelling, but it’s easy to overstate the implications.  For example, take the 90 or 95 percent confidence interval.  There’s no magic to those standards.  And we tend to focus on sampling error, which is easy to quantify, rather than measurement error, which may be much larger but harder or impossible to quantify.
I’m reminded of an op-ed in the Wall Street Journal by Scott Gottlieb basically claiming that Medicaid kills people because the death rate for people in hospitals with Medicaid coverage is higher than for those who are uninsured.  And in response Austin Frakt and colleagues asked, can you imagine a behavioral model that yields this result?  If there’s no behavioral model this is just a statistical artifact.  Again, we have to remember that quantitative results are easily misused.
So what are the implications?
Let’s say you have quantitative and qualitative data to show that the care patients receive is inadequate and inefficient.  And you have people who are intervening and putting in place different processes directly targeting those weaknesses in the existing delivery system. If there’s face validity, these approaches should not be dismissed just because you don’t get a statistically significant change in a particular metric.  At the end of the day, quantitative analysis can’t be the only way we decide what we should or shouldn’t do.
Of course you need quantitative data if you’re asking, “Does this save money?” But care must be taken not to lose nuance.  We now have Consumer Assessment of Healthcare Providers and Systems (CAHPS) surveys, so we’ve started treating patient satisfaction as a number.  I suspect there’s a lot more you can learn about patient satisfaction through qualitative methods.
Should every intervention necessarily be evaluated?
I don’t think so.  Let’s say a state is trying to pilot something that’s been tried successfully in another state, and they’re sticking closely to the tested model—I don’t think it’s critical to replicate the evaluation.
A big part of doing pilots is working out the operational details. Pilots allow you to refine the model.  I think there’s huge value in a pilot even without a formal evaluation process.
What lessons for policymakers do you take from the Mathematica/AHRQ reports, especially around continued state investment in the medical home model?
Our enthusiasm always gets out in front of the data. I don’t think the analysis of the evaluations to date should quash our enthusiasm.  The research synthesis is a reminder that we need to be careful about making promises.  It’s also a reminder that much of policymaking and the search for better care involves exploration, intuition, and trial-and-error.  We need formal evaluations to keep us on track, but we also need leaders to build the road ahead of us.
Do you agree with Alan?  Disagree?  Share your reactions in a comment below.