This story appears in the January 21, 2013 issue of Forbes.
JUST HOW RELIABLE ARE THOSE SURVEYS? Many doctors question their validity, starting with the sample size. Given that physicians are often judged on a handful of survey responses, though they see hundreds of patients, it seems crazy to tie their compensation to misleading results. Press Ganey admits that survey sample sizes sometimes are too small and says a minimum of 30 responses for an ER is necessary to draw meaningful conclusions from its data. But William Sullivan, the Illinois ER doctor, says that Press Ganey reports monthly results to his hospital even when there are as few as eight to ten surveys. His department has ranked in the first percentile one month and in the 99th percentile two months later.
“Response rates have been dramatically declining over the past decade,” says Paul Alexander Clark, founder of SmartPatient, a health care analytics company. He should know: Until 2007 Clark was in charge of Press Ganey’s patient-satisfaction improvement group. The response rates, he says, are now “too low to produce reliable results.” Insiders have known this for a decade. “This is a dirty little secret in our industry,” a senior Gallup executive wrote in a 2002 letter to the CMS chief. “At those levels the standard rules of probability don’t exist. … This means you may or may not be tracking real patient attitudes.” CMS declined several requests to comment on the record. Press Ganey says response rates are high enough to provide “scientifically valid results.”
Why not simply raise the response rate or increase sample size? Not so easy for an industry that largely still relies on inefficient mail and phone surveys. “It’s a very expensive proposition to mail surveys, and it’s a very labor-intensive proposition to have call centers to call people,” explains Clark. And regardless, physicians complain that patients who do respond are a self-selecting group, either extremely happy with or furious at their doctors. But Press Ganey’s Ryan says there’s no proof that “only the angry people respond.”
Many complicating factors, say Clark and others, further taint survey results, including geographical, cultural and racial differences among patients. Community-based hospitals in the Southeast generally rate far higher than large academic hospitals in the northern part of the country. One Cleveland Clinic study evaluating survey bias found that no U.S. hospital with 500-plus beds has scored in the top tenth percentile when it comes to basic communication by doctors and nurses.
What’s the right way to embrace a patient’s point of view? Experts like Josh Fenton, an author of the UC Davis study, advocate aligning incentives with evidence-based care, but that’s easier said than measured. He also suggests encouraging doctors to modify patient expectations. “Handing them a Z-Pac for a cold may make a patient very satisfied, but it costs our health care system $100 and puts the patient at risk of side effects.” Far better to explain why that’s not in a patient’s best interests. Again, easier said than done (especially for time-pressed physicians): You can tell people to eat their vegetables all you want; they’re still going to remember more fondly the person who gave them a slice of cake.