A recent study from JAMA Internal Medicine, “Meaningful Use and Quality of Care,” has cast some doubt on the $25.8 billion that the American Recovery and Reinvestment Act has apportioned for electronic health record (EHR) incentives.

The cross-sectional study of 858 physicians—63% of them defined as meaningful users of EHRs—found that “despite hope that achieving meaningful use improves quality…meaningful users did not consistently provide higher quality care,” researchers wrote.

The impact of meaningful use, which refers to three stages of criteria providers must achieve in order to qualify for federal incentives, was examined through seven quality measures among five chronic diseases. The ability to demonstrate meaningful use of EHRs “was associated with marginally better quality for two measures, worse quality for two measures, and not associated with better or worse quality for three measures,” researchers wrote.

They noted that EHRs seemed to work better for diseases with measurable characteristics, such as cholesterol and hypertension, rather than for other conditions which may be harder to evaluate, like depression.

The study took place at the end of 2012 in Boston’s Brigham and Women’s Hospital. At the time, the hospital used its own in-house system and followed stage 1 of the meaningful use requirements (data capture and sharing).

Stage 1 of meaningful use requires that 13 core objectives are fulfilled and five out of 10 electives or “menu set objectives,” are chosen and completed. Stage 2 requirements, which rolled out this past January, include that doctors provide 10% of their patients with education material specific to their condition.

For those ready to dismiss any link between meaningful use and quality care, however, the findings have some limitations. In an invited commentary on the study, Lisa Kern, MD, and Rainu Kaushal, MD, called aspects of the study’s methodology into question: “It is notable that this study was conducted at time when Brigham had an [in-house] EHR that had been iteratively defined for decades, but that since then Brigham has been transitioning to a commercially available EHR.”

Drs. Kern and Kaushal went on to explain that the study’s timing may have also skewed results: “It is not clear how long these physicians had been using EHRs prior to the study period…Studies on applications for e-prescribing have shown that, even 2 years after implementation, physicians are still on a learning curve, improving their care.

“Thus, studies conducted too soon after implementation may not find an effect even if one exists,” they added.

The commentators went on to explain that EHR value is not only limited to qualitative measuring: “Quality is also only one relevant outcome; measuring the value of healthcare, which incorporates both quality and cost is extremely important.”

Both the study’s authors and the invited commentators did find common ground in one conclusion—that more research is necessary: “Ongoing evaluation is critical to understanding the effects of the transformative MU [meaningful use] program, not only on patients but also on the healthcare system,” the commentators wrote.

For their part, the researchers themselves concluded, “The fact that we found no consistent benefit in quality for stage 1 supports the implementation of more stringent criteria in meaningful use stages 2 and 3.”

They added, “Throughout implementation, meaningful use should be monitored to ensure the large investment in effort, time and money translates into improved quality for patients.”

Whether EHRs correlate with improved care may be up for debate, but for many physicians EHR adoption is no longer an option. Medicare eligible professionals who do not adopt or successfully demonstrate meaningful use of an EHR program by 2015 will face Medicare compensation penalties.