A mere fifth of digital health startups are clinically robust, meaning they have the data and regulatory filings to back up any public claims made, a new analysis shows.

The findings, appearing in Journal of Medical Internet Research, suggest that the ingredients for instilling clinical confidence among physicians and patients, payers and employers is distributed unevenly across the digital health startup pack.

“There clearly is a small but sizable minority of clinically focused digital health startups that are setting themselves apart by pursuing these robust measures of evidence, clinical trials and regulatory filings,” said Sean Day, one of the study’s authors.

“We would have expected more,” added Day, who is a consultant for Rock Health, which also supplied the data for the analysis. Rock Health is an investor in such companies but said no individuals involved in investments were part of the study.

In addition to mining Rock’s database of digital health venture funding, researchers also pored over Food and Drug Administration regulatory filings. The study focused on venture-backed companies with more than $2 million in funding. 

One-fifth of companies garnered a clinical robustness score of 5 or more, with that measure defined using FDA filings and clinical trials completed by each company. But the average score for all companies was a fairly low 2.5 and the median score was 1. More than 40% of companies had a tally of 0, and 15% of companies earned a tally of 1. 

Findings suggest that most of the 224 companies in the sample lacked what it takes to back up any public claims made. If they were making them, that is. The study’s other surprising finding was that not many companies are even making such claims, even among the 20% of firms with higher clinical robustness scores.

“We would have hoped to see a positive correlation between the clinical process measures of filings and trials, and the claims that companies are making. And we did not find that was the case,” said Day.

The average number of public claims — defined as a quantitative statement about outcomes of any kind – was just 1.3. 

According to Day, these results imply a spectrum. On one side, there are companies making claims that may not be backed up by regulatory filings or clinical trials. On the other is a sizable chunk that do have the clinical measures and filings but are not actually promoting themselves. That dynamic could make it difficult for startups to get matched up with the right adopters and end users.

“Buyers of these digital solutions, whether that’s a payer or an employer, are not necessarily going deep into the FDA’s database or looking through the clinical trials to make their adoption decisions,” said Day. “They’re looking at public claims.”

Along similar lines, the researchers observed no correlation between venture funding and clinical robustness. It’s simply not the case that companies with higher levels of clinical robustness have on average raised more venture funding. 

Asked whether the factors native to digital health startups are what’s behind the dip in digital health funding seen in the first quarter, Day said he didn’t think so. That dip “is caused by much broader macroeconomic changes and pullback that we’re seeing across the venture industry in general.” 

Moreover, the investment calculus is often more complicated than just having a deep evidence base. 

“There are many other factors that contribute to the success of these business models — how are they going to market? What’s their distribution plan?” he explained.

That said, the findings do seem to align reasonably well with the digital health reimbursement landscape, which relatively few startups have been able to navigate successfully.

“There are a handful of companies out there that are going through the hoops to demonstrate in a robust way that they can drive clinical outcomes,” Day noted. “So it’s unsurprising that a similarly small subset of the overall digital health companies would manage to get coverage by payers who are going to make those decisions based on the evidence.”

The digital companies in the analysis were focused on either prevention, diagnosis or treatment of disease. Given that medicine and technology are becoming increasingly intertwined, improving the low overall scores should be an area of focus for the field moving forward, Day advised.

“These findings underscore the need for a broader base of robust evidence in digital health,” he said. “If digital health is going to be eventually just embedded into healthcare — and even becomes synonymous with healthcare — then the standards for evaluating evidence in digital health need to be commensurate with the way we evaluate more conventional healthcare solutions. It’s going to be part of the evaluation criteria for all sorts of enterprise buyers.”