In a recent report, Apple outlined the “rigorous scientific validation processes” used to develop the health and fitness features baked into its products.
Wearables like Apple Watch monitor our bodies around the clock, providing health insights in real time. That’s a new and unprecedented development in medical technology. The benefits are already clear, as the report illustrates, with anecdotes about how Apple Watch has saved lives.
But anecdotal evidence is not the same as scientific research. By cherry-picking the best outcomes, anecdotes risk overlooking the bigger picture. Scientists must look at all the outcomes, not just the good ones. With that in mind, I took a closer look at the scientific studies cited in Apple’s report, to find out what they tell us about the impact Apple Watch is having on our health.
The science behind Apple’s health and fitness features
According to the report, Apple Watch and iPhone offer features covering 17 different areas of health and fitness. To keep things simple, I’ll focus on four: the Mindfulness app, Activity rings, sleep tracking and atrial fibrillation detection.
I’m no scientist, so I can’t judge the quality of the research. Instead, I focused on where the report provides evidence — and what that evidence says.
Apple’s full report, entitled “Empowering people to live a healthier day – Innovation using Apple technology to support personal health, research, and care,” is available for download in PDF format here.
Apple’s Mindfulness app
Conventional medicine or an alternative treatment?
According to the report, the Mindfulness app is “backed by widely recognized physical and mental health benefits of mindfulness intervention programs ranging from cardiovascular improvements to stress relief.”
Personally, I do think there are benefits to meditation and mindfulness. But I’ve always thought of it as something alternative and New Agey, rather than backed by mainstream science. So I was curious to see what evidence the report offered to support this claim.
The report cites research on a third-party mindfulness app, not Apple’s
Apple’s report states that “in various randomized control trials, mindfulness apps like Calm on iPhone and Apple Watch were found to lead to reduced stress and decrease in fatigue, daytime sleepiness, as well as improvements in sleep quality.” But it only cites one study, and that study only tested the Calm app.
In the study, researchers randomly assigned 88 college students to either a test group or a control group. The test group used the Calm app for at least 10 minutes a day for eight weeks. Meanwhile, the control group members went on a waiting list to use the app. All participants completed surveys before, during and after the eight-week period. Those using the app reported reduced stress and “improved mindfulness and self-compassion.”
The challenges of researching mindfulness
There are two challenges when conducting this kind of research. Firstly, it’s not possible to do it “blind.” The participants obviously know if they’re using the app or not. Also, the participants self-report the results, meaning we can only go on what they tell us.
That, as the study’s authors note, can lead to “response bias where participants respond in what they believe is a more favorable manner.” In other words, the participants might just be saying what they think the researchers want to hear.
A footnote to the Calm study states that the lead researcher took a job as director of science at Calm almost a year after publication of the research.
Apple’s report also cited a local radio station
Apple’s report cited evidence for one other mindfulness claim: “Numerous studies have shown meditation can lead to improvements in patients with high blood pressure, inflammatory bowel disease, and rheumatoid arthritis.”
The source on this was a story from WBUR, a local radio station in the Boston area.
The WBUR story focused on one Harvard University study, published by the Journal of Alternative and Complementary Medicine, that looked into the molecular mechanisms by which relaxing mind-body practices reduce blood pressure.
The radio station noted that “the study is small, and it didn’t include a comparison group of non-meditators. So it doesn’t count as absolute proof that meditation lowers blood pressure.” It added that the report’s author faced “often-withering criticism from his Harvard colleagues.”
What exactly is a ‘better day,’ and how do you measure it?
When Apple CEO Tim Cook introduced Activity Rings in 2015, he made the vague claim they would help you “live a better day.” What constitutes a “better day” is subjective, and therefore hard to quantify, scientifically.
In last month’s report, Cupertino made a more concrete claim. It described the Activity Rings as one of a set of features that “improve everyday health and fitness.” So what evidence did the report cite to support this?
There’s more to fitness than Move, Exercise and Stand
Physical fitness consists of many components, including aerobic activity, muscle strength, bone strength, flexibility and balance. But Apple Watch can only measure what’s detectable from your wrist. That may be why two of the three Activity Rings — Move and Exercise — are different ways of measuring the same thing: aerobic activity. They tell us nothing about how much strength training, balancing or stretching we’re doing, because those activities can’t be detected from our wrists.
The third ring tracks the Stand goal. The risks of a sedentary lifestyle are well-documented. However, it doesn’t necessarily follow that standing for one minute every hour (the Stand ring’s modest goal) helps if you are sitting down for the remaining 59 minutes.
With all this in mind, there were two areas of research I was hoping to see in the report:
- Health benefits: Do users who consistently close their rings enjoy tangible health benefits over those who don’t?
- Behavioral change: Does using the Activity app help users become more active and stay that way?
I didn’t find any research in the report on the first area, but it did cite a blog post on the second.
Incentivizing behavioral change with Apple Watch
The report cited a post on tech blog Apple World Today to support a claim that “providing an Apple Watch and incentives to participants helped increase their activity by 30-40%.” The blog post referenced research conducted by Rand on behalf of Vitality, a health insurer.
The incentives in question are Vitality Points, which the insurance company’s customers earn by logging activity on an Apple Watch. The watch is purchased with a consumer credit agreement when you take out the insurance. The more Vitality Points you earn, the lower your monthly repayments. So the incentive is good, old-fashioned money.
The fact that people change their behavior in return for money doesn’t tell us anything about how effective Apple Watch is at supporting behavioral change without financial incentives.
Sleep stage tracking is still in beta
Apple Watch has been tracking sleep since watchOS 7 debuted in 2020. But unlike some third-party sleep-tracking apps, the built-in Sleep app doesn’t currently monitor sleep stages. It just logs when you fall asleep and wake up. (That’s known in the trade as sleep-wake tracking.)
However, that all changes this fall. According to the report, watchOS 9 will use “signals from the accelerometer and heart rate sensor, [to] detect when users are in Core, REM, or Deep sleep.”
Currently, watchOS 9 remains in beta. We’ll have to wait until it’s officially released to test this feature properly.
No research provided for Apple Watch sleep-wake tracking
The report cites no research on the accuracy of Apple Watch’s current sleep-wake tracking. So, I decided to look it up for myself on Google Scholar. The most relevant study I found was from researchers at Penn State University, who concluded that data from Apple Watch “strongly correlated with reference devices [polysomnograms] at the epoch level and can be used to develop epoch-by-epoch models of sleep-wake rivaling existing research devices.”
That’s a lot of jargon, so let’s break it down. Epochs are small chunks of time scientists divide our sleep into. They’re usually between 20 and 60 seconds. Polysomnograms are the gold standard for sleep tracking. They combine data from a variety of sensors: an ECG measures heart activity, movement detectors monitor the limbs and eyes, and electrodes on the head record brain activity.
The fact that data from a wrist-based device like Apple Watch strongly correlated with polysomnogram results is impressive. But the researchers used their own machine learning predictive model, rather than Apple’s Sleep app. Two of the study’s authors were from Sleepspace, developers of a sleep-tracking app.
Do sleep trackers do more harm than good for people with insomnia?
Whether consumer sleep-tracking devices like Apple Watch offer any benefits remains an open question. A recent study from Oxford University suggested that sleep trackers can actually worsen sleep anxiety.
One of the authors, Matthew Reid, a post-doctoral researcher at Oxford, questioned whether knowing about our sleep is actually beneficial. He argued that it can cause “anxiety and low mood over sleep loss – leading to further sleeplessness.” He concluded that “while sleep devices might be useful for those who have generally good sleep but are interested in tracking or establishing a better routine, people who have poor sleep or mental health conditions may want to avoid them.”
Since sleep-tracking is now an area of focus for Apple, I would like to have seen more discussion and research on this in the report.
Atrial fibrillation detection
Apple Watch’s irregular rhythm notifications
Unlike the features we’ve looked at so far, the report actually does cite detailed research on the accuracy of Apple Watch’s atrial fibrillation detection.
Atrial fibrillation, or “AFib” for short, is an irregular heart rhythm caused by rapid and irregular beating of the atrial chamber of the heart. It can lead to blood clots, stroke, heart failure and other cardiac complications.
Apple introduced irregular rhythm notifications in watchOS 5.1.2 in 2018. This feature enables you to receive a notification if your Apple Watch detects an unusual pulse that may be AFib, so you can get it checked out by a doctor.
The report cited a large study on this feature. A very, very large study. In fact, it was the largest AFib screening study ever conducted.
Apple Heart Study: The largest ever AFib screening study
Apple announced the Apple Heart Study at an event in 2017. Conducted by Stanford University, using Apple’s ResearchKit framework, the study’s objective was to answer a simple question: Can Apple Watch detect AFib?
Apple Watch users in the United States, with no history of AFib, were invited to participate by downloading an app. If the app triggered an irregular rhythm notification, they were sent an ECG patch in the mail to wear for seven days. ECG patches are smart stickers you wear on your chest that monitor electrical activity in your heart. The researchers used data gathered from these patches to confirm whether participants actually experienced AFib.
Of the 419,297 participants, 2,161 received notifications from the app during the trial and thus received a patch. Disappointingly, only 450 of those who received a patch returned it, and just a third of the returned patches detected AFib.
The study’s authors noted that most of the patches were worn two weeks after receiving the notification, and AFib can come and go. So the fact the condition wasn’t detected by the patch doesn’t necessarily mean the original notification was wrong. More encouragingly, of those who received another irregular rhythm notification from the app while also wearing the patch, in 78.9% of cases, the patch confirmed Apple Watch’s findings of AFib.
Irregular Rhythm Notifications get FDA clearance
Apple submitted a subset of the data from this study to the Food and Drug Administration to get clearance for the irregular rhythm notification feature. According to documentation on the FDA website, the study “failed to meet the pre-specified performance end-goal.” But a secondary study and additional analyses proved “supportive of the device’s effectiveness to detect [AFib].”
A follow-up study of the data, also funded by Apple, found that when an irregular rhythm notification turned out not to be AFib, in a lot of cases it was “something else that probably needed some clinical attention.”
Further research is needed
The authors of the original AFib study noted that “uncertainty remains about the benefits of diagnosing and treating asymptomatic atrial fibrillation.” And the follow-up study highlighted that ECG patches were only sent to those who got a notification. So we don’t know how many people had AFib that was not detected by Apple Watch. As is typical with this kind of research, the report concludes that further research is required.
Apple’s health report is a mixed bag
All in all, I found Apple’s report to be a mixed bag. Serious health features backed by massive research projects wind up mixed with wellness features with little or no supporting evidence cited. Where the report makes vague references to “numerous studies,” I would have liked to have seen those studies cited. It’s not that I don’t believe they exist. However, the report would seem far more useful and comprehensive if it included the relevant citations.
Overall, I don’t think this report lived up to Apple’s usual high standards. I found three problems with the sources:
- Uneven editing: Some of sources refer to stories on tech blogs and local radio stations, rather than the actual research itself.
- Typos: The sources are littered with extraneous hyphens. For example, American Heart Association is consistently written: “American Heart Associa-tion.”
- No links: Typically, PDF reports like this link to primary sources. The lack of URLs meant I had to use Google to track down all the citations.
These problems aside, Apple’s report does highlight many impressive areas, like the Apple Heart Study. It also casts a light on interesting things I wasn’t previously aware of, like how Apple’s HealthKit and CareKit technologies are enabling patients to be treated in the comfort of their own homes.
Apple’s commitment to rigorous scientific validation processes for its health and fitness products is commendable. But I think this report could have done a better job of shining a light on that research.
* Based on an original image by Rouibi Dhia Eddine Nadjm, CC BY-SA 4.0 https://creativecommons.org/licenses/by-sa/4.0, via Wikimedia Commons