84,395 Users, Effect Size 0.7: Why a Pretty Number Doesn't Mean a Self-Hypnosis App 'Works'

If you saw a headline — "84,000 users, an app significantly cuts stress, effect size of 0.7" — what should your very first reaction be?

It shouldn't be "great, downloading it." And it shouldn't be "must be an ad." It should be: let me look at how the study was actually designed. Because there's a paper from December 2025, in npj Digital Medicine, about a self-hypnosis app — and it's a perfect workout for exactly that instinct. The numbers look gorgeous. But sitting right underneath them is a whole row of question marks, and those deserve some daylight.

So let's set the stage. The app is called Reveri — and one of its co-founders and scientific advisors is David Spiegel, from Stanford. The dataset is big: over 84,000 users, nearly 283,000 stress sessions, collected from late 2021 through the start of 2025. Here's how the measurement worked. Before each session and after each session, users rated their own stress on a single ten-point scale — one is lowest, ten is highest. The study looked at the difference: after, minus before.

And the authors are refreshingly honest about why they used a single one-item rating instead of a proper, validated, multi-question questionnaire. Their reason? To maximize engagement and data collection. That's a real tradeoff, and worth pausing on — they traded measurement precision for an eighty-four-thousand-person sample. Coarser ruler, way more data.

Now the headline numbers. Across the first ten sessions, the effect size — Cohen's d — lands between negative 0.71 and negative 0.78. For reference, the rule of thumb is: 0.2 is small, 0.5 is medium, 0.8 is large. So this is knocking on the door of "large." In plain terms, average stress dropped about one and a half points out of ten.

A few interesting wrinkles in the details. People who scored as more hypnotizable tended to drop a bit more — but honestly that link was weak. The interactive, full-length sessions worked noticeably better than the quick brief ones — roughly 1.8 times the effect. Younger users got a bigger hit on their very first session; older users built up more benefit over time. Paying members improved more. And here's a curious one — repeating sessions did not stack the effect. Doing it over and over didn't compound. On safety, it looked clean: out of all 84,000-plus users, only ten reported anything getting worse, and all of it minor.

Okay. Now the heart of the whole thing — why a pretty number still isn't a conclusion. Let me walk you through the reasons, because each one chips away at that 0.7.

Reason one, and it's the fatal one: there's no control group. None. All we actually know is that people rate their stress lower after a session. What we don't know is whether they'd have dropped just as much doing nothing at all — or just sitting quietly with their eyes closed for five minutes. The authors admit this straight up: it limits any claim about cause.

Reason two: regression to the mean. Think about when you actually open a stress app. You open it when you're stressed. That's a high, unusual moment — and unusually high states naturally drift back toward your average the next time you check, with zero intervention. A before-and-after design bakes that natural rebound right into what looks like the "effect."

Reason three: it's self-reported, it's a single item, and expectation is in the room. The user knows they're using a stress tool. They know the question is about stress. They just spent time on something that's supposed to help. So how much of that drop is genuine physiological calm, versus "I expected this to work, so I typed a lower number"? A one-question scale simply cannot separate those two things.

Reason four: selection bias. Who's in this sample? People who downloaded a paid self-hypnosis app and were diligent enough to fill out ratings before and after. That's a heavily self-selected crowd — not a slice of the general public. And remember paying members did better? That's probably just "more invested people feel more effect."

And reason five — this is the one I really want to land. A big sample does not fix any of the previous four. People see N equals 84,000 and think, surely that many people can't be wrong. But here's the thing: a large sample only makes your estimate more precise. It does not make a biased design unbiased. Regression to the mean at 84,000 people is still regression to the mean — you've just measured a contaminated number with tremendous confidence. The authors say the large sample helps mitigate confounding, and that's true for random noise — but not for systematic bias. Precision and accuracy are not the same animal.

So — is the study worthless? No. Not at all. Put it back where it belongs and it's genuinely valuable. It shows you can systematically collect and analyze real-world usage data — that's the backbone of digital therapeutics. It hands you a signal and an effect-size range that are absolutely worth confirming with a proper randomized trial. The safety data, ten out of 84,000, is actually informative at that scale. And the moderators — the fact that interactive sessions worked better — have real product design implications. What it can't do is be read as "the hypnosis app is proven to reduce stress." What it can be read as is: among self-selected users, self-rated stress drops after a session, and that's worth a controlled trial.

So let me leave you with the part you can actually carry around — the data-literacy checklist. Next time you see a "huge sample, beautiful effect size" health or behavior study, ask these in order. One: is there a control group? If no, you're probably looking at correlation, full stop. Two: is it before-and-after? If yes, watch for regression to the mean and expectation effects. Three: how was it measured? Self-reported single item means subjective and expectation-prone. Four: where did the sample come from? Self-selected means don't extrapolate. And five: is that big sample solving random error, or is it being used to paper over a design flaw?

And here's the kicker — the exact same instinct applies in engineering. An A/B test without a control isn't an A/B test. A gorgeous metric with no control and no randomized split is usually showing you regression to the mean and selection effects — not the actual merit of your change. Reading papers and reading dashboards take the same caution.

So, three things to walk away with. First: a big N and a big effect size prove neither rigor nor causation — without a control group, you've got correlation, dressed up nicely. Second: before-and-after measurement quietly invites regression to the mean and expectation bias, especially when people show up precisely because they feel bad. And third: a large sample buys you precision, never accuracy — it'll measure a flawed design with beautiful, misleading confidence. Look at the design first. Always the design first.