PNAS and the eHarmony dating site: a perfect match

Well, here's a shocker.  eHarmony, the online dating service, commissioned a survey that found out that couples who get married after meeting online are more satisfied than other couples.

I'm sure there's no bias in that survey.

But here's another shocker: a leading scientific journal just published the eHarmony survey as a bona fide scientific study.  The lead author is a consultant to eHarmony (and a former advisory board member) and another author is eHarmony's former research director, Gian Gonzaga.  According to the published paper, Gonzaga designed the study.

The journal in question is the Proceedings of the National Academy of Sciences (PNAS), a high-profile journal that is published by the prestigious U.S. National Academies.

Why would PNAS publish an article that is basically an advertisement for eHarmony.com?  I'm sure the editors at PNAS would argue that it's a well-executed scientific study, but they sure got lots of publicity, with articles in the Los Angeles Times, USA Today, and Nature, not to mention a short piece in Forbes. (And yes, I'm doing it too.) A cynic might point out that both eHarmony and PNAS got what they wanted.

But what about the study itself?  Well, let's take a look.  The study, titled "Marital satisfaction and break-ups differ across on-line and off-line meeting venues," is based on a survey of 19,131 people who got married between 2005 and 2012.  6,654 of these people met online, 35% of the total, which is a surprisingly high percentage.

The headline result is the claim that couples who met online have "higher marital satisfaction." People who met this way reported a satisfaction of 5.64, on average, versus 5.48 for those who met offline.  That's a very small absolute difference, but with such large numbers in the survey, even this small difference is, in a narrow technical sense, statistically significant. The satisfaction scale ranged from 1 "Extremely Unhappy" to 7 "Perfect."  So all this fuss and publicity is over a difference between 5.5 and 5.6 in a survey.

But the higher average satisfaction might have nothing to do with how the couples met.  In my reading of the study, it seems much more likely that other factors explain the difference.

Several things are immediately obvious when one looks at the composition of the online versus offline couples.  The online couples are significantly older, wealthier, and more educated: for example, 40.5% of those who met their spouse online reported an income of $100,000 or more, compared to just 26.1% of the offline couples.  These factors alone could explain the difference in marital happiness.

The authors claim that they controlled for all of these confounding factors, and that the marital satisfication score was still significantly higher for online couples. Alas, they don't provide enough details, even in their supplementary data, to evaluate this claim. I guess we're just supposed to trust them. (Note: I believe them when they say they controlled for these variables.  I'm just not sure precisely how they did it, or if the control function really eliminated all bias.) They did recruit two independent statisticians as co-authors, which is supposed to assure us that they were unbiased. But these steps wouldn't eliminate bias that might have crept in earlier, when the eHarmony-sponsored survey was being conducted.

The article also reports the marital satisfication score of specific online dating sites. Guess which one scores the highest? That's right: eHarmony.  Why am I not surprised?

At the end of the article, the authors speculate about why couples who met online might have more satisfaction in their marriages.  (This assumes, of course, that the effect is real.)  They suggest that

"among on-line dating sites, it is also possible that the various matching algorithms may play a role in marital outcomes."  

There it is!  That's the conclusion that eHarmony wanted.  I think they got their money's worth.

In many ways, this study seems like the often-criticized studies funded by drug manufacturers that find small but significant benefit for the sponsors' drugs. Certainly there's a difference here, in that the only result is that someone might be convinced to try an on-line dating site, which might not be harmful at all.

It seems that eHarmony has found another perfect match: PNAS and eHarmony.com.

(Note: The authors did make all their raw data available, a feature that is still quite rare in scientific publishing.  They deserve kudos for doing so.  I've long advocated for more openness in data release and these authors have done the community a service by releasing theirs.)