Why Google Flu is a failure: the hubris of big data

It seemed like such a good idea at the time.

People with the flu (the influenza virus, that is) will probably go online to find out how to treat it, or to search for other information about the flu. So Google decided to track such behavior, hoping it might be able to predict flu outbreaks even faster than traditional health authorities such as the Centers for Disease Control (CDC).

Instead, as the authors of a new article in Science explain, we got "big data hubris."  David Lazer and colleagues explain that:

“Big data hubris” is the often implicit assumption that big data are a substitute for, rather than a supplement to, traditional data collection and analysis.

The folks at Google figured that, with all their massive data, they could outsmart anyone.

The problem is that most people don't know what "the flu" is, and relying on Google searches by people who may be utterly ignorant about the flu does not produce useful information. Or to put it another way, a huge collection of misinformation cannot produce a small gem of true information. Like it or not, a big pile of dreck can only produce more dreck. GIGO, as they say.

Google's scientist first announced Google Flu in a Nature article in 2009. With what now seems to be a textbook definition of hubris, they wrote:

"...we can accurately estimate the current level of weekly influenza activity in each region of the United States, with a reporting lag of about one day."

They obtained this remarkable accuracy entirely from analyzing Google searches. Impressive - if true.

Ironically, just a few months after announcing Google Flu, the world was hit with the 2009 swine flu pandemic, caused by a novel strain of H1N1 influenza. Google Flu missed it.

The failures have continued. As Lazer et al. show in their Science study, Google Flu was wrong for 100 out of 108 weeks since August 2011.

One problem is that Google's scientists have never revealed what search terms they actually use to track the flu. A paper they published in 2011 declares that Google Flu does a great job. The official Google blog last October makes it appear that they do an almost perfect job predicting the flu for previous years.

Haven't these guys been paying attention? It's easy to predict the past. Does anyone remember the University of Colorado professors who had a model that correctly predicted every election since 1980? In August 2012, they confidently announced that their model showed Mitt Romney winning in a landslide. Hmm.

Flu cases this year, which are dominated by H1N1.

A bigger problem with Google Flu, though, is that most people who think they have "the flu" do not. The vast majority of doctors' office visits for flu-like symptoms turn out to be other viruses. CDC tracks these visits under "influenza-like illness" because so many turn out to be something else. To illustrate, the CDC reports that in the most recent week for which data is available, only 8.8% of specimens tested positive for influenza.

When 80-90% of people visiting the doctor for "flu" don't really have it, you can hardly expect their internet searches to be a reliable source of information.

Google Flu is still there, and you can still look at its predictions, even though we know they are wrong. I recommend the CDC website instead, which is based on actual data about the influenza virus collected from actual patients. Big data can be great, but not when it's bad data.

A DNA Sequencing Breakthrough for Pregnant Women

DNA sequencing has made its way to the clinic in a dramatic new way: detecting chromosomal defects very early in pregnancy.  We've known for 25 years that traces of fetal DNA can be detected in a pregnant women's blood. But these traces are very small, and until now, we just didn't have the technology to detect an extra copy of a chromosome, where the DNA itself is otherwise normal.

Last week, in a study published in The New England Journal of Medicine, Diana Bianchi and colleagues showed how DNA sequencing can detect an extra copy of a chromosome with remarkable accuracy. This report heralds a new era in prenatal DNA testing.

First, some background: three copies of chromosome 21 causes Down syndrome, a genetic disease that causes intellectual disability and growth delays. Down syndrome is also called trisomy 21, where trisomy = 3 copies of a chromosome instead of the normal 2 copies. Much less common is Edwards syndrome, caused by three copies of chromosome 18. Edwards syndrome, or trisomy 18, has much more severe effects, with the vast majority of pregnancies not making it full term. Having an extra copy of any other chromosome almost always causes an early miscarriage. For many reasons, prospective parents want to know if a fetus carries any of these abnormalities.

The accuracy of the new test is remarkable. Out of 1914 young, healthy pregnant women, there were just 8 pregnancies where the fetus had an extra chromosome, and the test detected all 8. What was most impressive was its low false positive rate: in total, the new DNA-based test had just 9 false positives (for either chromosome 21 or chromosome 18 trisomy).  By contrast, the conventional screening test, which also identified all 8 true cases, produced 80 false positives, nearly 9 times as many as DNA sequencing.

Why does this matter? In most cases, women with a positive result on one of these tests will opt for amniocentesis ("amnio"), an invasive procedure where a doctor inserts a long needle directly into the womb and collects a sample of amniotic fluid. Amnio almost always gives a definitive answer about Down syndrome. With the conventional method, its false positive rate is so high that even with a positive test, over 95% of amnios will be negative, versus 55% with the new DNA sequencing test. Or to put it another way, as Bianci et al. wrote:

"if all women with positive results had .. decided to undergo an invasive procedure, there would have been a relative reduction of 89% in the number of diagnostic invasive procedures."

89% fewer invasive procedures is a huge reduction, not only in costs but in stress for the parents and risk to the baby (because amnio carries a small risk of miscarriage).

With DNA sequencing getting faster and cheaper every year, it might be surprising that we are only now seeing it used to detect trisomy. The difficulty with detecting an extra copy of a chromosome is that the DNA sequence itself is normal. If you sequence the genome, you won't find any mutations that indicate that the fetus has an extra chromosome copy. This is where the remarkable efficiency of next-generation sequencing comes in.

In a matter of hours, modern sequencing machines can sample millions of small fragments of DNA. We can use computational analysis to determine which fragments come from the fetus, and how many came from each chromosome. If any chromosome has three copies, we'll see a 50% increase in DNA from that chromosome. The power of sequencing lies in large numbers: because we can sequence many fragments from each chromosome, a 50% increase is easy to detect.

The method that Bianchi used to detect trisomy was published in 2011 by Amy Sehnert and colleagues from 2011, some of whom are contributors to the new NEJM study. [Side note: they use a software program called Bowtie, developed by my former student Ben Langmead, to do the analysis.] The method is likely to get even better over time, further reducing the false positive rate.

The American College of Obstetricians and Gynecologists has already recommended DNA testing for pregnant women at high risk of fetal aneuploidy (an extra chromosome). To be precise, they recommend that high-risk pregnant women be offered fetal DNA testing as an option, after they get genetic counseling. This new study, which was conducted in a low-risk population, shows that the benefits of prenatal DNA testing should offered to all women.

Can you patent a fraudulent stem cell method? Yes!

Woo-Suk Hwang talks to reporters after
fraud is revealed. Photo: Reuters.

At first I thought the Patent Office was having a little fun. Was it an April Fools Day joke?  No, it's only February - and the U.S. Patent Office never kids around.

What did they do? They issued a patent to Korean scientist Woo-Suk Hwang for a method to create human embryonic stem cells by cloning.  The problem is, Hwang's "invention" was one of the most famous frauds of the past decade. His publications in 2004 and in 2005, in the journal Science, are labelled in bright red letters as retracted, and Science wrote its own separate notice explaining

"the authors of two papers published in Science (2, 3) have engaged in research misconduct and that the papers contain fabricated data."

Hwang's apparent triumph, becoming the first scientist to create human embryonic stem cells in the lab, made him a national hero in South Korea, for a short time.  He was soon appointed the director of a new stem cell research center. But things quickly unraveled beginning in November 2005, when Hwang's co-author Gerald Schatten, a stem cell researcher at the University of Pittsburgh, announced that he was ending his collaboration with Hwang over ethical concerns. By January, Hwang admitted to publishing fake data, but blamed his junior colleagues. Hwang was fired from Seoul National University (SNU) in 2007 and later convicted of bioethical violations and embezzlement. The official investigation by SNU found that Hwang's laboratory

"does not possess patient-specific stem cell lines or any scientific basis for claiming to have created one."

So you wouldn't think this would be approved for a patent, no?  Is the patent office paying any attention at all?  As reported by Andrew Pollack at the New York Times, the patent office does indeed know Hwang's history, and the patent is 

"definitely not an assertion by the U.S. government that everything he is claiming is accurate."

Well, I must say I'm relieved to hear that. Hwang himself admitted the data were fake! As I've written previously, the USPTO simply can't keep up with biotechnology, and the courts don't do any better. In this case, it's hard to imagine a more obvious example of a patent that should be denied: the papers were retracted, and the lead scientist lost his job after his own university concluded that the data was fabricated. And yet the patent office is standing by their decision. What are they thinking?

Internet quack Joe Mercola is worried. Dr. Oz to the rescue!

Dr. Oz interviews Joe Mercola on his show.

After a series of studies showing that vitamins and supplements are usually a waste of money, including my recent article on the top 6 vitamins you shouldn’t take, internet supplement salesman Joe Mercola is worried.  He should be: his Internet-driven empire is largely based on sales of vitamins and supplements, for which his claims range from merely implausible to dangerously untrue, including:

• Himalayan salt is far better for you than regular salt. Coincidentally, Mercola sells the good salt.
• Vitamin D is "the new silver bullet for cancer." No, it's not.  Not even a little bit.
• Homeopathy cured a boy of autism. Of course Mercola doesn't provide any proof of this, only a testimonial.  This is not even remotely plausible, since homeopathic preparations are just sugar pills.

I could provide many more examples, but this should be enough to demonstrate that I'm not making this stuff up.

Mercola is also one of the loudest voices and worst offenders in the anti-vaccine movement. Among other misinformation, he claims that the hepatitis vaccine causes autism, and his website urges people to use his supplements instead of getting vaccinated.

So how do I know Mercola is worried? He's appearing on the Dr. Oz Show on Monday, February 10 (the day after I'm writing this) to talk about multivitamins. Apparently his 10 minute segment wasn't enough, so he posted an article on his website with the "Information I couldn't share" on Dr. Oz's show.

Does the article explain why multivitamins are actually good for you?  Well, no. Most of the article is a big red herring, in which he argues that supplements should not be regulated as drugs, because "we have all the regulations we need." Then he contradicts himself and says that the FDA already regulates supplements. (It doesn't - or to be more precise, the FDA does not require supplement makers to prove their products work. It can only step in if the products start to kill people. This is what Mercola calls regulation.) Besides, he says, supplements are harmless. As evidence, he cites a press release from a pseudoscientific organization that claims "no deaths from supplements in 27 years."

Not surprisingly, Mercola doesn't cite any actual science to support his claims. In contrast, several very large studies in major medical journals, cited in my own columns last month and last October, show that routine supplementation with multivitamins, especially with the megadoses that many people take, can indeed cause genuine harm. Those same studies showed that if you don't have a deficiency, there's simply no benefit to taking most vitamins.  Mercola's response is to cite opinion pieces from his own website that simply assert, without any evidence, that the studies are wrong.

In other words, Mercola's response is "Oh yeah?" He then goes off on a tangent and launches an irrelevant ad hominem attack on noted vaccine expert Dr. Paul Offit.

Why has Dr. Oz repeatedly had Joe Mercola on his show? This is a tough one. Does Oz believe that autism is caused by vaccines, something Mercola has claimed repeatedly over the years?  Does he understand that Mercola's anti-vaccination campaign leads to genuine harm? Does he know that the FDA has repeatedly issued warnings to Mercola to demand that he stop making false claims about his vitamins and supplements, as Chicago Magazine reported?

Or does Dr. Oz keep inviting Mercola back because he knows Mercola has a big audience that will increase his own viewership?

Despite my past criticism of Dr. Oz, I still think he has a better grasp of science than Joe Mercola. He also reassures viewers constantly that he doesn't sell the products that appear on his show. And yet Oz is giving a platform to someone who makes huge profits selling products based on unproven claims. By having Mercola on his show, Oz is giving him free publicity and helping him sell those same products. And whether or not Oz agrees with Mercola, he is helping to give credibility to Mercola's wildly inaccurate and dangerous anti-vaccine claims.

Brief note: a new vaccine mythbuster site appears

Vaccines are probably the single greatest advance in public health ever invented. Despite their enormous benefits, and the overwhelming evidence in their favor (we've wiped out smallpox and nearly wiped out polio), anti-vaccine activists continue to spread rumors and wildly inaccurate, but scary tales about the potential harms of vaccines.

Today a new website was published, a collection of 100 other sites that provide good information on the benefits of vaccines, and that bust many of the most damaging anti-vaccine myths.  This site is on the list, but you'll find many other excellent sources of good science about vaccines.  We need every bit of help we can get to counter the mis-information about vaccines promoted by groups such as Generation Rescue and Age of Autism.

What's up with the magic tape that athletes are wearing?

Li Na with kinesio tape on her knee, playing on clay.

When Chinese tennis star Li Na won the Australian open final this weekend, she had two long, wide strips of black tape criss-crossing her right knee. And in the semifinals, Poland’s Agnieszka Radwanska had two similar pieces of tape criss-crossed over her right shoulder.  Sports fans have been seeing this sight for a few years now: we saw these big pieces of tape slathered across the legs of many Olympic swimmers in the 2008 and 2012 Olympics. 

So what is this tape on the knees, shoulders, and thighs of top athletes? And more important, does it work?

Why is it so popular? After all, with so many world-class athletes using it, it must help somehow, right? Nope. Athetic tape has become hugely popular thanks to clever marketing: during the 2008 Olympic games in Beijing, tape manufacturer KT tape donated its “kinesio tape” to 58 countries for their athletes to use. Many athletes gave it a try, and millions of viewers saw it on television.

Athletes are notoriously superstitious. If they have a good performance, they will obsessively try to repeat everything they wore, ate, or otherwise did that might have led to that performance. Obviously, with so many athletes wearing elastic tape in 2008 games, some of them performed well. This merely reinforces the superstition that somehow the tape helped.

Tape man! (From the KT website.)

Fortunately, this claim is pretty easy to study, and multiple studies were done after the 2008 Olympics, looking at the possible benefits of athletic tape. A review of ten scientific studies published in the journal Sports Medicine in 2012 found little benefit. They reported that 

“The efficacy of KT [kinesio tape] in pain relief was trivial given there were no clinically important results.” 

Other studies looked at other effects, and for these the authors reported “KT had some substantial effects on muscle activity, but it was unclear whether these changes were beneficial or harmful.” Not a slam dunk for the benefits of kinesio tape, apparently.  

Tennis player Li Na was wearing the tape on her knee. What does the science say about tape for knee pain in particular? Here the evidence is not so clear. A small study last year by Marc Campolo and colleagues compared at Kinesio Tape® to another taping method called the McConnell Taping (MT) and to no tape at all. The difference between the tapes is that KT is elastic, while the MT technique uses rigid tape. This study found a small benefit for both types of tape on pain while climbing stairs or doing knee squats. This result is interesting, but the study only had 20 subjects, far too few to say anything conclusive. And it wasn't "blinded" - the subjects obviously knew they had tape on their knees. So they might have been telling the scientists what they wanted to hear, a common phenomenon in these types of studies.

Of course, the company that sells KT tape is not nearly so careful in their claims. Their website says:

“KT TAPE is lightweight, comfortable to wear, and can be used for hundreds of common injuries such as lower back pain, knee pain, shin splints, carpal tunnel syndrome, and tennis elbow, just to name a few.”

This is, to put it mildly, a bit of overstatement.  As the recent review article stated, when it comes to pain relief, the evidence shows “no clinically important results.”

There are a remarkable number of studies looking at various kinds of taping for joint or muscle problems, but the evidence is, on the whole, weak at best. My reading is that elastic tape probably has no more than a placebo benefit. But in sports, a placebo benefit can be significant: if you think it helps, it might really help.  

And think of the other benefits: KT tape comes in lots of cool colors, and for just a few bucks, you’ll look like a pro. A women's pro, that is: I didn't see any tape on Rafael Nadal or Stanislas Wawrinka in the men's final. Maybe that's because removing a big piece of tape from a hairy leg has got to hurt.

The top 6 vitamins and supplements you shouldn’t take

The evidence against supplements continues to pile up.

Recently I created a list of The Top 5 Vitamins You Shouldn’t Take. Now I’m expanding that list to include vitamin D, which is taken by almost half of older adults. Now, two new studies in latest issue of The Lancet show that most of these people are wasting their money.

The first study is a large review by Philippe Autier and colleagues, who found that taking supplemental vitamin D has no effect on a wide range of diseases and conditions. After looking at over 450 studies, the authors conclude:

“The absence of an effect of vitamin D supplementation on disease occurrence, severity, and clinical course leads to the hypothesis that variations [in vitamin D levels] would essentially be a result, and not a cause, of ill health.”

So it appears that we’ve been getting cause and effect backwards, at least as far as vitamin D is concerned. Autier looked at non-skeletal disorders, including heart disease, weight gain, mood disorders, multiple sclerosis, and metabolic disorders, all of which have been linked to lower vitamin D. In trial after trial, supplemental vitamin D had no beneficial effect on any of these conditions. Autier et al concluded that:

“associations between 25(OH)D and health disorders … are not causal. Low 25(OH)D [vitamin D] could be the result of inflammatory processes involved in disease.”

Instead, the researchers found, in study after study, that low vitamin D was the result of poor health, not the cause.

The Autier study didn’t look at the biggest supposed benefit of vitamin D: protection against osteoporosis. We’ve long known that vitamin D is associated with bone health. Fortunately, in the same issue of The Lancet, Ian Reid and colleagues looked closely at this question. They reviewed 23 studies with 4082 participants, all designed to determine at whether supplemental vitamin D improves bone density. Their findings? It doesn’t help. They concluded:

“Continuing widespread use of vitamin D for osteoporosis prevention in community-dwelling adults without specific risk factors for vitamin D deficiency seems to be inappropriate.”

Vitamin D supplements, to put it plainly, are a waste of money.

(For those concerned about osteoporosis, the widely used drug alendronate (Fosamax®), has been shown to increase bone density by about 5%, as explained in a 2011 article by Dr. Sundeep Khosia. But Fosamax has side effects.)

It’s pretty easy to get enough vitamin D in a normal diet. Or, as Dr. Mark Gillinov explained in the Huffington Post last week, just 10 minutes of sunlight gives you about 4 times your daily recommended vitamin D requirement.

So here's my expanded list of the Top Six Vitamins You Shouldn’t Take, with the newest entrant at the end:

1. Vitamin C
2. Vitamin A and beta carotene
3. Vitamin E
4. Vitamin B6
5. Multi-vitamins
6. Vitamin D

You can read more about the first five in the original list.

What’s left? Well, if you don’t have a deficiency, there’s no reason to take any supplemental vitamins at all.  As my Hopkins colleagues Eliseo Guallar, Lawrence Appel, and Edgar Miller wrote last month in the Annals of Internal Medicine : “Enough is enough: stop wasting money on vitamin and mineral supplements.”  As they wrote, after looking at three more large studies just published last month,

"most supplements do not prevent chronic disease or death, their use is not justified, and they should be avoided."

Save your money. Or better yet, if you must spend it, buy a bit more fresh fruit. You'll be healthier for it.