Field of Science

Clinical trials doctors claim they own patients' data, want others to pay for access

Image from "Honoring our promise: clinical trial data sharing"
at the NIH Director's blog.
Earlier this month, a consortium of 282 clinical trials doctors, in an article in the New England Journal of Medicine, threw down the gauntlet against the notion that they should share data. Under the misleading title “Towards fairness in data sharing,” these scientists, who label themselves The International Consortium of Investigators for Fairness in Trial Data Sharing, put forward the breathtakingly arrogant claim that doctors who collect data from patients should control it essentially forever.

I wonder what their patients would think if they knew.

This is the second salvo from NEJM this year. Back in January, the editors of NEJM published an opinion piece that used the derogatory phrase “research parasites” to describe scientists who want to re-analyze data from other scientists’ experiments. This caused an outcry, with many scientists pointing out serious flaws in the editors’ arguments. As I wrote at the time,   
“Drazen and Longo [the NEJM editors] are saying, essentially, that only the people who originally collect a data set can truly understand it, and anyone else who wants to take a look is a parasite.”
Keeping scientific data locked away is a recipe for bad science. In the absence of data sharing, flawed results may go unchallenged for years. If someone has distorted or misinterpreted data, we need someone else–someone not invested in proving the same result–to take a second look. This is how science corrects itself.

Even so, at least Drazen and Longo endorsed some data sharing. The authors of this new article (P.J. Devereaux, Gordon Guyatt, Hertzel Gerstein, Stuart Connolly, and Salim Yusuf, all from Ontario’s McMaster University) and their consortium are vehemently opposed to sharing their data with anyone, ever.

Let’s go through their arguments. First, they point out that it takes years of work to set up and run a clinical trial. Absolutely right–no argument there. But then they explain why they conduct these trials:
“A key motivation for investigators to conduct RCTs is the ability to publish not only the primary trial report, but also major secondary articles based on the trial data. The original investigators almost always intend to undertake additional analyses of the data and explore new hypotheses.”
Oh really? This is why they run clinical trials, in order to publish papers? Somehow I doubt that is what they tell the patients as they are asking for informed consent. Not “we want to figure out what is making you sick” or “we want to find a cure”, but “we want to publish a paper, or maybe several papers!” I’m sure that patients would be lining up around the block to join these studies.

Don’t get me wrong: of course the scientists want to publish their findings, and of course they would like to mine the data for more papers, year after year. That’s what some investigators do now. In this world, doctors running clinical trials would never share data. Why would they, when things are working so well for their careers?

But they seem to have forgotten THE motivation for clinical trials: curing disease. I am honestly dumbfounded that Devereaux and his colleagues don’t mention the words “disease” or “illness” in their article, not even once. Instead they focus on “risks” of sharing data, by which they mean the risk that someone else will make a discovery that they didn’t think of, versus the “benefits,” which in their world means either confirming the original study or possibly testing a new hypothesis. They seem oblivious to the notion that re-analyzing the data to contradict the original claims might actually be a benefit to the rest of the world.

Appalling. Did they feel this way when they first went into medical research? Probably not, but somewhere in the competitive struggle to succeed as researchers, they lost their way.

It gets worse. Devereaux and his consortium go on to lay out what they might be willing to consider:
  • Exclusive use of the data for a minimum of 2 years after the first publication
  • Another 6 months of exclusivity for every year the trial lasted, up to 5 years of exclusivity
  • Anyone who wants the data should pay the original investigators “for their efforts and investments in the trial.”
This last point is the most outrageous of all. Hello? Investigators in clinical trials are already paid for their efforts (though the patients are not), often totaling millions of dollars, and often from taxpayers’ money. Now they want to be paid again by each scientist who wants to look at the data?

These guys need to get over it. The reason we fund clinical trials is to cure disease, not to allow Devereaux and company to publish papers. Publication is the means by which we communicate results, but it’s not the reason that anyone pays for the experiments.

Interestingly, in the same issue of NEJM, U.S. Senator Elizabeth Warren writes about data sharing, expressing quite the opposite view from Devereaux and company. I won’t attempt to summarize her points here, but instead let me quote from one of the public comments published in response to the anti-data-sharing doctors:
“ `This is not working. I just hope that my death will not be in vain and someone learns from it so that others don't have to go through this’ was what my husband said when it became clear that the new drug combination was not working fast enough for us…. My husband died now more than 4 years ago. I have watched his and others' lives been plotted on curves at conferences like ASCO or EMSO and it is deeply upsetting that there are more than 280 researchers who are more concerned about their own careers than the lives of the patients on whom those careers are built. Claiming rights to data paid for with the lives of others has nothing to do with fairness.” Bettina Ryll, M.D., Ph.D., Melanoma Patient Network Europe
I couldn’t say it any better than that. The International Consortium of Investigators for Fairness in Trial Data Sharing–as the 282 investigators called themselves–should be ashamed.

Someone needs to tell Michael Phelps to stop bruising himself

It happens every four years. No, not the Olympics, though of course that happens too. I'm talking about new wackadoodle performance enhancement fads. In the 2008 and 2012 Olympics, it was magic tape, which we saw plastered across the arms and legs of many swimmers. This year it's "cupping," a crude technique that leaves nasty red or purple welts all over your body. Michael Phelps, one of the best male swimmers in Olympic history, featured these lovely welts in photos this week, and you can catch a glimpse of him getting the treatment in the UnderArmour "Rule Yourself" ad featuring Phelps (which is otherwise an excellent ad).

Phelps isn't the only one. Former Olympic swimmer Natalie Coughlin and gymnast Alex Naddour have displayed the distinctive circular welts too.

The New York Times, USA Today, and People all ran articles yesterday explaining what those nasty red bruises were all over Phelps' torso and shoulders. USA Today reported, unquestioningly, that athletes use cupping "to relieve tension in their muscles." The NY Times (from which we might expect a bit more skepticism) reported blithely that "Physiologically, cupping is thought to draw blood to the affected area, reducing soreness and speeding healing of overworked muscles." People used the same argument, and then added this bit of illogic:
"given that Phelps took home his 23rd Olympic medal last night, it's tough to argue."
No, it's not. Cupping is ridiculous. There's no scientific or medical evidence that it provides any benefit. The NY Times did express a bit of doubt (though not much), summarizing two small experiments that showed that cupping worked no better than placebo. But the Times couldn't help itself, and went back to quoting anecdotal evidence from true believers.

Rather than review the evidence here, I refer you to a thorough takedown of cupping written last month by Orac, a well-known science blogger who is also a surgeon:
"Among the silliest of alternative medicine therapies is something called cupping.... The suction from cupping breaks capillaries, which is why not infrequently there are bruises left in the shape of the cups afterward."
As Orac points out, repeated cupping in the same spot can destroy your skin and lead to dangerous infections. Another physician, Dr. Harriett Hall, in an article for Slate in 2012, made similar points.

I'm not sure who told Michael Phelps that cupping would help him swim faster, but I am sure that it's terrible advice–definitely not helpful and maybe harmful.

I know athletes are notoriously superstitious, and they get some psychological benefits from the various rituals they use to prepare for competition. There's no harm in believing that magic tape, or a lucky shirt, or always stepping onto the field with your left foot first–or whatever–will help you win. But Michael Phelps and his fellow Olympians should run as fast as possible from unproven treatments (and make no mistake, cupping is one of these) that can only cause them harm.

Did a biotech CEO successfully reverse her own aging process? Maybe not.

Elizabeth Parrish, CEO of BioViva.
Humans have been searching for the fountain of youth for millenia, dating back to ancient times. No one has found it yet, so I was very skeptical when I saw the recent announcement from BioViva, a biotech company, of what they called the first successful gene therapy against human aging:
"Elizabeth Parrish, CEO of Bioviva USA Inc., has become the first human being to be successfully rejuvenated by gene therapy, after her own company's experimental therapies reversed 20 years of normal telomere shortening."
That's quite a dramatic claim. If true, this would be a historic breakthrough: no one has ever reversed aging before. While human life expectancy has doubled over the past 150 years, virtually all of this progress has been from preventing early deaths, thanks to the developments of antibiotics, vaccines, and public health advances such as clean water.
Human life expectancy has doubled since the 1840's.
Figure source: Natl Institute on Aging.
Most claims about anti-aging therapies are easily dismissed as pseudoscience, nonsense, or scams. Not this one, though. BioViva has two experimental therapies, both based on legitimate science, and both with at least a chance of working. Neither has yet been proven to work in humans, but both are plausible.

According to BioViva and to interviews with its CEO, Elizabeth Parrish, Parrish received two therapies last year, one to protect against the loss of muscle mass, and one to lengthen her telomeres. The recent announcement claims that the telomere-lengthening therapy is already working, so I looked a bit deeper to understand what might be going on.

First a bit of background: telomeres are special DNA sequences that act as "caps" on both ends of every chromosome, providing a kind of protection for your genes. Each time a cell divides, its telomeres get a little bit shorter, and eventually they get too short and the cell dies. Telomeres therefore act as a kind of molecular clock that tells a cell how old it is. Our cells also have a special enzyme called telomerase that rebuilds telomeres. Cells with lots of telomerase can live much longer, and those without it die more quickly. Discovering how this all worked was a tremendous scientific achievement, for which Elizabeth Blackburn, Carol Greider, and Jack Szostak received the 2009 Nobel Prize.

Scientists have been speculating for years that telomerase might somehow hold the key to aging. BioViva's gene therapy delivers telomerase to the blood with the help of weakened viruses called adeno-associated viruses (AAVs), which they modified to carry the telomerase gene. The virus infects human cells and releases its payload into them, where the "transgene" produces extra telomerase.

This may sound very nice, but it's really, really complicated in practice. Gene therapy can have unexpected negative effects, and no human trials have yet shown that anyone can deliver telomerase effectively to human cells. However, studies in mice have shown some remarkable results: in 2012, a group of scientists at the Spanish National Cancer Centre used AAV to deliver telomerase to mice, and found that it "had remarkable beneficial effects on health and fitness" and that
"telomerase-treated mice, both at 1-year and 2-years of age, had an increase in median lifespan of 24 and 13%, respectively."
This exciting scientific result, and a few others like it, are what led BioViva and Elizabeth Parrish to try the same therapy in humans.

But did it work? Well, this is where things get a bit fuzzy. BioViva claims it did, based on their measurements of the length of telomeres in Parrish's white blood cells in September 2015, before therapy started, and again in March 2016. They claim that her telomeres got longer, from 6.71 kilobases (a kilobase is 1000 DNA letters) to 7.33 kilobases. This increase corresponds to about 20 years of aging: in other words, Parrish's white blood cells "have become biologically younger," as the company reported.

Setting aside the problem that we cannot really conclude anything from an experiment involving only one person, we can still ask: did Parrish's telomeres really get longer? As much as I want to believe BioViva's claim, there are several rather serious problems here. First, the company itself reported that Parrish's telomeres were unusually short for her age before the experiment began. Does this mean that the measurements were simply a bit off, and the second measurements were closer to the true number? Second, as UCLA's Prof. Rita Effros explained in an interview at geneticexperts.org,
"The overarching problem is that peripheral blood contains a mixture of many different cell types with disparate telomere lengths.... Thus, a simple change in the proportion of different cell types within the peripheral blood could easily explain the data."
In other words, it's possible that Parrish's telomeres did not get any longer. Despite the apparently precise numbers, BioViva has not provided any details showing that these measurements are accurate and reproducible (and they didn't respond to my request for these details). Their claim might be much more convincing if they made multiple measurements, both before and after treatment, and if these measurements showed that Parrish's telomere lengths really did increase.

There are a number of red flags about BioViva itself. Parrish herself is not a scientist, though she is an eloquent spokesperson for her company's therapies. More concerning is their Chief Medical Officer, Jason Williams, who previously ran "a dubious stem cell clinic," Precision StemCell (now located in Mexico) that offers stem cell therapies to patients with ALS (Lou Gehrig's disease), for which there is no evidence that they work. Personally, I would not trust Dr. Williams with my medical care.

The bottom line is that we simply don't know if BioViva's treatment worked on Elizabeth Parrish. They need to produce more data, on more patients, to construct even a mildly convincing scientific argument. Getting more patients may be very difficult, though: Parrish bypassed FDA regulations by traveling outside the U.S. (to Colombia) to conduct this experiment on herself.

Telomerase treatment to reverse aging is very promising, and it might really work, someday. I sincerely hope it will.  For now, though, BioViva's announcement leaves me very skeptical.

Legalized pot is already saving $165 million per year in medical costs

This news will give anti-marijuana crusaders fits.

A new study in the journal Health Affairs looked in great detail at prescription drug usage in U.S. states that have legalized medical marijuana. Researchers Ashley Bradford and David Bradford collected Medicare data for the period 2010-2013 to answer two questions: are patients choosing marijuana instead of prescription drugs for conditions that marijuana might treat, and what has been the overall effect on Medicare spending?

The bottom line: in 2013 alone, when 17 states had legalized medical marijuana, Medicare saved over $165 million. A simple extrapolation suggests that if all states legalize marijuana, annual savings could be triple that amount, $500 million. (Obviously this extrapolation is over-simplified: it depends on the actual populations of those 17 states, and it only considers Medicare. Commercial healthcare plans may save far more.)

The authors, a father-daughter team at the University of Georgia, looked at over 87 million prescriptions from the Medicare Part D database, focusing only on conditions where marijuana "might serve as an alternative treatment." These fall into nine specific categories for which at least some evidence suggests marijuana could help: anxiety, depression, glaucoma, nausea, pain, psychosis, seizures, sleep disorders and spasticity. The best clinical evidence of marijuana's benefits, according to the study, is for pain, for which the clinical evidence is "moderate"; most of the other conditions have evidence rated as low or very low.

Prescriptions fell for 8 out of 9 categories, with the biggest drop occurring for pain medication: in states with medical marijuana laws, physicians gave out 3645 fewer pain prescriptions per doctor, a difference that is highly statistically significant. Or, in raw numbers, the annual number of daily doses (in 2010-2013) per doctor in states without medical marijuana laws was 31,810. In states with medical marijuana, the number dropped to 28,165. That's an 11.5% drop. Depression and seizures both showed very significant reductions in prescriptions as well.

Prescriptions rose for only one condition: glaucoma. This too was expected–as the article explains,
"Clinical evidence is very strong that while marijuana sharply reduces intraocular pressure, the effect lasts only about an hour. As a result, new patients who seek glaucoma treatment after learning about the potential benefits of marijuana are likely to receive a prescription for an FDA-approved drug."
In an interview posted on the University of Georgia website, study co-author David Bradford states that they also wanted to know if medical marijuana laws were just a backdoor way to allow recreational marijuana use, or if people were really using it as medicine. He continues:
"What our evidence is suggesting is that ... there is a significant amount of actual clinical use at work here."
As of today, 24 states have medical marijuana laws on the books. This new study suggests that legalizing marijuana in all states would save hundreds of millions of dollars in Medicare costs, and even more money for people not covered by Medicare–not to mention the savings from no longer wasting law enforcement resources prosecuting marijuana use.

Opponents of legalization have argued since the 1930's that marijuana is a "gateway drug" that will lead to more serious drug abuse and that it increases criminal behavior, without (as Bradford and Bradford point out) any good evidence proving these links. This new study now provides a strong financial argument for legalization: legalizing marijuana, at least for medical use, will yield substantial savings for our health care system.

Nobelists call for Greenpeace to drop its anti-science, anti-GMO activism

Golden rice is fortified with vitamin A to prevent blindness.
Has Greenpeace lost its way? I still remember my excitement about the Save the Whales campaign when I was in college, one of the first and most visible of Greenpeace's campaigns. These were the good guys.

In recent years, though, they have adopted as one of their causes a rigidly inflexible opposition to all genetically modified foods, a stance that has no basis in science and that threatens to block technology that has great potential for good.

This past Thursday, a group of 110 Nobel Laureates released a letter excoriating Greenpeace for its long-term campaign against genetically modified organisms (GMOs) and especially against “golden rice.” The Nobelists' letter points out that science has shown that GM foods are just as safe and healthy as any other foods, and that Golden Rice has the potential to relieve a great deal of human suffering.

Unfortunately, anti-GMO activists at Greenpeace and elsewhere are ignoring the science. Their opposition seems to rely on two arguments, both of them superficially appealing, but both wrong.

First, there's the argument that "natural" is always better, whether it be applied to your food or to other aspects of life. Many people find this idea very compelling; after all, we humans are part of nature, so why not consume foods in their natural state? This argument is wrong for many, many reasons, far too many to list here, but I'll just mention a couple. We can start with cooking: it sure isn't natural, but cooking allows us to extract far more nutrients from our food, and is one of humankind's greatest inventions. Or consider pasteurization, an unnatural process that has saved countless millions of people from death by killing the bacteria that are present in purely "natural" milk and other products.

Suffice it to say that there's nothing wrong with modifying our food to make it easier to digest, healthier (as with Golden Rice), or just tastier. The fact that some genetic modifications fail to do any of these things doesn't make GMOs bad, it just means that GM technology can be applied in other ways.

The other argument against GMOs, perhaps the more emotional one, is that they're just a stealth method to allow big agricultural corporations to sell more herbicides and pesticides. This isn't exactly wrong: Monsanto's RoundUp Ready® crops are engineered to allow farmers to use more of the herbicide glyphosphate. Regardless of the arguments about herbicides, the fundamental problem with this argument is that it's a gross over-generalization: just because you don't like RoundUp Ready® soybeans doesn't mean that all GMOs should be banned. As the National Academy of Sciences concluded in a report published earlier this year,
"it is the product, not the process, that should be regulated."
Greenpeace, pay attention. That same NAS report also concluded that GM foods are generally safe and just as healthy as non-GM foods.

Now consider Golden Rice, a variety of rice that has been genetically modified so that it naturally produces beta carotene, which humans metabolize to produce vitamin A. Golden Rice has the potential to reduce vitamin A deficiency, which has devastating effects in parts of the world where children struggle to get enough nutrition. As the Nobel Laureates' letter points out, vitamin A deficiency affects 250 million people worldwide, and
"Vitamin A deficiency is the leading cause of childhood blindness globally affecting 250,000 - 500,000 children each year. Half die within 12 months of losing their eyesight."
How could Greenpeace oppose something that could eliminate so much suffering? I can only conclude that the anti-GMO forces within Greenpeace are so consumed by their rigid opposition that they simply cannot see that GMO technology has the potential for great benefits.

GMWatch, an anti-GMO organizationpublished a lengthy response to the Nobelists' letter the day after the letter appeared. Their rebuttal contains two arguments: first, that Golden Rice isn't yet ready for widespread distribution (Greenpeace's response, printed in The Washington Post, made the same argument, saying "we are talking about something that doesn’t even exist"); and second, that the Nobel Laureates don't have the "relevant expertise."

Hmm. Neither of these arguments stands up to even a tiny bit of scrutiny. First, even if Golden Rice isn't ready for prime time, it's still a great idea. Would Greenpeace support Golden Rice if it were ready to be shipped to hungry children today? They don't say. And both GMWatch and Greenpeace fail to mention (or deny) the fact that one of the major reasons that Golden Rice isn't yet on the market is that anti-GMO groups have worked very hard to block it, lobbying hard for regulatory barriers and even ripping up a test field.

Second, the argument about relevant expertise is ridiculous. Sir Richard Roberts and Dr. Phillip Sharp, two of the Nobel Laureates who spearheaded the effort to write the letter, are among the world's leading geneticists and molecular biologists, as are many of the other Nobelists who signed the letter. I also have to point out that this is a classic ad hominem attack: rather than address the actual topic, GMWatch are attacking the messengers. (Also, somewhat bizarrely, GMWatch contradicts its own argument in an update they posted to their own article. In the update, they write that "[Dr. Phillip] Sharp is a biotech entrepreneur with interests in GMO research," essentially acknowledging that he is an expert in GMO technology.)

Ad hominem attacks may be entertaining, but they fail to support the Greenpeace argument that GMOs should be banned. Nowhere in GMWatch's article, or in Greenpeace's anti-GMO policy, is there any comment about the extensive scientific evidence that shows that GM foods are safe. As I've written before, you're far more likely to be harmed by being hit on the head by a corn cob than by some kind of deviant GMO corn gene.

I asked Sir Richard Roberts if he had any response to the arguments from Greenpeace, and he replied that:
“Greenpeace just reiterate the old arguments that are adequately debunked elsewhere. Why won't they just admit they got this issue wrong? Is it because they have consistently introduced roadblocks and then wonder why it is taking so long to introduce Golden Rice to the market? Are they serious?”
Greenpeace would do well to reconsider their position, as Dr. Roberts and his colleagues argue. They are flat wrong on the science of GMOs, and their dogmaticism is losing them the support of many scientists (and others) who are strong backers of Greenpeace's other causes. For example, Greenpeace's website features their "Save the heart of the Amazon" campaign, an admirable effort to protect a large swath of the Brazilian rainforest. They also have a major campaign to save the Arctic, a cause I support even more enthusiastically.

One last note: organic food stores, led (in the U.S.) by Whole Foods, have been eagerly promoting the "non-genetically modified" nature of their foods, and pushing for laws to require that all GM foods be labelled. First, I have to point out that this is nonsense. Virtually everything you eat has had its genes modified from their natural state, through centuries of breeding by farmers. The only difference with modern GM food is that we can precisely select the genetic changes we want, unlike the slow, incredibly inefficient methods of traditional agriculture.

Ironically (and this is delicious in more ways than one), Whole Foods does sell massive quantities of one GMO: sweet potatoes. It turns out that all sweet potatoes contain bacterial genes! As Tina Kyndt, Dora Quispe and colleagues reported last year, 291 different varieties of sweet potato all contain genes from a bacterium called Agrobacterium tumefaciens. This bacterium has the ability to insert bits of its own genome into its host, and it did exactly that to sweet potatoes sometime in the recent past, after humans started cultivating sweet potatoes (wild relatives don't have the foreign genes). Apparently, ancient human farmers preferred the sweet potatoes with the bacterial genes, and these were passed on to all modern varieties. So sweet potatoes are not only genetically modified, but they are transgenic: they contains genes from a completely different species.

I was at Whole Foods today, and I failed to notice any labels revealing that their sweet potatoes are transgenic. I'll keep checking.

[Full disclosure: I have been privileged to have collaborated scientifically with Sir Richard Roberts, one of the leaders of the group of Nobel Laureates who authored the letter on GMOs. I was not involved in the writing of that letter and I was unaware of it until it was published.]

Zika virus poses a greater threat than we thought

The Zika virus outbreak in South America has caused thousands of cases of microcephaly, where an infant is born with an unusually small head and brain. The threat is so serious that the CDC has issued a level 2 alert for anyone attending the Olympic Games in Brazil this summer. The World Health Organization has also issued travel precautions.

Most of the attention has focused on microcephaly, understandably so, but Zika threatens more than just pregnant women. In recent months, the evidence has been building that Zika also causes Guillain-Barré Syndrome (GBS).

GBS is a rare but terrifying disease, in which your own immune system attacks your nerve cells, leading to rapid paralysis and, in some cases, death. With the best available modern care, the death rate is about 5%, but it's much greater when patients cannot get high-level care.

The first report of Zika as a cause of GBS appeared earlier this year in The Lancet, in a study of a 2013-14 Zika outbreak in French Polynesia (Tahiti). 42 patients were identified with Guillain-Barré syndrome (a startlingly high number for this otherwise rare disease), and 41 of them tested positive for Zika. None of those patients died, but the study provided convincing evidence that Zika was the cause of GBS.

This year, reports have emerged of a sharp increase in the number of cases of Guillain-Barré in South America, where the Zika outbreak is most severe. As the Washington Post reported back in February, the small town of Turbo, in Colombia, which normally sees at most one case per year, has already seen five cases, three of them fatal.

In response to this threat, a group of more than 200 doctors from around the world has signed an open letter to the WHO (read it here) that
"call[s] for the Rio 2016 Games to be postponed and/or moved to another location—but not cancelled—in the name of public health."
The letter points out that Rio is at the epicenter of the Zika epidemic, with 32,000 cases so far, and the holding the games there–with all the associated travel involved–may accelerate the spread of the dangerous Brazilian strain.

The newly emerging risk of Guillain-Barré syndrome makes it clear that Zika virus presents a threat to everyone. The Rio Olympics are likely to make it worse. Anyone planning to visit Rio for the Games should take all the precautions they can, but the best plan might be simply to stay home.

I took the SAT so you don't have to. It's a very poor test of math skills.

As I write this, tens of thousands of high school students are hunched over desks, filling in little circles with number 2 pencils, laboring to complete the SAT, a test that will have an outsized impact on where they go to college. In my state, the test starts at the teenager-unfriendly hour of 8:00am and last a grueling four hours or more.

One of my daughters is among those students. and as preparation she took the four practice tests provided by the College Board–the private company that owns the tests. (The tests are administered by the Educational Testing Service.) There are two main parts to the SAT, a math test and a verbal test, and as every student knows, the scores range from 200 to 800 on each part. For many decades, this test has been one of the main gatekeepers to college: the US News College rankings use it, and colleges advertise the average SATs of their freshman classes. Every student wants to know what SAT score they need to get into their preferred university. 1.7 million high school students will take the SAT this year, and many of them will take it twice.

I wanted to understand what the test was like, so I took the math test with my daughter–three times. On three successive weekends, we each took one of the practice tests, and then used the answers provided by the College Board to score ourselves and review what we got wrong.

Here's what I learned from taking the SAT math test: it's all about speed. The concepts are not difficult; you need to know algebra, geometry, a little bit of trigonometry, and a tiny bit of statistics. The main skill you need, though, is speed. It's a very poor test of how well you understand math. For the three tests, I was only able to finish everything on time once. Even so, I had to work very quickly and I didn't have time to go back and check my answers.

Question 27 from SAT test 2 from the College
Board. "D" is the correct answer.
The math test has two parts: one with 20 questions, for which you get 25 minutes, and another with 38 questions, for which you get 55 minutes and where you're allowed to use a calculator. The test is designed to trip you up if you work too quickly: many of the multiple-choice answers match the answer you would get if you made a careless error of a particular type.

Doing well on the SAT requires that you know the tricks of the test, and that you've memorized many formulas so that they come to mind instantly. And I mean instantly: if you have to think for 30 seconds to remember something, that's far too long.

What's more, the questions themselves can be lengthy, and students might waste precious minutes just trying to be sure they understand the wording. For example, one question shown here filled half a page: just reading it would take some students longer than they can afford for this speed-obsessed test. (If you want to see a full-sized image, get the tests here.)

Statistics is a relatively new topic area for the SAT, and if the practice test is any guide, they haven't yet figured out how to construct good stat questions.  Here is one of them:

A researcher conducted a survey to determine whether people in a certain large town prefer watching sports on television to attending the sporting event. The researcher asked 117 people who visited a local restaurant on a Saturday, and 7 people refused to respond. Which of the following factors makes it least likely that a reliable conclusion can be drawn about the sports-watching preferences of all people in the town?
A)  Sample size
B)  Population size
C)  The number of people who refused to respond
D)  Where the survey was given 
The official correct answer is D, because (says the College Board) the survey was not collected from a random sample. However, I could argue that A is at least as good an answer, because 117 people is a tiny sample from what is called a "large town," and because we don't know that the people who visit this restaurant are un-representative of the town. Whether you think the answer is A or D, this is a lousy question to put on a test where the answers should be unambiguous.

You might be wondering what my score was. I'm not going to reveal that, but I will say that I had a higher score when I took the SAT in 1975, as a 15-year-old high school student. I must have been faster then, but I'm pretty certain that I understand math better now, after 35 years of working in a mathematical field. I suspect that the current test puts a greater emphasis on speed than the 1975 version, but there's no way to check that without copies of the 1970s-era SAT exams.

Let me put this another way: there's not a single question on any of the practice math SAT exams that I would call difficult. Most of them are quite easy if you know a bit of algebra and geometry. But unless you are fast, answering all 58 questions in 80 minutes is darn near impossible. For example, consider this simple problem:

At a lunch stand, each hamburger has 50 more calories than each order of fries. If 2 hamburgers and 3 orders of fries have a total of 1700 calories, how many calories does a hamburger have? 
This question is not multiple choice; you have to write down a number, which means you have to work it through. Any student who knows basic algebra should be able to solve this, but can s/he do it in less than 75 seconds? And if takes 90 seconds, does that mean s/he should be rejected by Yale?

As a measure of true understanding, the SAT math test is terrible. We should not be using it to make enormously consequential decisions about where almost every high school senior in the country goes to college. On a positive note, a growing number of colleges have rebelled and no longer require the SAT. Two years ago, a large study showed that, at colleges in this group, there was no difference in college performance between students who submitted SAT scores and those who didn't.

Perhaps pressure from colleges that are making the SAT optional will force the College Board to create a math test that measures something that matters for college success. They could go a long way towards a better test by simply giving students twice as much time. If that made the test too long, they could simply ask fewer questions. Meanwhile, I hope that more colleges will make the SAT optional, and instead use high school grades and other, more meaningful measures of a student's knowledge.

(*This was written on June 4, 2016, the date of the second SAT test in the "new" format. Prior to March 2016, the test had three parts, each with a maximum score of 800.)