- After the original report, Pevzner (privately) pointed out statistical problems, and the authors revised their findings, admitting in a letter to Science in September 2007 that one of the peptide fragments was a statistical artifact. One down, six remaining.
- In January 2008, Science published a new Technical Comment in which 27 authors (Buckley et al.) used a standard set of authentication tests developed for ancient DNA, and reported that the T. rex sample failed those tests. (Another sample from mastodon, also reported by Asara but 100 times younger, passed the same tests.)
- In July 2008, Thomas Kaye and colleagues published a report that re-examined the microscope evidence of the T. rex "soft tissue". The original findings by Schweitzer were based on this soft "tissue" being original T. rex organic material. Kaye et al. report that the soft material was a bacterial biofilm - not original material at all. They also report on carbon dating of the biofilm showing it to be modern, not ancient.
- Pevzner et al. report this week that the peptide mass spectrometry evidence - which Asara and Schweitzer repeatedly used to defend their results against the earlier criticism - are also flawed. One way to resolve this, Pevzner points out, is to release the mass spec data, which is a common practice in that community. This would allow others to re-interpret the data and test more rigorously for statistical artifacts. However, Asara and Schweitzer refuse to release their data. Instead, they wrote another response which simply gives more details about how they ran the software to search their spectra against a peptide database, but doesn't really answer Pevzner's questions.
If Science truly cared about getting this story right, they would publish the critiques just as prominently as the original article. It seems that Science is eager to get publicity for a "discovery", but not so eager for publicity when it turns out the discovery is false. Yes, it's true that they did publish the critiques, but they should have done better.
Finally, I recommend Rex Dalton's story in Nature on this controversy, which does a good job of summing it up, with links to all the articles. We'll see what happens next, but it appears that Schweitzer and Asara will keep defending their claims. The mounting evidence seems to show that they were wrong - wrong about the soft tissue, wrong about the mass spec identifications, wrong about the age of the sample, and wrong to continue to refuse to release their data.
Why is Science not being taken to the cleaners for not releasing the raw data? (as a naive recent grad) I thought that's mandatory for publication. You agree to make stuff public.
ReplyDeleteIf Science truly cared about getting this story right, they would publish the critiques just as prominently as the original article. It seems that Science is eager to get publicity for a "discovery", but not so eager for publicity when it turns out the discovery is false. Yes, it's true that they did publish the critiques, but they should have done better.
ReplyDeleteYou would think that you would save more 'face' by being open and reputable at all times. Supporting science for science instead of trying to cower and hide any mistakes.
thanks for the update! and btw... tag!
From my experience I would say this the rule rather than the exception these days. IMHO, the popularity of high throughput technology and lack of training in programming and statistics among users of these technologies has made this so. Releasing raw data should be mandatory but its not (microarrays studies are an exception). Furthermore, most statistical analyses are done using point and click software. So most published biology papers are not reproducible in two fronts.
ReplyDeleteWhen you do manage to get raw data it is still a few days of work and emails to the authors to figure things out. Here is a good example of paper about this: Kevin R Coombes Jing Wang & Keith A Baggerly. Nature Medicine 13, 1276 - 1277 (2007)
This has prompted me to start asking for data as a referee. In my first experience with one of the big name journals I found an artifact 30 minutes into the analysis: a microarray batch effect. Outcome was confounded with month of the year. No way this get caught otherwise. But how many papers get check this way?
Something needs to be done.
I have published a paper based on these T. rex peptides. "Ancient Fossil Specimens of Extinct Species Are Genetically More Distant to an Outgroup than Extant Sister Species Are"
ReplyDeleteRev. Biol. 2008, 101: 93-108
see http://precedings.nature.com/documents/1676/version/2
I hope the sequences are right since the my result with them makes sense. I dont think the verdict is clear that they are wrong. Even if they are, my result still stands for neandertals and mastodons. My general conclusion would still be true. Furthermore, it has just now been confirmed by the recent Cell paper on Neandertal complete mtDNA.
Since your blog is in part about pseudoscience, I would like to point out a pseudoscience that is unfortunately commonly practiced by professional scientists of certain field. A general pattern is like this. First, scientists make a factual observation A. Second, they derive from A hypothesis B to explain A. Third, they make and test predictions from B to see if it is right. Unfortunately, they find that none of the predictions tested positive. Finally, instead of giving up on B, they still assume that B is correct, citing that A supports B.
ReplyDeleteI hope you can recognize this behavior as pseudoscience and can identify the scientific field in which this behavior has been going on for years and name one popular hypothesis that is like B.
Shi, a quick response to your comment: first, I want to point out to other readers of this blog that Nature Precedings, where you posted your paper, is a forum that Nature provides for posting results prior to peer review. So when you write "published", you should qualify that by saying your paper was not peer-reviewed.
ReplyDeleteSecond, your analysis is based on just 1 of the 7 "T. rex" protein fragments, and it's only 15 amino acids long. You based your discussion in your paper on a single amino acid change - hardly enough to make any strong conclusions. But since many of us don't believe these protein fragments came from T. rex in the first place, your conclusion cannot be supported by this data!
When I say published, I did mean in a peer reviewed journal. If you had read my original post here carefully, you would find the citation: Rev. Biol. 2008, 101: 93-108.
ReplyDeleteMy result was first made public in 2007 in SMBE meeting in Canada. I then posted it at Nature Precedings. It has since been published in a peer reviewed journal after more then a year of slow review. The journal Rev Biol is not Cell or Science but you dont expect to see a truly anti-dogma paper to be in Cell or Science. It is so much more fun to see a Cell paper confirming a paper in obscure journal, which is what happened in this case.
if the T. rex peptide is indeed wrong, it would not affect my conclusion a bit since it is still supported by neandertals and mastodons and independently by the recent Cell paper on neandertals.
If the T rex peptide is true, then 1 aa change is still a big deal because that position is 100% conserved in all extant amphibians, birds, and mammals analyzed (9 species in total).
I am puzzled by your certainty that the T rex data is wrong. How do you explain its close relationship to chicken? Do you imply that these authors committed the worst sin as scientists? Perhaps they are biased and are looking for what they want to find. Assuming they are not, it would be very strange for them to find 7 chicken-related peptides all by chance. Or their sample was contaminated by chicken sandwiches. Highly unlikely.
Gnomon writes "I am puzzled by your certainty that the T rex data is wrong.... it would be very strange for them to find 7 chicken-related peptides all by chance."
ReplyDeleteIn fact, this is precisely the point that Pevzner tried to explain in his Technical Comment. It would not be at all strange to find 7 chicken-related peptide fragments (short fragments, not whole proteins) if you were looking for them, and if you had a large number of spectra. This happens all the time in mass spec experiments, as Pevzner explained - they generate tens of thousands of spectra (or more), and false positives are a very real problem. So when you write "highly unlikely", I'm afraid you're mistaken.
I never said I was "certain" that Asara et al.'s claims are wrong, but the only way for Asara et al to back up their case is to release their data (all the spectra generated in their experiment), which they are refusing to do. This makes their claims more suspect, to my mind.
By the way, I can't find any reference to "Rev. Biol." in PubMed, nor in Google searches.
Hi All,
ReplyDeleteJust want you to be aware that we will be releasing the entire T. rex MS/MS dataset on the PRIDE proteomics repository at EMBL. The dataset contains 48,216 spectra and is the same dataset that was used in the recent response to Pevzner et. al. that appeared in last week's Science. The peptides in the database that that we know are contaminants from our lab are human keratin, BSA, IgG and casein, so you can ignore those. Besides the collagen peptides (which do not show up as contaminants in dry runs) we see a few, but not that many bacterial
proteins. Not surprisingly we see a lot of masses from these rock-like structures that are almost certainly not peptidic and probably not even organic. Their fragmentation patterns match nothing in any of the forward or reverse protein data bases and the
mass separations of the fragments aren't consistent with amino acids.
Hope you enjoy the data and let me know if you have any questions. Hopefully, this will convince you that we are not hiding anything as we tend to only believe protein data that we can verify biochemically through synthetic peptide comparisons of MS/MS data and evidence with antibodies.
Regards,
John
Excellent news from John Asara! This is what I (and others) have been asking.
ReplyDeleteI expect it will take a little time for others to analyze the data - once they appear - and I'm looking forward to seeing the results.
steven salzberg said:
ReplyDelete"By the way, I can't find any reference to "Rev. Biol." in PubMed, nor in Google searches."
click
Rev. Biol. 2008, 101: 93-108 for the website of Riv Biol.
If you do keyword search 'neanderthal' (but not neandertal) at PubMed of NCBI home page, you will find my paper listed as item number 2.
Riv Biol is a 90 year old Italian journal of theoretical biology.
Just realized a mistake. The key word search of PubMed must be 'neanderthals' in order to retrieve my paper.
ReplyDeleteIt will not work if you use 'neanderthal' or 'neandertal' or 'neandertals'.
Where's pevzner now? John Asara released his data, but I don't seen anything from Pevzner. Maybe the analysis wasn't so bad afterall?
ReplyDeleteDon't be naive. This is complex data and it takes time to analyze it properly. It has only been a few weeks.
ReplyDeleteNow thanksgiving. The data has been released for 3 months...no word from Pevzner.
ReplyDeleteGnomon/Shi Huang:
ReplyDelete"Riv Biol is a 90 year old Italian journal of theoretical biology."
Indeed, and I guess you appreciate the fellowship of other pseudoscientists like JA Davison, J. Wells ang G. Sermonti.
I love to talk about pseudoscience. And I have solid proof of pseudoscience perpetuated by famous Darwin followers today.
ReplyDeleteI below prove that Jerry Coyne cheated in his book 'Why Evolution is True.' He turned evidence against NeoDarwinism into something of the exact opposite.
The purpose of Coyne latest book is to explain to laymen why evolution theory, which means NeoDarwinism at least to a reader of this book, is true. Of course, the modern evolution theory consists of two opposite sub-theories, NeoDarwinism/natural selection and the neutral theory. NeoDarwinism or natural selection is largely irrelevant to molecular evolution, or, more precisely, contradicted by molecular data. As a result, a theory based on the negation of NeoDarwinism or natural selection, the neutral theory, is used to explain molecular evolution, in particular the molecular clock.
But the only theory the book ever talks about is NeoDarwinism or natural selection. There are very few sentences that mention molecular evolution. And these in fact mislead the readers into believing that NeoDarwinism is supported rather than contradicted by facts of molecular evolution. Here is what Coyne wrote: "Evolution theory predicts, and data support, the notion that as species diverge from their common ancestors, their DNA sequence change in roughly a straight-line fashion with time. We can use this "molecular clock", calibrated with fossil ancestors of living species, to estimate the divergence time of species that have poor fossil record."
Does Coyne really expect the lay readers to know that the `evolution theory' here means the neutral theory, when it is never mentioned in the book and must negate the key idea of Darwin? If the lay readers, after reading this, then believe incorrectly that NeoDarwinism predicts the molecular clock, is it the readers' fault or the author's?
When cheating is employed to explain why you are right, it merely shows you are wrong. And cheating is pseudoscience, would not you agree?
If you represent genuine science, prove that you do by simply say in your next post here that Coyne cheated in his book.
Since God is still an open question, it is certainly possible for God to be proven one day. When that happens, are we going to say that mainstream journals only published pseudoscience and Riv Bio represents the real science. No I would not. The quality and data of the paper should speak for itself not where it is published.
My paper in Riv Bio reported a striking observation that was later confirmed by a Cell paper (Green et al, Cell. 2008 Aug 8;134(3):416-26.). So, here in my case, Riv Bio beats Cell handily.
In any event, prove you are honest first before calling others pseudoscience. Do you dare to say that Coyne cheated in your next post here? come on you have nothing to lose here since no one knows who you are.
Gnomon: you don't seem to understand what the word "proof" means, nor how to construct a logically valid proof. Your post is filled with logical fallacies - e.g., you assert that "when cheating is employed to explain why you are right, it merely shows you are wrong." Uh, no. When someone "cheats" (e.g., suppose they misinterpret data), then their argument is invalid. But their conclusions could still be wrong. E.g., if I assert that "A implies B", and it turns out that A is false, we don't know anything about B - it could be either true or false.
ReplyDeleteBut I doubt you're likely to understand or agree with logic, since you assert that "God is still an open question." Right. And so is the tooth fairy. And leprechauns.
Okay, I dont know what proof means, and you know better. Now show me your proof that there is no God.
ReplyDeleteIf you dont have any, to my knowledge no has, why is God not an open question?
There are literally an infinite number of propositions that have no proof. In general, proving that something doesn't exist is not possible. You can't prove that there is no tooth fairy. Is the tooth fairy's existence an open question, then?
ReplyDeleteSpeaking logically (and scientifically), we require evidence - verifiable events, facts that other people can check out independently - before it's even worth investigating a phenomenon. Superstitious beliefs in supernatural beings are not "open questions," at least not in any rational sense.
Until then, my belief in the Flying Spaghetti Monster is just as valid as any other god.
Below is purely a logical exercise so no position is taken for granted without reasons.
ReplyDeleteSo, you position of no God is just as weak as someone's yes God. It is only fair to call it an open question, then. The tooth fairy is too trivial to be called a question. But, the God question is the most fundamental that humanity is capable of asking. If you cannot ever prove your no God position, it does not follow that the yes God position shall also be un-provable. Of course we are not here talking about any religious personal God. Only a law abiding God is provable. A God who arbitrarily creates miracles is just as random as a dice tossing God and both are unprovable by nature. BTW, your no God position is in fact equivalent to a yes God position where the God happens to work by tossing a dice. So, you do believe a God, a random God. On the God question, there are really only two views, a random unprovable God versus a law-abiding provable God. I seriously doubt that the best minds of humanity would ever settle for a random God. It is simply insulting to human rationality. It is a lazy answer. That question can only be solved by hard work like any other hard questions.
The very best thinkers in history (scientists and philosophers) who believe in God vastly out number those who do not. It is indeed a curious phenomenon that while average scientists (including most NAS members and Nobel laureates and those who aspire to be among them) mostly do not believe in God, the very top notch like a Newton or Einstein mostly do. This would make sense if it must take the very best mind to appreciate God in a reasoned way. Indeed, you don’t expect the likes of Chimpanzees to come up with reasons for God. Similarly, you don’t expect the average minds or poor minds to prove God or to intelligently address the question of God. It would surely take the best mind of humanity to prove God. There cannot be more than one unique mind to do so independently because that would make the question a trivial one. On this question, the opinion of a million average scientists counts far less than the brilliant mind of a single man.
For routine questions of science, the assumption of no God works well. But God may not bother with daily routine stuffs of man. If it only acts directly once in some million years, it would be completely consistent with everything science knows. The fact you don’t see God daily may be precisely the way it works. Just picture this, a random God may never be able to give you a beautiful garden in your backyard. And that, to a reasoned man, is sufficient evidence to justify looking for the law abiding God.
Asara and colleagues just replicated their earlier study in another fossil. Where's Pevzner now? His tone was way out of line, but I bet he won't ever issue an apology
ReplyDeleteActually, Pevzner and colleagues are working on a response, and they are analyzing the new data as well. I've been in touch with him and others, and I hope to post something on this blog soon.
ReplyDeleteI'm skeptical of the new findings, but it will take some time and careful analysis to look at them.
when you speak to pevzner, explain to him that while many people think he may be correct in his criticism, his tone in his letter was insulting and turned many people off to his cause.
ReplyDeletePevzner is correct, Jon asara and colleagues need to withdraw these papers as they are inherently flawed; they were/are young/junior colleagues wanting to get established with a frequently in absentia boss, Lou Cantley; however, it severely backfired. They refused to submit the raw data originally and only made it public after the fact, hence damaging their credibility. they need to formally withdraw their papers and apologize publically for their overzealous, misinterpretation of the facts. This has hurt/damaged permenantly Jon's career as a credible scientist as his earlier denial, defensive posturing, and arrogance has cost him much support and empathy.
ReplyDeleteJust recently, John Asara and colleagues have made further statements defending their results - and they have published another paper with data from a separate fossil - slightly older than T. rex - claiming they found collagen in that one too. See my more recent post on this subject.
ReplyDelete