March 13, 2010
Vast majority of published research claims may be false
Tom Siegfried in Science News:
It’s science’s dirtiest secret: The “scientific method” of testing hypotheses by statistical analysis stands on a flimsy foundation. Statistical tests are supposed to guide scientists in judging whether an experimental result reflects some real effect or is merely a random fluke, but the standard methods mix mutually inconsistent philosophies and offer no meaningful basis for making such decisions. Even when performed correctly, statistical tests are widely misunderstood and frequently misinterpreted. As a result, countless conclusions in the scientific literature are erroneous, and tests of medical dangers or treatments are often contradictory and confusing.
Replicating a result helps establish its validity more securely, but the common tactic of combining numerous studies into one analysis, while sound in principle, is seldom conducted properly in practice.
Experts in the math of probability and statistics are well aware of these problems and have for decades expressed concern about them in major journals. Over the years, hundreds of published papers have warned that science’s love affair with statistics has spawned countless illegitimate findings. In fact, if you believe what you read in the scientific literature, you shouldn’t believe what you read in the scientific literature.
“There is increasing concern,” declared epidemiologist John Ioannidis in a highly cited 2005 paper in PLoS Medicine, “that in modern research, false findings may be the majority or even the vast majority of published research claims.”
More here.
Posted by Abbas Raza at 03:40 AM | Permalink
Comments
The take-away sentence: "any single scientific study alone is quite likely to be incorrect, thanks largely to the fact that the standard statistical system for drawing conclusions is, in essence, illogical"
How many stories in the NYT and elsewhere fall afoul this fundamental truth: the latest prediction of rising sea levels, a promising new cure for cancer, how to fix inner-city schools or close the racial learning gap? One wonders if the major news organizations could attract an audience without these kinds of stories. Certainly their editors don't think so.
Posted by: Luke Lea | Mar 13, 2010 10:28:29 AM
What a silly article. They act like the scientific community is unaware of the ambiguities present in statistical interpretation (it isn't), like every significant test at the p=0.05 level comes out right there (much of the time p values are much lower), like scientists don't know how to apply bonferroni corrections, etc. etc. The list of mistakes in reasoning, incorrect assertions about practices, and so on in this article is enormous.
Look, some scientists use statistics badly. Some scientists do science badly. Is that a surprise? But to say that "any scientific study" is likely to be wrong is ludicrous. My impression is that the author is primarily worried about a certain subset of studies - big, medical/genetic studies - because most of his criticisms are irrelevant to a lot of stuff.
Posted by: Adam | Mar 13, 2010 11:34:49 AM
Agree with Adam. Scientists are well aware of the interpretative problems associated with standard statistical hypothesis tests.
The problems of data interpretation are deep, and go beyond the statistical issues the author mentions. Real science requires real thinking: assessing patterns of results across different experiments, preferably from different labs and different methods. It requires common sense. It also requires a certain amount of self-knowledge -- you have to keep track of your own motivations and of what you "want" to be true.
Some scientists don't succeed in doing this difficult thing. How many fail? I don't know....a minority.
What bothers me about the article is that to many readers it serves as a characterization of science as an entereprise that is blinkered and in thrall to funding agencies and ideology. This is (largely) not the case. Most scientists, especially in biomedical and biotech fields, are trying to solve problems that are real & important & difficult, and are doing their best to determine the truth.
Posted by: CJH | Mar 13, 2010 12:26:08 PM
This article repeats the mistaken claim that Bayesian methodologies are necessarily wedded to a "subjectivist" interpretation of probability, because of the use of priors:
"But Bayesian methods introduce a confusion into the actual meaning of the mathematical concept of “probability” in the real world. Standard or “frequentist” statistics treat probabilities as objective realities; Bayesians treat probabilities as “degrees of belief” based in part on a personal assessment or subjective decision about what to include in the calculation."
Firstly, this "confusion" (the distinction between objective frequencies and degrees of belief) has been around since people started thinking about probability. See Ian Hacking's classic book 'The Emergence of Probability'.
Secondly, use of Bayesian inferential techniques need not commit you to a subjectivist philosophy of probability. So-called "objective" Bayesians eschew the use of "informative" priors---they say you should start with completely uninformative priors, i.e. something like the uniform distribution.
Posted by: DF | Mar 14, 2010 1:02:43 PM
The trouble with statistics is that it is so easy to be fooled by them and so easy to use or misuse them to fool others with them. Unlikely events happen every day and likely events fail every day. If you had a better than 90% chance of winning a $100 million dollars would you bet your estate or life on it?
It is one thing to use the rules of statistical methods to seek to improve a production line as in statistical quality control or, as Feynman once did, seek to improve the odds at Blackjack tables, or discuss the general safety of commercial airplane flight as having a small probability of an accident.
But one must always keep common sense at the forefront; namely, that small unknown probability of success or failure that can throw a monkey wrench into any plan to break the bank.
But when it comes to involving risks to human life situations, a good article to read is Feynman’s Addendum to the Challenger Space Shuttle Disaster. He discusses how one group of mostly managers estimated the probability of failure at about one in ten thousand whereas the engineers estimated the same thing about one in one hundred.
Why the large difference in estimates? Because of prejudice and bias and political considerations of the managers vs. the engineers. After many pages discussing all the intricate details, Feynman concluded that they were basically playing a game of Russian Roulette with the lives of the crew. The engineering problems of the Space Shuttle as a research project, were not at all understood at the level of commercial airline flight. False premises and assumptions were made. Many sought to fool themselves.
Part of the difficulty with statistical analysis is that one always studies problems in certain closed and very limited systems with very definite idealized assumptions that may deviate enormously from real life situations. These may be intellectually stimulating problems to do but often lead one astray when it comes to making life and death decisions about a drug for a cancer patient or a particular type of surgery or radiation procedure etc. The devil is always in the details and assumptions.
For example, one is taught early on that if all the forces and initial conditions were known one could theoretically predict the winner of a horse race with certainty. Of course this is a meaningless claim since one never knows all the forces in almost any situation, even a simple one in the laboratory, let alone complex situations like a horse race. Hence the need for statistics right away.
Even with a two headed coin, is the probability of heads exactly 1? No, because one possible outcome would be if it landed on its edge. All possible outcomes are rarely included in any statistical analysis of anything. This is the central problem. Garbage in, garbage out. Or, as Mark Twain said, “There are lies, damn lies and statistics.”
And anyone who believes that medical research is immune from these problems, the exact opposite is usually true. Just read a few references like “The Cancer Industry” by Ralph W. Moss, Ph.D., Equinox Press, N.Y., 1996 first published at “The Cancer Syndrome” about 1980 or “The Truth About Hydrazine Sulfate-Dr. Gold Speaks” by Joseph Gold, M.D., at www.hydrazinesulfate.org. Dr. Moss, who has a Ph.D. degree in classics from Stanford University, was fired from Memorial Sloan Kettering Cancer Center in New York in the 1970’s for refusing to lie about medical “research” on Laetrile, and Dr. Gold has spent the greater part of his career seeking to have the drug hydrazine sulfate approved by the NIH, NCI and FDA, which has not only denied its approval but posted false information about it on a government website despite the fact that it has been shown effective in scientific tests from the Soviet Union to U.C.L.A.
All approved cancer treatments are life threatening. Therefore, if a cancer patient dies while under treatment, they could have died either from the cancer, the treatment or a combination thereof. But most of the time the deaths are reported as being due to the cancer rather than the treatment. But rarely an autopsy, let alone a true and objective autopsy is ever performed to determine the true cause of death. All this is arranged so as to protect the attending physician from criminal liability. Obviously no doctor wants to be placed in the position of having to admit that some drug or procedure they recommended and signed off on killed the patient do they? It is sort of like don’t ask, don’t tell. But over half a million cancer victims die every year either from cancer, treatment or both. Do you think every one of them only died of the cancer and not the treatment?
A very distinguished professor of medical physics, physiology and an expert in medical statistics, at UC Berkeley, one professor Dr. Hardin Jones, Ph.D. once carefully analysed the data on cancer victims. He found that those cancer patients who refused the orthodox treatments of surgery, chemotherapy and radiation lived up to four times longer than those who accepted those treatments. This study was in and of itself vivid proof of the failure of the war on cancer then. But of course the members of the orthodoxy ignored and disparaged his work and claimed he was wrong. The bottom line is that if the medical orthodoxy had been successful all these years, the cancer would be cured if they knew what they were doing. Obviously they do not know what they are doing.
They are engaged in the biggest scam on the planet.
I know of one very intelligent and honest medical doctor who admitted responsibility for the death of a patient. His name was Max Gerson, M.D. and he admitted this in a book called “A Cancer Therapy Results of Fifty Cases” by Max Gerson, MD., The Gerson Institute, Bonita, California, 1958, 5th edition, 1990. Dr. Gerson felt terrible about what he had done. He was one of the few medical doctors with a conscience.
Posted by: Winfield J. Abbe | Mar 16, 2010 12:19:02 AM
Post a comment