Bad Science and Confirmation Bias

Good honest science is first approached by collecting observational data, analyzing for certain relationships, and finally forming hypotheses around the data. But that's just the beginning. The good scientist then does his level best to disprove the hypotheses suggested by the data. To the extent he tries and fails, the hypotheses gain more acceptance. Then, other good scientists set about to disprove, and to the extent they try and fail, the hypotheses become to be "established science." The underlying methodology is one of falsifiability in scientific propositions. Put another way, a scientific proposition must be formed in a way that methodologically makes room for a negative result. Here's my favorite illustration, from the late Carl Sagan in his book, The Demon Haunted World. "A Fire-Breathing Dragon Lives in My Garage."

Now, what’s the difference between an invisible, incorporeal, floating dragon who spits heatless fire and no dragon at all? If there’s no way to disprove my contention, no conceivable experiment that would count against it, what does it mean to say that my dragon exists? Your inability to invalidate my hypothesis is not at all the same thing as proving it true. Claims that cannot be tested, assertions immune to disproof are veridically worthless, whatever value they may have in inspiring us or in exciting our sense of wonder.

If you pay attention, that's how it almost never goes in the world of diet studies, intervention trials, and the like. Most everything is designed to attempt to confirm certain hypotheses, particularly about supposed links between dietary fat, cholesterol, and heart disease. How it goes is that if even the most tenuous link in the desired results can be stated, that's what gets stated. Other potential — even more significant and life threatening – relationships are often ignored, and if the sought after relationship doesn't exist at all, or is even suggested to be disproved, then it's chalked up to an aberration and the press releases often don't find their way out of the organization. Dr. Briffa has a recent post that illustrates.

While studies do indeed show that raised cholesterol levels are associated with an increased risk of heart disease, it does make sense to take a wider view if at all possible. Recently I have reported on a couple of studies which link lower cholesterol levels with an increased risk of cancer. One of these studies found lower cholesterol levels were associated with an increased risk of death too. You can read about there studies here and here. The bottom line is that focusing only on heart disease can give us a somewhat skewed impression of the association between cholesterol and health.

[…] However, lower cholesterol levels were found to be associated with an increased risk of bleeding into the brain (intracranial haemhorrage). Here are some other findings from this study: “A serum cholesterol level less than 4.14 mmol/L (less than 160 mg/dL) was also associated with a significantly increased risk of death from cancer of the liver and pancreas; digestive diseases, particularly hepatic cirrhosis; suicide; and alcohol dependence syndrome. In addition, significant inverse graded associations were found between serum cholesterol level and cancers of the lung, lymphatic, and hematopoietic systems, and chronic obstructive pulmonary disease.”

So, according to this data, the low cholesterol levels are associated with increased risk of, among other things, several forms of cancer. The authors of this week’s JAMA article were also authors of this study revealing the apparent hazards of low cholesterol. But this week’s JAMA article makes no mention of these findings.

So, how low of a cholesterol do you want, and why in the world, with something so complex, would anyone want to just look at a single health consequence? The bottom line:

The result? The interventions did not bring a statistically significant reduction in risk of death from heart disease nor overall risk of death either. This dismal result was not mentioned in this week’s JAMA paper either. We can’t judge from this study whether taking dietary steps to reduce cholesterol is worthwhile or not, because of multifactorial nature of this trial. However, one thing is for certain: the results of the original MR FIT study in no way support the notion that reducing fat and cholesterol is beneficial to health.

Now let's turn to another aspect of this same tired old behavior. This one is from Doc Eades (again). This goes back a few years, but is essentially quite a joke. Some researchers get together, including one who promotes his own obscure, "high fiber diet," then they plug macro and micronutrient data into a computer program they damn well knew in advance would spit out the usual conclusions, so they could then "confirm" the soundness of the sorts of diets they recommend in the first place.

When these authors looked at saturated fat, evil incarnate in the minds of the lipophobes, they found both Atkins and Protein Power to be full of the stuff. Especially Atkins. And we all know that saturated fat increases cholesterol, don’t we? Well, don’t we? Maybe the readers of this blog don’t know that. But the authors of this study along with fellow lipophobes Mensink and Katan know it. The latter know it so well that they’ve created equations predicting how much dietary saturated fat will raise cholesterol. And if we look at the graph below, we can see just how high the cholesterol levels are predicted to be in the blood of the followers of the various diets.

Apparently cholesterol will be way up in followers of Protein Power and off the charts in those following Atkins. But, remember, these are just their predictions. They aren’t reality.

Picture 1

Indeed. Now, I'm not going to take the space to outline the procedures. You can get that from Eades' post. In brief, he secured the services of an independent statistician to analyze results from his own patients over years of observation on low-carb, high-fat diets. In pictures.

Picture 2

Of course, none of this is surprising to me, my wife, or others who've recently had blood work done.


  1. Edwin on September 22, 2008 at 07:10

    Hi, thanks for linking to my blog.

  2. Nancy R. on September 22, 2008 at 10:52

    This is the kind of information people need to see. It's great that you're aggregating it and distilling it so that people can understand. I remember reading "The Great Cholesterol Con" by Dr. Malcolm Kendrick and being amazed by how much data is out there showing that all cause mortality actually increases as cholesterol drops in various populations. But we never hear about that in the media. Or from our physicians. It's all about heart disease and total cholesterol. Infuriating, to be sure. Keep on rocking.

Leave a Comment

You must be logged in to post a comment.