The science is working with bad data.
In 1945 the US Marine and Army forces fought an 82 day long battle against the Imperial Japanese army at Okinawa. This battle is one of the bloodiest in the Pacific and resulted in the death of roughly a third of the population. Widespread deliberate destruction of foodstuffs lead to a huge number of civilians struggling to return to a sufficient diet after the war. 1949 data from US national archives indicated 85% of Okinawans calories came from carbohydrates with sweet potatoes comprising 69% of all calories and just 1% of calories from fish.
In 2016 a paper pointed out the Okinawans 1949 post-war diet had a ratio of protein to carbohydrate is similar to an experimental high carb diet used to improve lifespans in rodents. In January 2019 a BBC article referring to the 2016 paper under the headline HIGH CARB DIET MAY EXPLAIN WHY OKINAWANS LIVE SO LONG.
While people did historically eat their fair share of sweet potatoes is it appropriate to draw conclusions about longevity based on Okinawans post war diet?
When you dig in a bit it becomes apparent this 2019 article is lacking context.
I’d like to share with you some insight into the shortcomings of research in order to help you understand what makes for a weak or strong piece of supporting evidence.
Let’s say a detective wants to determine who killed John. He will follow clues and investigate while considering the strengths and shortcomings in each piece of evidence he finds. A witness saying they saw someone that sort of looked like the butler is much weaker evidence than a security camera capturing an image of the butler.
It’s good to employ a similar approach when trying to make conclusions from research. Last year, I picked up this book with the title, the best diet simple and evidence-based guide to healthy eating written by a doctor at UCLA. I was considering the recent trend of butter coffee, and I was curious as to what this book had to say. It says butter is a bad fat as shown by several studies.
Now the first most obvious step in evaluating a claim is to investigate the evidence that the claim is based on. The idea that butter is bad comes from observational studies in which butter seems to raise the bad LDL cholesterol.
However, evidence butter intake affects your risk for disease is pretty weak.
Eggs, especially the yolk, can be a cheap source of good nutrients, like fat-soluble vitamins, which aren’t contained in the majority of typically eaten foods. Yet, the book recommends limiting your egg intake to only one a day.
The high-fat diet used was 20% protein, 45% fat and 35% carbohydrates.
This is relatively high fat.
However, half of the carbohydrate was refined table sugar by weight. There’s almost as much pure sugar as there is fat.
So do you think this might confound the effect of fat?
A common challenge is isolating the effect of one food or gene on disease risk from the effects of all the other foods and genes that could also potentially increase the disease risk.
One of the goals of doing a low-carb diet is keeping your insulin low and to achieve that people replace carbs with fat or protein, but protein seems to raise insulin levels. However, does the context matter? Does protein by itself reliably raise insulin levels. If we take a look at this study and canines as presented by Dr. Benjamin Bickman, we see that dogs receiving an infusion of glucose get spikes in their insulin levels when given the amino acid alanine. So it looks like protein does raise insulin, but what about dogs without the glucose infusion? The dog’s not receiving the glucose, didn’t see their insulin change to any noticeable degree.
So then imagine how this fact would confuse the data. In, for example, a study looking at how protein affects risk for diabetes and insulin driven disease, you might look at how many servings of meat people are having per day and then look at who develops diabetes, but the physiological effects of a hamburger patty tucked in a whole-wheat bun and served with French fries are going to be much different from a steak served only with butter and rosemary.
Another good example of the importance of context comes from the work of Dave Feldman. Dave is an independent lipid researcher who has developed something called the lipid energy model.
The logic behind why, when it comes to heart disease, LDL, the so-called bad cholesterol, isn’t all that important in the context of high HDL and low triglycerides. That is you don’t need to worry so much about sky high, bad cholesterol. If your HDL is high and your triglycerides are low, the enhanced data is certainly exciting because while it’s true, if you look at LDL all by itself, it can be associated with higher mortality.
When you look at LDL, when grouped with high HDL and low triglycerides, it’s associated with low mortality. This huge enhanced data set that Dave recently got his hands on is showing that this idea that HDL and triglycerides are more important than LDL cholesterol, indeed pans out surprisingly well.
It removed everybody that had a low LDL, so everyone was 159 milligrams per decilitre in lower. He then separated out everybody with HDL cholesterol of 49 or lower, and everybody above 100 milligrams per decilitre of triglyceride. This was pretty thrilling because now we can actually look at what the mortality data was that was left.
And that mortality was pretty exciting because not only did they have an all-cause mortality that was lower than the average, but believe it or not diseases of the heart were extraordinarily low. The youngest person in that group that was leftover once all three of these markers were accounted for was 68.
The oldest in the group, 94 and outside of those two, everybody else died in their eighties, a total of 18 total deaths from diseases of the heart. And almost everyone died of old age.
So let’s return to the previous point about the recommendation to consume only one egg a day. The author of the earlier mentioned book explains that according to a 2013 meta analysis, those who ate more than one egg a day had a 42% higher risk for developing type two diabetes than those who hardly ate eggs.
What an analysis does is pile the data from multiple studies together to try and produce more accurate conclusions. So here’s another point for investigating a claim. The cumbersome task of actually digging through the reference study.
The data from one study did suggest high levels of egg consumption is associated with increased risk for type two diabetes. However, another study observed no statistically significant associations between egg consumption and diabetes. And a third also noted no association between egg consumption or dietary cholesterol and increased risk of incident type two diabetes.
But by taking these three studies with divergent conclusions and pulling the data together, therein a meta analysis, the conclusion becomes compared with those who never consume eggs, those who eat one egg per day or more are 42% more likely to develop type two diabetes. Initially, this seems like a more accurate picture, right? However, in this study, the women eating the most eggs are smoking the most, consuming higher amounts of trans fat, eating 500 more calories per day, and exercising the least. The men who ate more eggs also drank more alcohol and smoked more. The researchers do take these unhealthy habits into account and make adjustments when analyzing the data, but it is overly ambitious to assume you can quantify the effects of all things that increase diabetes risk and then subtract these to accurately understand how just eggs by themselves affect diabetes risk. In any case, the studies employing this meta analysis are not all adjusting for potential confounding variables.
One only adjusts for age and sex. One doesn’t even account for how many calories the people ate along with the eggs. What if the people eating two eggs a day are getting those two eggs from a Denny’s grand slam seven days a week?
While this kind of meta analysis study is a clue to the puzzle of eggs it’s not as strong a piece of evidence as it appeared superficially. Naturally, the other reason for us being told not to consume eggs comes from the theory that fat and cholesterol cause heart disease.
The very first clue for this theory comes from research found that feeding cholesterol to rabbits had caused them to develop very high levels of blood cholesterol and atherosclerosis. But rabbits are herbivores, so their natural intake of cholesterol hovers right around zero milligrams.
So let’s move on to my next point. The shortcomings of using animals as a model for understanding humans. The basic idea is the mice used in experiments are not very diverse. It’s kinda like having, maybe having a breed of dog is the best analogy.
So like, if you did all the experiments on golden retrievers, is that really representative of what would happen if you did the experiment on 10 different dog breeds?
To give you a picture of how this can affect research, consider the work of Louis Dhall. In 1963, he fed rats, a high salt diet and found some, but not all developed high blood pressure.
He then went on breeding, producing a strain of rats genetically sensitive to salt. Then in 1970, he fed the salt-sensitive rats commercial baby food, and about half of the salt -sensitive rats died. So he concluded the high salt content of the baby food was to blame after his study was published.
The US Senate issued a mandate for lowering salt in baby foods. Now think about that for a moment.
Does anything sound odd to you about this sequence of events?
Anyhow, let’s get back to our discussion on mice models. And well, it’s, its well-known mice trials are not always very good and the pharmaceutical industry knows that really well.
And so it’s really easy to think about, which is, well, one is like, how do drugs get approved for humans? Well, one is they do early stage preclinical work, which is usually like on cell lines and then on mice, and then they move to canines and then apes or something.
And then they start human trials. But you may be familiar with the phenomena where in clinical trials, one like the drug pass, but it failed in clinical three trials. But if it failed at one of these human based trials, but, well, it passed the mouse trial, right?
So I think that right there, it gives you some idea the mouse model did not model what we expected to happen in humans. Why does that happen? Well, a mouse is different from a human and the model they make, for example, where the mass is a certain type of tumor that tumor may not really perfectly modeled the tumor in humans or diabetes and mice might not be the same, like the model they make situations where like you’ll pass the mouse phase, but then you fail at the human phase. That’s not a rare occurrence. It’s likely the opposite. With all this said, studies based on mouse models are still pieces of evidence. Not to be completely dismissed whenever we don’t like their findings.
But we should try and investigate the specifics of why a particular mouse model wouldn’t be appropriate for emulating humans. The one of the easiest differences to point out is just that that lifetimes are shorter. So they mature faster. These are really easy differences to spot between mice and humans. These are just some of the things you want to start to think about when you’re thinking, is this a good model to use for humans?
Let’s say it’s a typical Saturday morning. You’ve just made your coffee and are sitting down to read a research paper and you see the words, HIGH FAT DIET INDUCES OBESITY, but you are considering doing a keto diet to shed some weight. And you think, if a high fat diet is a reliable way to produce obesity in rodents, why would I want to do a high-fat keto diet? However we should first investigate? If there are some specific metabolic differences between rodents and humans. The high fat diet for the mice used is 20% protein, 60% fat and 20% carbohydrate. An actual ketogenic diet for humans would need to be restricted to around 10% or even 5% carbohydrate.
But at 20% of calories coming from carbohydrate, this rodent chow is actually a relatively low-carb diet for a human. Presently, it’s thought that most of the weight loss magic comes from a keto or low carb diet comes from lowering insulin and entering ketosis. However, rodents don’t enter ketosis nearly as easily as humans.
According to Dr. Benjamin Bickman in rodent experiments without calorie restriction to get rodents into ketosis, you need to reduce their diet down to just 1% carbohydrate, 9% protein and 90% fat, or even a diet that is 95% fat may not put rodents into ketosis. By the way a 95% fat diet would be like an entire cup of butter and about 80 grams or eight thin slices of bacon for the day. Any more bacon than that would be too much protein.
Simply put the amount of carb restriction that qualifies as low carb or keto for humans does not qualify for a rodent. These are the kinds of specific differences that should be acknowledged when using rodents as models for humans.
Let’s move on to my next point.
Food compounds and food. We’ll start with chocolate. In his book, Doctoring Data, Dr. Malcolm Kendrick talks about a headline he saw saying chemicals found in chocolate protect against heart disease. He explains that according to the research catechins and procyanidins found in dark chocolate inhibit the enzyme angiotensin converting enzyme ACE. When ACE is blocked blood pressure drops.
This is actually how blood pressure lowering drugs work. So compelling reasoning behind a compelling headline. The only catch is there weren’t any actual clinical effects. While dark chocolate did result in an 18% drop in ACE activity there was no actual drop in blood pressure in those taken cocoa extract.
Another example is the idea that a compound in such and such food has been found to cause disease, so that food itself must cause disease. As example, there is the idea that heterocyclic amines and cooked meat cause cancer, however, studies are finding these heterocyclic amines are cancerous where giving rodents amounts of heterocyclic amines equivalent to 1000 to 100,000 times the standard amount consumed by humans. As this paper says, comparison of the carcinogenic dose in rodents and the actual human daily intake suggest the latter is definitely too low for cancer production to be explicable in terms of heterocyclic amines alone.
In any case the point is claims like such and such food prevents or causes disease are very different from claims compounds in such and such food prevent or cause disease.
My next point on what to consider when evaluating pieces of evidence is why and from where certain ideas arose. For example, why would people think to start looking at red wine to see if it benefited heart disease? Well, sometime around 1991, people were trying to make sense of the fact that the French eat very large amounts of saturated fat yet have low rates of heart disease. One idea was that this so-called French paradox could be explained by France’s high red wine consumption. So this is what’s called an ad hoc hypothesis, a hypothesis added to a theory in order to save it from being falsified, in this case, because saturated fat and cholesterol must cause heart disease, it was assumed there must be some protective factor in the French diet. But another way the logic is, let us construct a new hypothesis; red wine prevents heart disease to explain data that does not support our initial hypothesis, fat causes heart disease. Now just because a hypothesis is ad hoc doesn’t mean it’s wrong, but it deserves scrutiny.
And what really deserves scrutiny are things that go quickly from the idea stage to clinical practice. An example is how a seemingly good idea arguably killed millions of people from the early 1900s. For about 50 years or so it was thought strict bedrest for about six weeks was the appropriate prescription for someone who had just had a heart attack. It sort of makes sense. After such a traumatic event, it sounds like it would be best to let the heart rest and keep exertion minimal, so as not to stress the heart. And I really mean minimal. To quote, Thomas Lewis, a prominent physician from the 1930s. “The patient is to be guarded by day and night nursing and helped in every way to avoid voluntary movement or effort.”
So what exactly is wrong with bedrest? First lying in bed stationary for six weeks means there is a very good chance of developing a deep vein thrombosis in the legs. A high percentage of these break-offs travel to the lungs and block the arteries in the lungs, causing a pulmonary embolism and event with a very high mortality rate.
In fact, even a multi hour flight carries this risk. In 1977 the term Traveler’s thrombosis was coined for people developing deep vein thrombosis. Flight’s low oxygen, low humidity, and low cabin pressure at high elevations, plus sitting motionless in a chair for several hours is a good recipe for thrombosis.
The second issue with bed rest is that without any exercise and especially after a heart attack, the heart atrophies very rapidly, it becomes weaker and deadly heart rhythms develop. So you are far more likely to die of ventricular fibrillation. Dr. Kendrick estimates that hundreds of thousands of people were dying from bed rest each year.
And this approach wasn’t being questioned until the mid 1950s. So, where did this idea come from? Well, in 1912, Dr. James Herrick of Chicago published an article titled clinical features of sudden obstruction of the coronary arteries, where he essentially described the first documented heart attack. And that article, he stated the importance of absolute rest in bed for several days is clear post infarction.
To quote, Dr. Kendrick “Herrick managed to describe the world’s first heart attack in 1912, and then without missing a beat, he instantly knew strict bedrest was an essential form of treatment for a condition never before described.”
Another example of seemingly good ideas harming people is that of hormone replacement therapy.
It had been recognized that women under 60 had far lower rates of heart disease than men of that age. For various reasons, it became accepted that female sex hormones were what was protective against heart disease. According to Dr. Kendrick one key piece of evidence amidst the very limited amount of evidence for this concept was in 1987.
Observational study observational studies, by the way, are widely accepted to be very weak pieces of evidence in general, yet the idea was still accepted so well in fact that replacing the declining female sex hormones in menopausal women became incorporated into the 1992 American college of physicians guidelines in the US failure to prescribe hormone replacement therapy for menopausal women was akin to medical malpractice later.
A randomized primary prevention trial using hormone replacement therapy involving nearly 17,000 women published its results in 2002. This trial found there was a 29% increase in coronary heart disease risk. Now I wonder how many women would have greed to receive hormone replacement therapy if they knew the practice originated from a sort of good idea, backed by a weak observational study. My very last point is the circular situation where existing ideas can influence research in a way that biases the research towards acting as evidence for that idea.
So, what do I mean by that?
One of my problems with cholesterol research is it often lumps soft endpoints with hard end points. And this can be a bit of a challenge because our existing opinion on cholesterol can make a difference in how the data are recorded.
So to provide you an example on the patient side. You and I have a steak dinner tonight, and then afterwards we both go our separate ways, but each of us that night experiences a 30 minute prolonged chest pain.
Now for me, this is the warning I’d been hearing about from my doctor this whole time, after all he’s been telling me about my elevated cholesterol and how I need to do something about it, or I’m going to have a heart attack.
Sure enough, when I go to the hospital, it does in fact prove true that I did have a non-fatal myocardial infarction. Consequently all of that data then becomes recorded.
However, you also have a myocardial infarction, but the difference is because your cholesterol has been low, you went ahead and took a Tums because you thought it was heartburn and went to sleep.
This also plays into the hands of medical professionals because this reinforces the opinion that high LDL is a risk factor.
That’s a big deal.
Biology is incredibly complicated. There are so many variables that may affect a given output.