All Research Is Not Created Equal

It seems like science can be nothing more but contradictions and large words. One week a study says eggs are good, the next a better study links them to early death. How do you decipher what is true? Which study holds more weight? 

While science does deserve some flack for being contradictory, if the lay person can understand different types of research it will go a long way in clearing up the confusion. Not all research is created equal. Many studies have flaws. Others hold more weight because of the “strength” of the study. I’ll examine the different types of research and a few things to look out for.

Two Different Worlds

It’s crucial to know the difference between the two major kinds of research, observational and experimental.

Observational Research

An observational study is just what it sounds like, the researcher(s) will look at subjects, what has happened to them and try to make associations between a certain behavior and end result. These types of studies are usually very long and involve a large number of subjects. The purpose of observational research is to find associations, it’s a great hypothesis generator for more controlled trials. What they cannot do is provide direct causation between a behavior and the end result. This is due to a lack of control.

Correlation does not equal causation

For instance, a research team can look at a group of 1,000 subjects who competed in the Olympics and survey their habits over the past decade. They may find that 70 percent of them supplemented with creatine monohydrate, therefor creatine monohydrate usage is associated with Olympic athletes, but this does not give direct causation that it aided them. However, this does provide a great hypothesis for a experimental research.

Experimental Research

This is where we can really dig deep into concepts and put them under more controlled scrutiny. Experimental research can take our first example of Olympians and creatine monohydrate and test the hypothesis. Researchers make ask the question “Does creatine monohydrate improve athletic performance?” Now having a solid question, they form an experiment where they gather 100 subjects, control for as much as possible (training age, gender, diet, etc.) and can test whether creatine monohydrate does in fact improve performance. The length of the study can vary greatly, but say in this example it’s 16 weeks. Another factor researchers must consider is the placebo effect, therefor there is usually a control group that is given something non-ergogenic, like sugar pills.

In our example, 50 subjects would be given 5 grams of creatine monohydrate everyday while maintaining regular training, diet, sleep and the other 50 subjects would be given a sugar pill. At the end of the 16 weeks they run a variety of tests, such as sprinting time, endurance, and one rep max strength. Now having a more controlled experiment the researchers can now look at creatine monohydrate vs sugar pills and see if it did indeed improve performance over sugar pills. And if so, by how much? Did it improve strength? Speed?

Of course, one single study does not provide enough evidence to say that something is truth or works. Multiple studies need to implemented with a variety of dosing, subjects, testing of different performance. There is really a never ending search, which is what makes it fun! Another point to make, scientific studies can never “prove” anything. The most we can say is that something has strong evidence for.

Which is better?

This is certainly a loaded question and the truth is neither is better than the other and they both have their place. Below is a table I’ve set up to allow you to simply see the pro’s and con’s of each type of research. Note: these are generalities and there are certainly exceptions to the following.

 What strengthens a study & what to look for

  • Who funded the study? Funding bias is huge. If a supplement company funded a study and the conclusions were that the subjects gained 10 lbs. of muscle in 2 months, it deserves to be looked at in more detail.
  • Number of subjects. A study with more subjects will always hold more weight than a study with fewer. This is due to small subject size having a greater chance of experimental error showing a false-positive. This is also known as statistical power. So if two studies show a supplement to improve strength, but they each only had 10 subjects, but 10 other studies with 100′s of subjects show no improvement, then the weight of evidence wouldn’t support its use.
  • Study length. The protocol showed a boost in testosterone 2 hours after training? Awesome, but what does that mean over the course of days, weeks, months? Many studies are performed in a very short duration such as days or even hours. While they are good starting points, they can’t do much for long term evidence. The longer duration of a study the more weight it holds and more closely mimics real world scenarios.
  • Who was studied? Just because a study or even large of amount of evidence supports something doesn’t mean it actually will work for you. Maybe the subjects were untrained. Maybe the subjects were obese. Different subjects can cause very different outcomes.

So there you have the nuts and bolts of two different types of research, which most exercise and nutrition topics fall under. There is also something called a meta-analysis, which I may cover in more detail later. A meta-analysis is more or less a collection of studies.

Hopefully this clears up some confusion you may have about some topics in health. Many topics such as red meat, artificial sweeteners, egg consumption and many others have been taken out of context and wrongfully accused of negatives health outcomes due to not knowing the limitations of research. Now you won’t be fooled