Understanding Research Evidence

Health claims are based on evidence from research.¹

All claims need evidence to support them. This is especially true in health care, where our decisions can have life-changing consequences. While there are many different types of evidence, research is generally accepted to be the best source for evidence about health.

Research evidence is based on the findings of research studies. Research studies use scientific methods to seek answers to the questions we have about health and illness. Research evidence is sometimes comprised of the findings of just one study; at other times, it can be made up of the findings of hundreds of different studies.

There are many different sources of health information. We may hear about a friend’s experience, read a news article online, or simply listen to a doctor’s advice. However, this information may not always be as accurate as we believe.

Common problems with health information:

• All people can have conscious or unconscious beliefs that affect their judgments, even when those beliefs are not true. These biases can influence which treatments a doctor recommends or how a reporter writes about a treatment, in ways that might not be true.

• When we hear about another person’s experience with a treatment, we often assume that our experience would be the same, which is unlikely to be true – it takes very large groups of people to get an accurate picture of the effects of a treatment.

• We often make assumptions about the connections between a treatment and an outcome. However, unless strict controls are put in place, we cannot know for sure what actually caused an outcome. For example, it can be impossible to know if a medical treatment helped a person recover after an SCI, or if it was just the result of natural (spontaneous) recovery.

The importance of research evidence

Some of these scientific methods may include:

The process of obtaining research evidence should be systematic and unbiased.³

• Using research techniques like blinding, control groups, and randomization to minimize bias.

• Studying large groups of people to identify patterns that may not be seen in smaller groups.

• Using special statistics to find out whether the findings could have simply been caused by chance.

• Providing a clear explanation about how the study was done, so you can think for yourself about how to interpret its findings. This also allows other researchers to repeat (replicate) the study to see if they get the same results.

• Most published research is peer-reviewed to ensure it meets research standards before it is published.

• Researchers are usually required to report any conflicts of interest (like if the authors have a financial interest in a product they are testing) to ensure that their research findings are independent of outside influences.

Although research provides the most reliable way of gathering information about a subject, research alone cannot tell us everything that we need to know about health. Some of the limitations of research as a form of evidence include:

• Conducting research is costly, challenging, and time-consuming. Only a small number of the questions we have will ever be answered directly through research.

Conducting and interpreting research are often challenging.⁴

• It is difficult to conduct high quality research. Even the most carefully designed studies can be faced with circumstances that create bias. Because of this, the majority of research studies do not provide strong evidence.

• Research can often be difficult and time-consuming to understand. This makes it challenging for everyone, including your health providers, to easily use research as a part of everyday decision-making.

Because of these challenges, it is important to consider that health decisions should be made by considering all the best available evidence, including evidence from research, the clinical experience of your health team, and your individual values and preferences.

There are many different types of research study designs. Below, we briefly outline the most common study designs used in SCI research.

The research design determines a study’s quality of evidence.⁵

Randomized controlled trials

In health research, the study design that provides the strongest evidence (as a single study) is called a randomized controlled trial, or RCT. Randomized controlled trials are the most rigorous type of experimental study and can be used to determine whether a treatment actually caused the result.

Randomized controlled trials are research experiments that place participants into at least two groups by chance, like the flip of a coin. One group (the experimental group) is given the treatment being tested and the other group (the control group) is given a comparison treatment. The two groups are then compared at the end of the study to see if they had different results.

Controlled trials without randomization (Prospective controlled trials)

In controlled trials without randomization, there is also an experimental group and a control group that are compared at the end of the study. However, unlike in randomized controlled trials, participants in these studies are not randomly assigned to their groups.

Because the groups are not randomly assigned, they may have additional differences that make a true comparison impossible. However, in some cases researchers cannot randomly assign participants into different groups, so this type of study design is used.

Pre-post studies

Pre-post studies are one of the most common types of study designs used in SCI research. In this type of study, a group of people is tested before receiving a treatment and then afterwards. The difference between the ‘before’ and ‘after’ tests is thought to show the effects of the treatment.

Pre-post studies are often used because they are often more convenient, ethical, and appropriate in a variety of different situations. However, because this study design does not control many of the factors that could affect the results of the study, it can be difficult to determine if changes in the results are caused by the treatment itself or by these other factors.

Observational studies

Another type of research involves observing what happens to a group of people over time when the researcher cannot control which participants receive which treatments. This is called observational research. This type of research is used to observe connections and relationships between different factors.

Cohort studies are a type of observational study that follows or looks back on what happened to two (or more) comparable groups over time. The groups differ by an important characteristic, such as a health condition, risk factor, or treatment. The outcomes of the two groups are then compared to see how they differ over time.

Laboratory studies (Animal studies)

Rats are often used in animal studies.⁶

Laboratory studies involving animals are usually done in an early stage of research to determine safety and if a treatment has potential before a risky procedure is used on people. Like human studies, there are strict ethical guidelines for performing studies involving animals. It is important to note that many treatments that are effective in animal studies have not been found to be effective in humans, so animal studies are considered introductory research that cannot simply be applied to humans as is.

Case studies

Case studies describe the results of a treatment in a single individual (or case). Case studies are often used to communicate information when larger studies have not been done, or when it is difficult to do larger studies, like when a condition or treatment is extremely rare. A disadvantage of case studies is that because it is only based on one person, we do not know if the study’s conclusions also apply to other people.

Systematic reviews and Meta-analyses

Systematic reviews and meta-analyses summarize the findings of the research studies to answer specific research questions.⁷

Systematic reviews and meta-analyses combine the findings of all the research studies on a subject together. Systematic reviews involve systematically searching for all the studies that address a specific question, reviewing the quality of the studies, and interpreting the findings of all of the studies together. Sometimes, systematic reviews may pool the data from different studies together and then analyze this grouped data. This is called a meta-analysis.

Systematic reviews and meta-analyses are considered the strongest form of research evidence to help with decision-making. These studies give greater context to the research and can weigh the findings of different studies against each other. However, systematic reviews and meta-analyses are only as strong as the studies they are based on, so they can still have some types of error.

Qualitative Research Designs

While the research methods listed above are most often used for making treatment decisions; qualitative research methods like interviews and focus groups provide other important knowledge. Qualitative research seeks to describe the qualities of something to develop a deeper understanding about it.

For example, qualitative research may be used to describe the qualities of pain after SCI or the effect that it has on people’s daily lives. However, qualitative research studies cannot determine whether a treatment works, so this type of research is not the focus of SCIRE Community.

Expert opinion

Expert opinion is a form of evidence based on the opinions of experts in the field, such as doctors and scientists. This includes the advice of health providers and researchers and clinical consensus statements. Clinical consensus statements are written documents that include the recommendations of an organized group of experts on clinical issues. This type of evidence is based on clinical experience or foundational medical principles. Expert opinion and clinical consensus have an important place in interpreting research evidence and making decisions when no high quality research has been done.

The quality of an experimental study is determined by how effectively the researchers reduce biases and errors in the study. Some of the features of high quality experimental studies include:

Randomization

Randomization, control groups, blinding, and large numbers of participants help reduce biases and errors during data collection and analysis.⁸

Randomization is when study participants are randomly placed into the experimental group or the control group of a study. This is done to reduce biases in how participants are assigned to the groups within the study. Randomization means that all groups start off the same so they can be compared fairly at the end of the study.

Control groups

A control group is a group of participants in a study that receives an alternative treatment instead of the treatment being tested. This may be a placebo, a comparison treatment, or simply usual care (the care you would have if you were not in the study). At the end of the study, the control group is compared to the experimental group to see if they are different. Because the two groups only differ by which treatment they received, differences are thought to show the effects of the treatment.

Blinding

Blinding is when the type of treatment (experimental or control) that a participant receives is intentionally withheld from that person. The type of treatment may also be withheld from the researchers who are collecting information. This is called a double-blind experiment. Blinding is done to reduce the impact that people’s biases can have on how they report on something.

Large numbers of participants

When a study looks at a large group of participants, the people being tested are more likely to represent the general population and statistical analyses are more likely to be accurate. This allows the results of the study to be applied more accurately to real world situations.

All studies are not created equal – the type of study design and the quality of a study affect how convincing a study is as evidence. There are many different rating systems that have been developed to determine which studies provide the strongest evidence. These rating systems classify studies into groups called levels of evidence.

Levels of evidence can help us determine the value of research evidence in our treatment decisions. Higher quality evidence is usually weighed more heavily. However, lower quality evidence is still valuable when conclusions are made about a treatment, especially if there is no other research to help us understand it.

Levels of Evidence on SCIRE Community

The SCIRE Project (SCIRE Professional) uses a scale to rate evidence into six categories based on the type of study design and the quality and number of randomized controlled trials. This is explained in detail in the “SCIRE Systematic Review Process: Evidence” Chapter. On SCIRE Community, we have combined these levels into three categories:

Strong evidence

Strong evidence is research evidence based on two or more high quality randomized controlled trials (SCIRE Professional Level 1a). When there is strong evidence to support the use of a treatment, we can be confident that it has been proven to work. Likewise, if there is strong evidence that a treatment is ineffective, we can be confident that it does not work. In general, the more studies that have the same findings, the more confident we can be that a conclusion is accurate.

Moderate evidence

Moderate evidence is research evidence that is based on one randomized controlled trial, or on one or more prospective controlled trial or cohort study (SCIRE Professional Level 1b and 2). When there is moderate evidence to support the use of a treatment, we have some guidance on whether the treatment works, but cannot be completely confident in its findings. More research is needed to be sure.

Weak evidence

Weak evidence is based on lower level non-experimental study designs (like case studies and observational studies) or clinical consensus (SCIRE Professional Level 3, 4, and 5). These study designs are most susceptible to biases and errors so they are considered lower quality evidence when weighing treatment options. However, these studies often have different purposes, such as communicating unique and uncommon findings or making connections between different factors.

That being said, when we are trying to decide whether a treatment is effective, weak evidence provides early support that a treatment is effective or ineffective, but is not enough to draw conclusions from. More research is needed.

Conflicting evidence

Conflicting evidence is when some studies support the use of the treatment, and others do not. Conflicting evidence can be challenging to interpret, requiring careful assessment of the involved studies to determine potential problems or errors that could affect their results. Conflicting evidence often makes it difficult to draw conclusions about whether a treatment works or not.

Expert opinion

As mentioned above, expert opinion is evidence based on the opinions of experts in the field. Expert opinion has an important place in interpreting research evidence and making decisions when no high quality research has been done, however expert opinion is considered a weak form of evidence compared to research evidence.

Making decisions using research evidence

Keep in mind that these levels are used to approximate the strength of evidence for ease of use and understanding. However, each category will vary somewhat and it is important to interpret this in the context of other important information. On top of the conclusions drawn from research evidence, other factors like potential risks, cost concerns, personal suitability, and your preferences also need to be taken into account when deciding on treatment options for your health.

Friends and family may be a valuable source of health information.¹¹

Designing and carrying out high quality research studies is costly, challenging, and time-consuming. It can take many years and hundreds of thousands of dollars for just one high quality randomized controlled trial to be done. Because of these challenges, many of the questions we have about treatments cannot be answered through research alone. Research is just one of many forms of evidence. Other types of evidence include:

• Your personal experiences and reasoning

• The clinical experience and reasoning of health providers or other experts

• Traditional or common practices

• The opinions and experiences of your family and friends

Want to learn more about research evidence?

Understanding Health Research. Useful Information. Available from: http://www.understandinghealthresearch.org/useful-information/.

Understanding Health Research. How to read a scientific paper. Available from: http://www.understandinghealthresearch.org/useful-information/how-to-read-a-scientific-paper-4.

Understanding Health Research. How science media stories work. Available from: http://www.understandinghealthresearch.org/useful-information/how-science-media-stories-work-3.

Cochrane Consumer Network. Available from: http://consumers.cochrane.org/.

Looking for more information about how SCIRE does its systematic reviews?

SCIRE Professional – Methods of Systematic Review. Available from: https://scireproject.com/about-scire/methods-of-systematic-review/.

Foley NC, Bhogal SK, Teasell RW, Bureau Y, Speechley MR. Estimates of quality and reliability with the physiotherapy evidence-based database scale to assess the methodology of randomized controlled trials of pharmacological and nonpharmacological interventions. Phys Ther. 2006 Jun;86(6):817-24.

Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine: how to practice and teach EBM. Toronto, Ontario: Churchill Livingstone, 2000.

Sackett DL, Rosenberg WM, et al. Evidence based medicine: what it is and what it isn’t. BMJ 1996;312:71-72. 52.

PEDro: http://www.pedro.fhs.usyd.edu.au/scale_item.html

Image credits:

1. Research ©luckey_sun, CC BY-SA 2.0
2. Brainstorming ©Icons8, CC0 1.0
3. Resorting to paper… #research# #proposal ©catherinecronin, CC BY-SA 2.0
4. NCSR Researcher using Ubuntu ©Simos Xenitellis, CC BY-SA 4.0
5. Today’s reading #bigpicture by catherinecronin, CC BY-SA 2.0
6. A lab friend ©Anna Marchenkova, CC BY-SA 4.0
7. Library books ©timetrax23, CC BY-SA 2.0
8. Research Data Management ©janneke staaks, CC BY-NC 2.0
9. Magnifying glass ©kaosnoff, CC0 1.0
10. Scientist ©H Alberto Gongora, CC BY 3.0 US
11. Image by SCIRE Community Team