Understanding Research Evidence

Health claims are based on evidence from research.¹

All claims need evidence to support them. This is especially true in health care, where our decisions can have life-changing consequences. While there are many types of evidence, research is generally accepted to be the best source for evidence about health.

Research evidence is based on the findings of research studies that use scientific methods to seek answers to the questions we have about health and illness. Research evidence is sometimes comprised of the findings of just one study and at other times it is made up of the findings of hundreds of different studies.

There are many different sources of health information. We may hear about a friend’s experience, read a news article online, or simply listen to a doctor’s advice. However, this information may not always be as accurate as we believe.

Common problems with health information:

• All people can have conscious or unconscious beliefs that affect their judgments, even when those beliefs are not true. These biases can influence which treatments a doctor recommends or how a reporter writes about a treatment, in ways that might not be accurate.
• When we hear about another person’s experience with a treatment, we often assume that our experience would be the same, which is unlikely to be true – it takes very large groups of people to get an accurate picture of the effects of a treatment.
• We often make assumptions about the connections between a treatment and an outcome. However, unless strict controls are put in place, we cannot know for sure what actually caused an outcome. For example, it can be impossible to know if a medical treatment helped a person recover after an SCI, or if it was just the result of natural (spontaneous) recovery.

The importance of research evidence

Research evidence is important because the scientific methods used in well-designed research studies are more objective (unbiased) and accurate than conclusions based on other sources of evidence.

Some of these scientific methods may include:

The process of obtaining research evidence should be systematic and unbiased.³

• Using research techniques like blinding, control groups, and randomization to minimize bias.
• Studying large groups of people to identify patterns that may not be seen in smaller groups.
• Using special statistics to find out whether the findings could have simply been caused by chance.
• Providing a clear explanation about how the study was done, so you can think for yourself about how to interpret its findings. This also allows other researchers to repeat (replicate) the study to see if they get the same results.
• Requiring researchers to report any conflicts of interest (like if the authors have a financial interest in a product they are testing) to ensure that their research findings are independent of outside influences.
• Peer-reviewing a study to ensure it meets research standards before it is published.

There are many different types of research study designs. Below, we briefly outline the most common study designs used in SCI research.

The research design determines a study’s quality of evidence.⁴

Randomized controlled trials

In health research, the study design that provides the strongest evidence (as a single study) is called a randomized controlled trial, or RCT. RCTs are the most rigorous type of experimental study and can be used to determine whether a treatment actually caused the result.

RCTs are research experiments that place participants into at least two groups by chance, like the flip of a coin. One group (the experimental group) is given the treatment being tested and the other group (the control group) is given a comparison treatment or placebo. The two groups are then compared at the end of the study to see if they had different results.

Controlled trials without randomization (Prospective controlled trials)

In controlled trials without randomization, there is also an experimental group and a control group that are compared at the end of the study. However, unlike in RCTs, participants in these studies are not randomly assigned to their groups.

Because the groups are not randomly assigned, they may have additional differences that make a true comparison impossible. This type of study design is used when researchers cannot randomly assign participants into different groups.

Pre-post studies

Pre-post studies are one of the most common types of study designs used in SCI research. In this type of study, a group of people is tested before receiving a treatment and then afterwards. The difference between the “before” and “after” tests is thought to show the effects of the treatment.

Pre-post studies are used because they are often more convenient, ethical, and appropriate in a variety of different situations. However, because this study design does not control many of the factors that could affect the results of the study, it can be difficult to determine if changes in the results are caused by those other factors or the treatment itself.

Observational studies

Observational research involves observing what happens to a group of people over time when the researcher cannot control which participants receive which treatments. This type of research is used to observe connections and relationships between different factors.

Cohort studies are a type of observational study that follows up on or looks back on what happened to two (or more) comparable groups over time. The groups differ by an important characteristic, such as a health condition, risk factor, or treatment. The outcomes of the two groups are then compared to see how they differ over time.

Rats are often used in animal studies.⁵

Laboratory studies (animal studies)

Laboratory studies involving animals are usually done in an early stage of research to determine safety and if a treatment has potential before a risky procedure is used on people. Like human studies, there are strict ethical guidelines for performing studies involving animals. It is important to note that many treatments that are effective in animal studies are not effective in humans, so animal studies are considered introductory research that cannot simply be applied to humans as is.

Case studies and case series

Case studies describe the results of a treatment in a single individual (or case). Case studies are often used to communicate information when larger studies have not been done, or when it is difficult to do larger studies, like when a condition or treatment is extremely rare. A disadvantage of case studies is that because it is only based on one person, we do not know if the study’s conclusions also apply to other people. A case series is a study that includes multiple case studies.

Systematic reviews and Meta-analyses

Systematic reviews and meta-analyses summarize the findings of the research studies to answer specific research questions.⁶

Systematic reviews and meta-analyses combine the findings from all the studies on a topic together. This includes doing a systematic search for all the studies that address that topic, assessing the quality of each study, and interpreting the combined findings of all the studies together. Sometimes, systematic reviews may pool the data from different studies together and then analyze this grouped data. This is called a meta-analysis.

Systematic reviews and meta-analyses are considered the strongest form of research evidence to help with decision-making. These studies give greater context to the research and can weigh the findings of different studies against each other. However, systematic reviews and meta-analyses are only as strong as the studies they are based on, so they can still have some types of error.

Qualitative Research Designs

While the research methods listed above are most often used for making treatment decisions, qualitative research methods like interviews and focus groups provide other important knowledge. Qualitative research seeks to describe the qualities of something to develop a deeper understanding about it. For example, qualitative research may be used to describe the qualities of pain after SCI or the effect that it has on people’s daily lives.

Evidence quality can help us determine the value of research evidence in our treatment decisions. Higher quality evidence is usually weighed more heavily. However, lower quality evidence is still valuable when conclusions are made about a treatment, especially if there is no other research to help us understand it.

The quality of an experimental study is determined by how effectively the researchers reduce biases and errors in the study. Some of the features of high-quality experimental studies include:

Randomization

Randomization, control groups, blinding, and large numbers of participants help reduce biases and errors during data collection and analysis.⁷

Randomization is when study participants are randomly placed into the experimental group or the control group of a study. This is done to reduce biases in how participants are assigned to the groups within the study. Randomization means that all groups start off the same so they can be compared fairly at the end of the study.

Control groups

A control group is a group of participants in a study that receives an alternative treatment instead of the treatment being tested. This may be a placebo, a comparison treatment, or simply usual care (the care you would have if you were not in the study). At the end of the study, the control group is compared to the experimental group to see if they are different. Because the two groups only differ by which treatment they received, differences are thought to show the effects of the treatment.

Blinding

Blinding is when the type of treatment (experimental or control) that a participant receives is intentionally withheld from that person. The type of treatment may also be withheld from the researchers who are collecting information. This is called a double-blind experiment. Blinding is done to reduce the impact that people’s biases can have on how they report on something.

Large numbers of participants

When a study looks at a large group of participants, the people being tested are more likely to represent the general population and statistical analyses are more likely to be accurate. This allows the results of the study to be applied more accurately to real world situations.

When considering all the studies on a topic, the trends and comparisons between different studies can impact the relevance of the overall evidence.  Some factors to look at in a body of evidence include:

Number of research studies

The number of research studies published is important because each new study can validate, verify, or contradict the results of previous studies. If there are many studies on a specific topic with consistent results, the evidence is more likely to be reliable and be applicable to a more general population.

Consistency

Consistency is whether all the studies on a topic have similar results. When different studies produce opposing results and there is no explanation for the inconsistency, one should be cautious about making decisions using the evidence.

Doing research in populations with SCI is essential to improving treatment, rehabilitation, and management options for people with SCI, but there are some obstacles. Some research limitations unique to research in SCI populations include:

Low study participation

You may notice that many SCI studies have small sample sizes. SCI is not a common condition, so the number of people with SCI in a given location is often small. Even within that population, the level of injury and level of function will be very diverse. To make sure that this variation in injury types does not impact research results, studies often have strict participant criteria that require specific ranges for level of injury, level of function, time since injury, secondary health issues, medication use, etc. Also, people with SCI are more likely to have trouble accessing and maintaining participation in a study due to transportation, mobility, and ongoing health issues. All these factors contribute to the small sample sizes in studies of SCI and SCI treatments.

RCT challenges

Although RCTs are considered the gold standard for treatment research, they may not always be possible or ethical. There are ethical concerns over some of the strategies used in RCTs to reduce study bias when used for certain treatments. For example, randomizing participants to a non-treatment group in an exercise study, when it is commonly known that exercise is beneficial to health could be considered unethical. Invasive procedures such as surgeries are also difficult to study in RCTs because blinding participants might require a “sham” treatment (e.g. prepping the patient and making incisions but not doing the procedure). If a sham treatment is invasive and carries some risk, recruitment of willing participants in an already small participant pool becomes even more challenging. For the SCI population, it can be difficult to come up with a matched control group because of large variations in level of function and level of injury.

Although research provides the most reliable way of gathering information about a subject, research alone cannot tell us everything that we need to know about health. Some of the limitations of research as a form of evidence include:

Conducting and interpreting research are often challenging.⁹

• Conducting research is costly, challenging, and time-consuming. Only a small number of the questions we have will ever be answered directly through research.
• It is difficult to conduct high quality research. Even the most carefully designed studies can be faced with circumstances that create bias. Because of this, the majority of research studies do not provide strong evidence.
• Research can often be difficult and time-consuming to understand. This makes it challenging for everyone, including your health providers, to easily use research as a part of everyday decision-making.

Due to the limitations described above, many of the questions we have about treatments cannot be answered through research alone. Research is just one of many forms of evidence. Expert opinion, clinical consensus, and lived experience all have an important place in interpreting research evidence and making decisions when no high-quality research has been done.

Expert opinion

Expert opinion is a view or statement on a topic from an expert in the given field, based on clinical experiences or reasoning using foundational medical principles.

Clinical Consensus

Clinical consensus statements are written documents that include the recommendations of an organized group of experts on clinical issues.

Lived Experience

Friends and family may be a valuable source of health information.¹¹

Lived experience is the knowledge a person gains from direct, first-hand experience. There is value in understanding the impact and meaning of direct experiences for the development of research and treatments. Views of the same experience vary based on the person, their unique experience, and their environment.

Other sources of health information may include:

• Traditional or common practices
• Your personal experiences and reasoning
• The opinions and experiences of your family and friends

On top of the conclusions drawn from evidence other factors like potential risks and your preferences also need to be taken into account when deciding on treatment options for your health. Some questions to ask yourself before making a decision with the evidence can include:

1. Does this address your problem?
2. Based on the potential risks and benefits, is this suitable for you? (make a pros and cons list!)
3. Is this accessible for you? (finances, location, transportation)
4. How will this impact your life? (work, school, activities)
5. Do you have sufficient social, emotional, and physical supports? (family, friends, caregivers, other supports)
6. What are your personal preferences/goals?
7. What questions do you have?
8. What are the next steps that need to be taken?

For a list of included studies, please see the Reference List. For a review of how we assess evidence and advice on making decisions, refer to the SCIRE Community Evidence.

Want to learn more about research evidence?

Understanding Health Research. Useful Information. Available from: http://www.understandinghealthresearch.org/useful-information/.

Understanding Health Research. How to read a scientific paper. Available from: http://www.understandinghealthresearch.org/useful-information/how-to-read-a-scientific-paper-4.

Understanding Health Research. How science media stories work. Available from: http://www.understandinghealthresearch.org/useful-information/how-science-media-stories-work-3.

Cochrane Consumer Network. Available from: http://consumers.cochrane.org/.

Looking for more information about how SCIRE does its systematic reviews?

SCIRE Professional – Methods of Systematic Review. Available from: https://scireproject.com/about-scire/methods-of-systematic-review/.

Foley, N. C., Bhogal, S. K., Teasell, R. W., Bureau, Y., & Speechley, M. R. (2006). Estimates of Quality and Reliability With the Physiotherapy Evidence-Based Database Scale to Assess the Methodology of Randomized Controlled Trials of Pharmacological and Nonpharmacological Interventions. Physical Therapy, 86(6), 817–824. https://doi.org/10.1093/ptj/86.6.817

Sackett, D. L., Straus, S. E., Richardson, W. S., Rosenberg, W., & Haynes, R. B. (2000). Evidence-Based Medicine: How to Practice and Teach EBM. Churchill Livingstone.

Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996). Evidence based medicine: what it is and what it isn’t. BMJ, 312(7023), 71–72. https://doi.org/10.1136/bmj.312.7023.71

PEDro: http://www.pedro.fhs.usyd.edu.au/scale_item.html

Guyatt, G. H., Oxman, A. D., Kunz, R., Woodcock, J., Brozek, J., Helfand, M., Alonso-Coello, P., Glasziou, P., Jaeschke, R., Akl, E. A., Norris, S., Vist, G., Dahm, P., Shukla, V. K., Higgins, J., Falck-Ytter, Y., & Schünemann, H. J. (2011). GRADE guidelines: 7. Rating the quality of evidence—inconsistency. Journal of Clinical Epidemiology, 64(12), 1294–1302. https://doi.org/10.1016/j.jclinepi.2011.03.017

Martin Ginis, K. A., & Hicks, A. L. (2005). Exercise Research Issues in the Spinal Cord Injured Population. Exercise and Sport Sciences Reviews, 33(1). https://journals.lww.com/acsm-essr/fulltext/2005/01000/exercise_research_issues_in_the_spinal_cord.9.aspx

Zimmermann, G., Bolter, L., Sluka, R., Höller, Y., Bathke, A. C., Thomschewski, A., Leis, S., Lattanzi, S., Brigo, F., & Trinka, E. (2019). Sample sizes and statistical methods in interventional studies on individuals with spinal cord injury: A systematic review. Journal of Evidence-Based Medicine, 12(3), 200–208. https://doi.org/10.1111/jebm.12356

Tator, C. H. (2006). Review of treatment trials in human spinal cord injury: issues, difficulties, and recommendations. Neurosurgery, 59(5), 957–987. https://doi.org/10.1227/01.NEU.0000245591.16087.89

Ellis, C. (1992). Investigating subjectivity: Research on lived experience (Vol. 139). Sage.

Moberg, J., Oxman, A. D., Rosenbaum, S., Schünemann, H. J., Guyatt, G., Flottorp, S., Glenton, C., Lewin, S., Morelli, A., Rada, G., & Alonso-Coello, P. (2018). The GRADE Evidence to Decision (EtD) framework for health system and public health decisions. Health Research Policy and Systems, 16(1), 45. https://doi.org/10.1186/s12961-018-0320-2

Image credits:

1. Research ©luckey_sun, CC BY-SA 2.0
2. Brainstorming ©Icons8, CC0 1.0
3. Resorting to paper… #research# #proposal ©catherinecronin, CC BY-SA 2.0
4. Today’s reading #bigpicture by catherinecronin, CC BY-SA 2.0
5. A lab friend ©Anna Marchenkova, CC BY-SA 4.0
6. Library books ©timetrax23, CC BY-SA 2.0
7. Research Data Management ©janneke staaks, CC BY-NC 2.0
8. Magnifying glass ©kaosnoff, CC0 1.0
9. NCSR Researcher using Ubuntu ©Simos Xenitellis, CC BY-SA 4.0
10. Scientist ©H Alberto Gongora, CC BY 3.0 US
11. Image by SCIRE Community Team