Common Pitfalls In The Research Process

Issues of Concern

There are five phases of research: planning phase, data collection/analysis phase, writing phase, journal submission phase, and rejections/revisions/acceptance phase.

Phase I Pitfalls: Planning a Study

The highest yield preempting of pitfalls in the research process occurs in the planning phase. This is when a researcher can set the stage for an optimal research process. Below are pitfalls that can occur during the planning phase.

Pitfall: Underestimating what committing to a research project requires

Conducting a research study and achieving publication sounds fulfilling, right?

Consider the many steps: conducting a literature search, writing an IRB proposal, planning and having research meetings, long and cumbersome data collection processes, working with statisticians or analyzing complex data, having unexpected research setbacks (e.g., subjects drop out, newly published papers on same topic, etc.), the possibility that after data collection you have no statistically (or clinically) significant findings, conducting an updated literature search, writing introduction, methods, results, and discussion sections of a paper, going through the many journal options to determine best fit while aiming for high impact factors, adhering to journal guidelines/fixing drafts, writing cover letters stating importance of the topic to respective journals, creating journal portal accounts, possibly being rejected numerous times, waiting months for journal decisions, working on numerous revisions and being informed by numerous individuals about all of the flaws in your writing and research.

Does it sound, maybe less fulfilling?

Conducting a research project from inception to publication can be a rewarding experience. Research requires significant time. Setbacks are normal. To produce an important and sought-after research product, an individual must understand the magnitude of commitment required.

Pitfall: Choosing the wrong research pursuit/topic lacks precision

Consider an investigator interested in substance use research. The first challenge is the immense amount of research already published on this topic. Fortunately, there is still a massive amount of uncharted territory in substance use research.

It is important to understand what has been done and what is still undiscovered in your area of research. Do not simply study a topic because you find it interesting; passion is advantageous, but you should ensure that your study will contribute to some field/specialty or research in a significant way.

How does your research differ from what has been done?

How will it impact practice in a way that no previous study has?

Consider these questions when choosing a topic for research. Otherwise, you may struggle to get the work published. It can be demoralizing if you have already written your paper and realize that your paper is not going to get accepted by a reputable journal due to the presence of other papers already describing the same concepts you have.

As always, the first step is a thorough literature search.

Pitfall: Not considering research bias

A common theme noted in literature is that bias can, unfortunately, lead to failure to reproduce results, raising concerns regarding the integrity of science.[4] Bias can be considered various (inadvertent) poor strategies related to data design, analysis, and results reporting that produce spurious results and papers that perhaps should not be published.[5]

While one cannot completely eliminate bias from the research process, researchers should take steps to understand research bias in study endeavors and determine how to minimize bias during the planning phase of the study. 

Pitfall: Not focusing on which variables to collect

Researchers often want to collect as much data as possible but should not build a list of variables that includes every single detail about subjects if the variables collected are unlikely to yield insight into the topic of research. The longer the data collection instrument, the higher likelihood of (human) errors (if manually data entry) and the longer duration of the data collection phase. Instead of taking time to build a database with many variables, consider cutting irrelevant variables and use that time to increase the sample size. Determine, based on your own clinical knowledge and published empirical works, which variables are most crucial. 

Pitfall: Worrying about the statistics after the data has been collected

A vital part of the research process is ensuring you have a rigorous statistical approach. Involve your statistician very early in the project, preferably in the planning stages. They will have insight into the types of variables to collect and help shape the research methods. Statistical power is an important concept to consider before data collection to avoid false-negative results (Zlowodzki et al., 2006). Furthermore, other concepts, such as covariates, need to be part of the planning phase. Do not wait until after the data collection phase to give data to the statistician who cannot transform the data you have into outputs you want.

Pitfall: Not setting defined author roles

It is important to define who will be declared authors at the beginning of the research process to avoid conflict. Do most people want to be an author? Sure. Does everybody do the work worthy of authorship? No. While placing general comments in a shared document's margin may make the paper slightly better, it probably should not qualify for authorship. Review authorship criteria to determine what constitutes authorship. Clear expectations can ensure that everyone is on the same page and that everyone feels the process is fair, especially for individuals who plan to invest significant time in the project. Clear expectations for each author should occur before any writing begins, including deadlines and specific contributions.[6][7][6]

Pitfall: Not considering limitations of work before the paper is written

Avoid this pitfall by reviewing recent manuscripts and reading the limitations sections of these papers. Many of these limitations sections will make notions about generalizability to other populations. Some will discuss low power. Even the best papers in the top journals have many limitations. The best way to avoid or mitigate your work's limitations is to consider them during the planning phase.

How can you set up your project to limit your limitations section?

What (types of) samples should you include in your study?

Were you originally thinking of retrospective design, but it could be prospective?

What steps can you utilize to control baseline characteristics between groups?

Consider all limitations and think about how you can control these before data collection.

Phase II Pitfalls: Data Collection and Analysis

After the planning has occurred, typically after institutional review board (IRB) approval, the data collection and analysis phase can transpire. The entire team should typically stay involved throughout these phases. Below are pitfalls to avoid.

Pitfall: Not being involved in the data collection phase

It is important to be involved with the data collection phase, even if you do not personally collect data. Train the individuals who collect data to ensure all are on the same page and provide periodic oversight to ensure accuracy and quality of the data over time.[8] Do not assume the data collection phase is going smoothly – you may find yourself with a huge dataset riddled with inconsistencies or errors. Schedule periodic meetings to review data.

Pitfall: Not being involved with the statistical analysis phase

If you are not conducting the statistical analysis, do not assume that the person who is analyzing the data is 100% on the same page. Have meetings about the data, how to interpret the data, and the limitations of the data. Ask what other ways the data could be analyzed and how reviewers might negatively critique the data itself or the statistical methods.

The person conducting the analysis will not have the same familiarity with the topic. You are not going to be as familiar with the outputs. By understanding each other, you will a) have clearer, more robust methods and results in sections of the paper, b) limit critiques regarding the statistical approach/data outcomes, c) understand your research better for any presentations, discussion, or future work, and d) develop a positive collaboration for future work.

Phase III Pitfalls: The Writing Phase

The next phase is the writing phase. While this section covers pitfalls during the writing phase, for recommendations on conducting a literature search, writing, and publishing research, see StatPearls Evidence-Base Medicine Chapter: How to Write and Publish a Scientific Manuscript.[9] Below are pitfalls that can occur during the writing phase. 

Pitfall: Poor or outdated references

When writing your paper, perform multiple literature searches to ensure all recent, salient references are covered—claims about recent similar work or research that frames your study if the references are outdated. Journals may even ask reviewers to comment on the presence or absence of up-to-date/suitable references. Conduct a literature search prior to data collection and stay on top of references throughout the research process as new papers become available.

Pitfall: No clearly defined purpose of the paper

Many aspects of manuscripts can get overlooked. Lack of a clear purpose statement can doom a paper to futility. Remind the readers of the goal of the project. You do not want consumers of your research to read the results section and forget what the goals/main outcomes are. The purpose statement should be located at the end of the introduction section. 

Pitfall: Unclear methods making research hard to reproduce

A common concern in science is the lack of transparency in methods for reproducibility. The methods section should allow a reader to understand exactly what was done and conduct the study. Consider examining the StrengThening the Reporting of OBservational studies in Epidemiology (STROBE) checklist for the methods (as well as other paper sections) to ensure best reporting practices for reproducibility.[10]

Pitfall: The tables and narratives are the same

Reviewers prefer you not to state findings in narratives that are in tables. Tables focus readers on the most important results and are not redundant with the written content. Make call-outs to the table in the paper's narrative sections, but do not state information found in tables. 

Pitfall: Not reporting all data/outcomes

Some authors will state the main outcome of interest or have a statement such as “there were no other statistically significant findings between other groups.” Authors must report all outcomes and statistical analyses for reproducibility of the research. While this may be difficult to do with a broad approach, utilize tables and appendices to report all outcomes to show transparency and limit researcher bias.

Pitfall: Repeating results in discussion

Do not simply restate in the discussion what you already have in the results section. Utilize this section of the paper to link other references to your work and reflect on other empirical investigations' similarities or differences. Explain why your research provides an impactful contribution to the topic.  

Pitfall: Making conclusions that do not align with your work

Authors sometimes note in their conclusions how the work impacts a topic due to X reason when X may be too broad a claim and the work doesn’t really support or prove that notion. Researchers should align their conclusions to their own results and highlight the significance of their findings.

Pitfall: Thinking the title is not a big deal

A strong title will help with the impact/readership of your paper. Consider keeping a short title that provides the main takeaway. Papers with more concise titles and present the study conclusion result in a bigger impact/receive more citations.[11]

Pitfall: Completing the abstract last minute

Similar to the title, do not underestimate an abstract. Journal and conference reviewers (and the general audience) may only read your abstract. The abstract must have the key results and contributions of the study and be well-written.

Phase IV Pitfalls: Submitting to a Journal

After the paper has been written, it is time to choose the journal. This phase also has numerous pitfalls. Below are pitfalls that can occur during this phase.  

Pitfall: Choosing the wrong journal

Choosing the journal for your work can be overwhelming due to the number of options. Always look at the aims and scope of prospective journals. Look through the author guidelines to ensure that your manuscript adheres. This will save time. Review your reference list for any journals that appear more than once; if so, consider submitting to that journal. You do not want to submit your paper, wait two weeks, and then get a desk rejection because the editors state the paper is not aligned to the journal's aims and scope.

Additionally, researchers can aim too high and spend months (and numerous hours in journal submission portals) trying to publish a manuscript in a journal with a very large impact factor. Though admirable, if the research design and results lacking “gold standard” reporting, authors should consider a journal that is more likely to accept. Find a balance between the quality of your paper and the quality of the journal. Seek feedback from the other authors and/or senior colleagues who can provide honest feedback.

Pitfall: Poor cover letter on journal submission

Do not submit work with a flawed cover letter (errors or lack of clarity in how your work contributes to the body of literature). Spend time writing a detailed cover letter once, have it edited by someone else, and utilize that for all future projects. You can highlight the differences (e.g., the purpose of this work, our results showed) with each project. Use the cover letter to highlight the significance of the study while adhering to the disclosure guidelines (e.g., conflicts of interests, authors contributions, data releases, etc.), which will help the editorial board determine not only the suitability of the paper for the journal but also streamline the review process.[12]

Pitfall: Assuming that after the paper has been submitted to a journal, the work is done           

The paper has been submitted! You think you are finished…but, unfortunately, the publishing game may still be far from over. Researchers often do not recognize the amount of time going into the submission/rejection/revisions phases. Revisions can sometimes be total overhauls, more work than writing a whole new paper. Be prepared to continue working.

Phase V Pitfalls: The Rejections, Revisions, and Acceptance Phase

Finally, perhaps the most unpredictable phase, the rejections, revisions, and acceptance phase, has unique pitfalls and other obstacles.

Pitfall: Mourning rejections too long/ “sitting on” a rejected paper           

Did you get a desk to reject (i.e., the manuscript was not even sent for blind review)? That is unfortunate but common. You do not have time to sulk. Get that paper submitted somewhere else. The older the data, the less desirable your paper becomes. If the paper went in for a full review and was rejected, that may be even tougher than a desk reject because more time has elapsed. The good news is that (hopefully) you received feedback to incorporate in a revision. Do not spend too much time grieving rejections.

Pitfall: Not laying to rest rejected papers when it is indeed their time to go

Did you write a paper a couple of years ago, and you’ve submitted it to 20 different journals? The data is getting old. The topic wasn’t focused on. The sample size was small. Perhaps the project is not worth pursuing any longer. Do not give in to the sunk cost fallacy. If, however, you are proud of the work and stand by the paper, do not give up. If you believe after the numerous rejections that the topic/project is flawed, you can use this failure as a personal learning/growth opportunity. Do not repeat controllable mistakes on future projects.

Pitfall: Not addressing all of reviewer feedback

Did you get a revise and resubmit? Great news! The reviewers and editors will likely ask you to respond to each comment when you resubmit. Address all of the reviewer feedback. Take your time reading through the feedback, digest it, and re-read it. Carefully respond and decide how to revise your manuscript based on the feedback. Share the reviews and the duties of revision with coauthors. In your response to reviewers, stay professional and address each statement, even if you disagree with what is stated. If you do not respond to each statement, the reviewers often highlight the concern(s) again.

Pitfall: Thinking you know what the reviewers are going to say

Research reviewers are like a box of chocolates. You never know what you are going to get. You may be worried about a section of your paper/research approach, and the reviewers do not mention it at all in their review; instead, they criticize a section of your manuscript that you are most proud of.

In some reviews, you may get feedback like the following:

Reviewer #1

Please change lines 104-108 as I believe they are irrelevant to your study.

Reviewer #2

Please build on lines 104-108, as I believe they are the foundation of your study.

Sometimes, after multiple revisions, there are new concerns presented by the reviewers. This can be disheartening. Should some regulations restrict reviewers from bringing up new ideas/concerns during revision #7? Perhaps. Does any current rule prevent them from doing this? No.

During the review process, we must have faith that the reviewers are knowledgeable and provide fair, insightful, and constructive feedback. While the review process can be arbitrary or frustrating in some cases, peer review remains the gold standard in a scientific publication. Stay positive and persistent. Stay professional in responses to the reviewers. Remember that the review process can be very beneficial as it often leads to feedback that truly elevates your work and makes the product (and you) look better.[13]

Pitfall: Not rewarding yourself for a published paper

You did it! Celebrate your accomplishment. Reflect on the merit of your effort before you move on to other work or re-enter the cycle of IRBs, data coding, journal submissions, etc. Remember and appreciate how remarkable it is that you just contributed knowledge to the world.

Clinical Significance

Many pitfalls can occur throughout the research process. Researchers should understand these pitfalls and utilize strategies to avoid them to produce high-quality, sought-after research results that are useful for basic science and clinical practice.