Searching for Systematic Reviews & Evidence Synthesis: Managing References & Software Tools

Many systematic reviewers choose to use reference management/bibliographic software to manage the references that they retrieve from the different databases. There are many different programmes that you can use, each with strengths and weaknesses.

NHS staff who are alumni of King's College London will have alumni access to RefWorks (if you are an alumni student email libraryservices@kcl.ac.uk for details of how to access Refworks). 

Other programmes which you may hear of include Mendeley, Zotero, EndNote Web, CiteULike and Papers for the Mac. Many of these are free (although you may be asked to pay for some advanced features).

The King's Guide to Referencing has a short quiz you can take to get a suggested piece of referencing software:

Key features for systematic reviewers:

• Deduplication - you are very likely to retrieve the same articles from different databases that you search. A paper in the journal Nature, for example is likely to be included in Medline, Embase and others. It is very useful to be able to use the remove duplicates feature so that you only have to sift through each result once.
• Ability to add full text and store PDFs (or at least link to it) for the articles which are included after the initial sift.
• Ability to share the references/export them to Excel, for example, if you are going to have multiple people sifting through the results.
• Ability to edit the reference in some way/group in folders if you wish to record your decisions about inclusion/exclusion of papers.
• Works with Word (or your chosen word processor) to quickly format citations in the reference style of your choosing when you are writing up.

A recent article comparing software/methods :

McKeown, S., Mir, Z.M. Considerations for conducting systematic reviews: evaluating the performance of different methods for de-duplicating references. Syst Rev 10, 38 (2021). https://doi.org/10.1186/s13643-021-01583-y

The rapid development of artificial intelligence has enabled the launch of many new screening tools. The screening process of a systematic or scoping review is where eligible studies/research are identified which should be included and analysed in the review. This can often be done in 2 stages; first a screen looking only at the title and abstract of papers, followed by a second screen of those articles which passed the first screen where the full-text of the paper is reviewed. This is an intensive process and PRISMA indicates that at least 2 reviewers should be involved in the screening step to minimise bias. Screening tools aim to reduce the amount of time that reviewers have to spend screening.

Research has been done on the various screening tools to suggest which might be best in different contexts:

Qiyang Zhang & Amanda Neitzel (2023) Choosing the Right Tool for the Job: Screening Tools for Systematic Reviews in Education, Journal of Research on Educational Effectiveness, DOI: 10.1080/19345747.2023.2209079

• This review identified and evaluated 12 screening tools for educational reviewers. They encourage researchers to report their tool usage in publications and select tools based on suitability instead of convenience.

Harrison, H., Griffin, S.J., Kuhn, I. et al. Software tools to support title and abstract screening for systematic reviews in healthcare: an evaluation. BMC Med Res Methodol 20, 7 (2020). https://doi.org/10.1186/s12874-020-0897-3 

• This review identifies and evaluates the usability of software tools that support title & abstract screening for systematic reviews within healthcare research.

When reading systematic reviews you may see mention of two tools commonly used by Cochrane systematic reviewers and others: Covidence and RevMan. If you are undertaking a systematic review you may have them recommended to you by a supervisor or colleague. Whilst King's does not have an institutional subscription to these tools at this stage there are free trial versions available to download and some departments/divisions or research groups may have purchased access independently. Please note the Library is not able to support with the use of these tools.

• Covidence is one of Cochrane’s recommended tools to support Cochrane Reviewers in some of the most labour-intensive stages of their systematic review. Covidence allows a team to upload search results, screen abstracts and full text, complete data collection, conduct risk of bias assessment, resolve disagreements and export data into RevMan or Excel. It is compatible with bibliographic software such as EndNote.
  • Access a free trial of Covidence (note restrictions). 
    • King’s College London – Florence Nightingale Faculty of Nursing, Midwifery & Palliative Care have a research staff faculty-level subscription.
  • Covidence Resource Hub including 'Data Extraction for Intervention Systematic Reviews' ebook; Quick-start guide and 'Step-by-step guide to completing your Systematic Review'.
• RevMan is the software used for preparing and maintaining Cochrane Reviews. RevMan facilitates preparation of protocols and full reviews, including text, characteristics of studies, comparison tables, and study data. It can perform meta-analysis of the data entered, and present the results graphically. 
  • RevMan 5 (free to download) is not being developed and will be unavailable for download after April 25, 2023.
  • RevMan Web, replacing RevMan5 is also now available to non-Cochrane authors but requires a pay-for subscription. See information about free trial (with limited functionality).

Many systematic reviews (particularly ones which are smaller scale than the standard Cochrane ones) are successfully completed without using these tools and there are also other products available. If you would like to explore a free tool you may find the following to be of interest (please note we cannot offer support on using this system):

• Rayyan is an application tailored for researchers conducting systematic literature reviews (SLRs) and meta-analyses. The platform serves as a centralized hub for the entire review process to assist reviewers efficiently manage, screen, and analyze scholarly literature.. The application is designed to enhance collaboration, streamline workflow, and ensure the rigor and reproducibility of systematic reviews, providing a comprehensive solution from importing articles to screening, data extraction, and analysis.
  • Rayyan Help Centre

This paper is part of a series of methodological guidance from the Cochrane Rapid Reviews Methods Group. Rapid reviews (RRs) use modified systematic review methods to accelerate the review process while maintaining systematic, transparent and reproducible methods. This paper guides how to use supportive software for RRs.

"We strongly encourage the use of supportive software throughout RR production. Specifically, we recommend (1) using collaborative online platforms that enable working in parallel, allow for real-time project management and centralise review details; (2) using automation software to support, but not entirely replace a human reviewer and human judgement and (3) being transparent in reporting the methodology and potential risk for bias due to the use of supportive software."

Affengruber L, Nussbaumer-Streit B, Hamel C, et al. Rapid review methods series: Guidance on the use of supportive software. BMJ Evidence-Based Medicine Published Online First: 19 January 2024. doi: 10.1136/bmjebm-2023-112530