Skip to main content

Searching for Systematic Reviews: Grey Literature

This guide brings together information and guidance on effective searching for journal articles and grey literature for those undertaking a systematic review.

Publication bias

Papers with ‘interesting’ results are more likely to be

  • Submitted for publication
  • Accepted for publication
  • Published in a major journal
  • Published in the English language

Goldacre, B. (2012) What doctors don't know about the drugs they prescribe, Available [Accessed 2nd June 2014 2014].

Publication bias refers to the fact that studies with more ‘positive’ results – i.e. which show a definite effect for a particular treatment – are more likely (three times more likely in fact) to be published than ones which show little or no positive effect for a treatment.

Researchers themselves often think that studies which show no or little effect for a treatment aren’t worth publishing so these studies are less likely to be submitted for publication – the so-called ‘file-drawer’ problem. Once submitted they are less likely to be accepted by a journal, less likely to be published in a ‘high-impact’ journal and less likely to be published in English making them all the more difficult to find.

Less scrupulous researchers are sometimes known to selectively report the results of trials, reporting results that show treatments in a good light while glossing over ones which show that it is ineffective in a process called HARKing – Hypothesizing After the Results are Known.

Looking for grey literature can unearth the unpublished trials which show interventions in a less-than-glowing light and can significantly affect the outcome of a systematic review. A good example of this is this study into the anti-depressant Agomelatine (see the box on this page).

Grey Literature resources/Databases

Other internet resources

Agomelatine: an example of publication bias

Howland, R.H. (2011), 'Publication bias and outcome reporting bias: agomelatine as a case example', Journal of Psychosocial Nursing & Mental Health Services, 49(9): 11-4. DOI: 10.3928/02793695-20110809-01

Publication bias and outcome reporting bias contribute to distorted perceptions of drug efficacy and the underreporting of adverse events. To demonstrate these biases, this article describes how the clinical profile of the antidepressant agent agomelatine (Valdoxan) has been presented in the literature. Agomelatine has been systematically assessed in 10 short-term placebo-controlled studies and three long-term placebo controlled relapse prevention studies. Five published trials demonstrated clinically modest but statistically significant benefits over placebo. Five unpublished trials did not find agomelatine more effective than placebo, but in two of these studies the active comparison drug fluoxetine or paroxetine was found to be more effective than placebo. 

Agomelatine was more effective than placebo in only one of three relapse prevention studies, but only the positive study was published. Based on what is evident in the entire published and unpublished dataset, agomelatine does not have a tremendously superior sleep and sexual effects profile. The risk of liver toxicity is also not prominently highlighted in the published literature. 

Data repositories

A data repository is an archive that manages the long-term storage and preservation of digital resources and provides a catalogue for discovery and access. There are a large number of data repositories some with a general coverage and many that are subject specific. Depending on the subject of your systematic review you may find it useful to consider searching for data. The links below are just a small number of the data repositories that are available. Some funders and journal publishers specify specific data repositories so check the guidelines for researchers in your field to find which ones they recommend.

What is Grey Literature?

“There are many definitions of grey literature, but it is usually taken to mean literature that is not formally published in sources such as books or journal articles.” Cochrane Handbook - 6.2.1.8  Grey literature databases.

Formats include:

  • Registered Controlled Trial Registers
  • Technical or research reports from government agencies
  • Reports from scientific research groups
  • Working papers from research groups or committees
  • Doctoral (PhD) dissertations
  • Some conference proceedings and official publications
  • Preprints (journal articles not yet peer-reviewed and/or published)

PhD Theses and Dissertations

One of the best ways of searching for dissertations and theses is the WorldCat database produced by OCLC which is available via the university's database page. Follow the links through to WorldCat in the same way as you access other databases. When you get there you will need to select King’s College and log in with your King's user name and password.

 

 

Controlled Trial Registers

Trial Registers are a useful source of unpublished and ongoing trials:

Conference Proceedings

You can search for conference proceedings using a number of the databases available via the Library Services’ web page. On Medline you can search for them by combining a search for Congresses with a subject search for whatever topic you are looking for. You can also restrict your search to conference proceedings when using the Web of Science database. 

Hand-searching and scanning reference lists

Hand-searching involves looking through the contents pages of journals, conference proceedings and abstracts page by page. This process can identify articles (and other items, e.g. letters) which have not yet been included in electronic databases, and those which are not indexed or have been indexed incorrectly. Deciding which journals to search through in this way can be done by analysing the results of your database searches to see which journals contain the largest number of relevant studies.

You can also find other studies by looking through the references of papers you have found by searching electronically.

Citation indexes are another extremely useful tool to help you locate papers and other relevant research which may have been missed in your more formal database searches. Consider using a database like Web of Science or Scopus, or undertake a search on Google Scholar, Microsoft Academic, Semantic Scholar or ResearchGate to look at 'Cited by' information for papers you know are of interest and seeing who has cited them since they were published. Some search engines, and journal websites also offer 'recommended for you' links which can also surface research of interest.

 

Contacting authors, experts, and other organisations

You may be able to get hold of further information about unpublished or ongoing research by making contact with study authors, and organisations, e.g. drugs companies and research centres. Experts can also be contacted and shown the list of evidence which you have found to see if they know about any articles you have missed out. If you have a supervisor for your systematic review then remember to check with them to see whether they can spot any key research not included in your results. 

Why is Grey Literature important?

Searching Grey Literature to include in your systematic review:

  • gives currency to your review as upcoming research can be located, for example in conference proceedings, preprints or trial registers, which show research which is ongoing but hasn't yet been formally published.
    • PhD theses can be either at the cutting edge of research and/or in a niche or esoteric area where there might not be much (if any) published research.
    • Conference proceedings – either a paper, presentation or just a poster – can often be the first place where new developments or breakthroughs are announced
  • helps avoid positive results publication bias (negative results are less likely to be published in a formal journal but may have been reported at a conference). 
  • acts as a double check – it may allow you to find important published research which has been missed in your database searches (allowing you to edit your search strategy to improve results if there is time or simply report as one located during the grey lit search).
    • One example was where a search in Medline for RCTs on dietary interventions during pregnancy failed to return a relevant result because the abstract mentioned “lifestyle interventions” only so the keyword search failed and the MeSH headings also did not pick up the specific ‘diet’ element in the indexing. However, the diet intervention was highlighted on the clinicaltrials.gov website and so when this was searched it flagged up that results had been published and gave a link to the abstract.

Guide to searching Grey Literature

There are a variety of ways you can search grey literature but you need to look in the right places for the right resources. Some databases specialise in technical reports, unpublished research and the work of government departments while others specialise in clinical trials, conference proceedings and theses. Depending on what kind of material you are looking for, what topic you are researching and how much time you have available you might decide to search only one of these databases, a few of them or, if you want to be really thorough, search them all.

Grey Literature Guides

Exercise on Grey Literature

Finding out more details - Conference Proceedings

If you have located the abstract of a conference paper published in a journal and then indexed in a database then you may wish to follow this up in order to get more information.

Two common further actions you may wish to take:

  • See if there is a website for the conference in question (search the internet for the name of the conference) as sometimes you find archived information over and above what is included in abstracts which appear in journals.
  • Search for the author(s)/their institution (this information is normally included under author affiliation in the abstract) to locate a list of their publications.
    • Some institutions have an institutional repository where conference posters or fuller information than an abstract provides may have been stored.
    • Many institutions have profiles for academics where they list their publications (or link to a full list on a site like ORCID)
      • This can be useful to see whether the conference proceeding has been written up later as a full article (just in case you have not located this in your database searches).
    • You could also consider contacting the author directly to see if they can send you some further information/are happy to discuss their research with you with regard to the specific findings you are interested in.
    • Sometimes authors have moved institutions so it becomes a bit of a mission to track them down – LinkedIn, ResearchGate or Author ID systems where academics create profiles (including ResearcherID on Web of Science and ORCID) are all sites which can bring together all publications for an author and hopefully a link to the current institution.

Preprint servers


A preprint is a version of a scholarly or scientific paper that often precedes formal publication. Free preprints may be available before and/or after a paper is published in a journal. There is no guarantee when reading a preprint that the article has undergone formal peer review (or even publication) in a peer-reviewed scholarly or scientific journal, without undertaking further checking for publication details.
A preprint will therefore allow free access to an article by all interested parties regardless of whether they subscribe to a particular journal or not. On some occasions the preprint may not yet have been formally accepted to a journal or undergone peer review but the author may have chosen to make a preprint available in order to quickly circulate some new information or ask for informal feedback from their peers prior to publication. 
Preprints are hosted in many places:

  • Institutional repositories e.g. King's Research Portal
  • Preprint databases or archives or servers e.g.  arXiv.org; (Physics, Mathematics, Computer Science, Quantitative Biology, Quantitative Finance, Statistics, Electrical Engineering and Systems Science, and Economics); biorxiv (Biology); SSRN (multi-disciplinary including Medicine, Biology and Chemistry); OSF Preprints (covers wide subject area across arts and humanities, social sciences, life sciences and medicine).
  • Open access publishing platform e.g. Wellcome Open Research; F1000 Research

Many preprint servers and open access publishing platforms do undertake a basic screening of content uploaded and may check for plagiarism but whilst some offer a post publication peer review option (community mediated peer review) others are simply repositories.