Papers with ‘interesting’ results are more likely to be
Goldacre, B. (2012) What doctors don't know about the drugs they prescribe, Available [Accessed 2nd June 2014].
Publication bias refers to the fact that studies with more ‘positive’ results – i.e. which show a definite effect for a particular treatment – are more likely (three times more likely in fact) to be published than ones which show little or no positive effect for a treatment.
Researchers themselves often think that studies which show no or little effect for a treatment aren’t worth publishing so these studies are less likely to be submitted for publication – the so-called ‘file-drawer’ problem. Once submitted they are less likely to be accepted by a journal, less likely to be published in a ‘high-impact’ journal and less likely to be published in English making them all the more difficult to find.
Less scrupulous researchers are sometimes known to selectively report the results of trials, reporting results that show treatments in a good light while glossing over ones which show that it is ineffective in a process called HARKing – Hypothesizing After the Results are Known.
Looking for grey literature can unearth the unpublished trials which show interventions in a less-than-glowing light and can significantly affect the outcome of a systematic review. A good example of this is this study into the anti-depressant Agomelatine (see the box on this page).
Howland, R.H. (2011), 'Publication bias and outcome reporting bias: agomelatine as a case example', Journal of Psychosocial Nursing & Mental Health Services, 49(9): 11-4. DOI: 10.3928/02793695-20110809-01
Publication bias and outcome reporting bias contribute to distorted perceptions of drug efficacy and the underreporting of adverse events. To demonstrate these biases, this article describes how the clinical profile of the antidepressant agent agomelatine (Valdoxan) has been presented in the literature. Agomelatine has been systematically assessed in 10 short-term placebo-controlled studies and three long-term placebo controlled relapse prevention studies. Five published trials demonstrated clinically modest but statistically significant benefits over placebo. Five unpublished trials did not find agomelatine more effective than placebo, but in two of these studies the active comparison drug fluoxetine or paroxetine was found to be more effective than placebo.
Agomelatine was more effective than placebo in only one of three relapse prevention studies, but only the positive study was published. Based on what is evident in the entire published and unpublished dataset, agomelatine does not have a tremendously superior sleep and sexual effects profile. The risk of liver toxicity is also not prominently highlighted in the published literature.
A data repository is an archive that manages the long-term storage and preservation of digital resources and provides a catalogue for discovery and access. There are a large number of data repositories some with a general coverage and many that are subject specific. Depending on the subject of your systematic review you may find it useful to consider searching for data. The links below are just a small number of the data repositories that are available. Some funders and journal publishers specify specific data repositories so check the guidelines for researchers in your field to find which ones they recommend.
The King's Research Data Management System is a a research data repository service providing long term storage and public access for datasets that support published research and/or have long term value. Search strategies fall into this category. View the Libraries & Collections Research Support webpage (Preserve > Deposit with King's tab) for more information on how to submit. Please note that this is normally limited to researcher and PhD level.
Other data repositories are also linked from the Libraries & Collections Research Support webpage (Preserve > Deposit your data tab) and some key ones are listed below:
"The term ‘grey literature’ is often used to refer to reports published outside of traditional commercial publishing. Review authors should generally search sources such as dissertations and conference abstracts" Cochrane Handbook v.6.3 - Part 2, section 4.3.5 Other sources.
Formats include:
One of the best ways of searching for dissertations and theses is the WorldCat database produced by OCLC which is available via the university's database page. Follow the links through to WorldCat in the same way as you access other databases. When you get there you will need to select King’s College and log in with your King's user name and password.
Trial Registers are a useful source of unpublished and ongoing trials:
You can search for conference proceedings using a number of the databases available via the Library Services’ web page. On Medline you can search for them by combining a search for Congresses with a subject search for whatever topic you are looking for. You can also restrict your search to conference proceedings when using the Web of Science database.
Hand-searching involves looking through the contents pages of journals, conference proceedings and abstracts page by page. This process can identify articles (and other items, e.g. letters) which have not yet been included in electronic databases, and those which are not indexed or have been indexed incorrectly. Deciding which journals to search through in this way can be done by analysing the results of your database searches to see which journals contain the largest number of relevant studies.
You can also find other studies by looking through the references of papers you have found by searching electronically.
Citation indexes are another extremely useful tool to help you locate papers and other relevant research which may have been missed in your more formal database searches. Consider using a database like Web of Science or Scopus, or undertake a search on Google Scholar, Microsoft Academic, Semantic Scholar or ResearchGate to look at 'Cited by' information for papers you know are of interest and seeing who has cited them since they were published. Some search engines, and journal websites also offer 'recommended for you' links which can also surface research of interest.
You may be able to get hold of further information about unpublished or ongoing research by making contact with study authors, and organisations, e.g. drugs companies and research centres. Experts can also be contacted and shown the list of evidence which you have found to see if they know about any articles you have missed out. If you have a supervisor for your systematic review then remember to check with them to see whether they can spot any key research not included in your results.
Searching Grey Literature to include in your systematic review:
There are a variety of ways you can search grey literature but you need to look in the right places for the right resources. Some databases specialise in technical reports, unpublished research and the work of government departments while others specialise in clinical trials, conference proceedings and theses. Depending on what kind of material you are looking for, what topic you are researching and how much time you have available you might decide to search only one of these databases, a few of them or, if you want to be really thorough, search them all.
If you have located the abstract of a conference paper published in a journal and then indexed in a database then you may wish to follow this up in order to get more information.
Two common further actions you may wish to take:
A preprint is a version of a scholarly or scientific paper that often precedes formal publication. Free preprints may be available before and/or after a paper is published in a journal. There is no guarantee when reading a preprint that the article has undergone formal peer review (or even publication) in a peer-reviewed scholarly or scientific journal, without undertaking further checking for publication details.
A preprint will therefore allow free access to an article by all interested parties regardless of whether they subscribe to a particular journal or not. On some occasions the preprint may not yet have been formally accepted to a journal or undergone peer review but the author may have chosen to make a preprint available in order to quickly circulate some new information or ask for informal feedback from their peers prior to publication.
Preprints are hosted in many places:
Many preprint servers and open access publishing platforms do undertake a basic screening of content uploaded and may check for plagiarism but whilst some offer a post publication peer review option (community mediated peer review) others are simply repositories.