How to conduct effective desk-top research?

This article is for any university student about to embark on writing essays or completing dissertations and projects for the first time. I have also run workshops introducing these methods and they do seem to be overwhelmingly useful even to more experienced researchers. This article is also intended to help dissertation supervisors who may want to produce a ‘mini-systematic review’ for an undergraduate or postgraduate research project. This provides a robust methodology for the students to follow and is a much more rewarding and exacting project than a mere literature review. It will also satisfy requirements of those professional bodies who look for an element of ‘data analysis’ within the project.

So, let us embark on an interesting and hopefully informative journey about how to carry out effective desk-top research.

Introducing the systematic review

The word “systematic” in relation to a review involves the use of precise methods to gather and assess the results of research publications that (most importantly) minimises bias within the process. The result should be a robust and reliable assimilation of evidence in order to reach a reliable conclusion. Medical systematic reviews are conducted and published through the Cochrane Library named after Archie Cochrane a Scottish doctor who established the idea of evidence-based medicine. Why do I mention systematic reviews in relation to desk-top research? Well – if you understand the premise and approaches of a systematic review and apply them to your essays, coursework and dissertations, then you will be undertaking a high quality piece of work (or suggesting a high quality assignment if you are setting the work). The steps highlighted below would also provide you with a methodology and the basis of a methods section for a dissertation.

Figure 1 illustrates the systematic approach. The details on the left hand side are the minimum approach that could be undertaken in an essay or piece of desk-top research. For more in-depth undergraduate projects, and certainly for full systematic reviews, the details on the right hand side would need to be fully understood and reported.

Full systematic reviews can be conducted on any subject, not just medical ones. I have written  ones on education subjects – and here too, they are useful to  pool knowledge about best practice, or to evaluate new innovations in teaching for example. In education, often the methods are more relaxed as generally education papers do not meet the high quality standards of medical papers and their research designs. This is often due to not being able to randomise groups of students / learners due to the constraints of timetabling and classrooms. This isn’t the entire story though, as generally there is a feeling that much medical research and education research is simply not conducted as well as it could be.

Systematic principles – we should all use them!

A full systematic review is a serious piece of research and I like to teach the principles to my university students wherever possible because it provides them with a basis for doing high quality literature reviews, essays and dissertations. In fact I believe that anyone conducting research should know these principles. How many times do we hear that people are just using a Google Search or even Scholar, and they think it is research? The mainstay for any professional research must be the use of peer-reviewed and edited articles, and Scholar will not provide a robust enough search of these, and will also retrieve non-peer-reviewed reports and documents. Interesting as background reading certainly, but not for citation within a professional piece of work.

1) Setting the research question

The formulation of a precise research question is the starting point for any research and can be quite tricky. In medicine the PICO framework is used to define the various elements – population, intervention, comparison and outcome measure. For example I might be interested whether probiotics help people with diarrhoea.

Population – patients with diarrhoea
Intervention – probiotics
Comparison – no treatment
Outcome – alleviation of diarrhoeal symptoms

So a question might be,

In patients with diarrhoea, do probiotics compared to no treatment, alleviate symptoms?

A PICO based question is the starting point of any dissertation student of mine, although not all the categories may apply. Once the question is set, the search strategy evolves and we can start generating keywords around the question categories.

But let’s take an education example. I’m interested in free online learning in the form of massive online open courses – MOOCs and the student experience.

P = learners
I = MOOCs
C = face to face/ traditional learning
O = student experience

The question might be,

Do MOOCs enhance the experience of learners compared to traditional methods?

2) Deciding where to search?

A systematic review will aim to find ALL the articles in the world! This means not just using electronic databases, but hand searching books and journals, and contacting experts for unpublished or ongoing research. This can be quite a time intensive process. Today, the process is greatly helped by being able to save your searches within electronic databases, so once established (e.g. you might run a search at the start of your student project), you can simply run it again at the end to check for recent articles. Be pragmatic with the time you have – you might not be able to search everywhere, and the school of thought is that actually a good search of electronic databases will retrieve you the majority of articles these days, although do take care if you are particularly interested in more historic ones that may not be digitised.

So, where you decide to search will depend on what your organisation or local library has access to. Web of Knowledge and Medline are the mainstays of my research – which is both medical and educational. For my review on MOOCs I also used SCOPUS, IEEE and others. These cover both conference proceedings and workshop proceedings alongside published articles (original research, literature reviews, comments, opinions, letters etc).

3) Building up keyword lists for searching

From our PICO categories, we can start building up lists of keywords on similar themes.

P = learners, students, users
I = MOOCs, xMOOC, cMOOC, massive online open course, free course
C = face to face teaching, traditional teaching
O = student experience, learning gain, knowledge gain

The next step is building up these words further. This is where I recommend using Wikipedia. It is a great keyword generator. I will also run some searches at this point to find relevant studies and look at their keywords to add to the list. If you were carrying out a full systematic review to publish, you would spend some time building up your keywords and then testing the results to ensure you were retrieving relevant articles. This iterative process might go on for some time, although for shorter-time scale projects such as undergraduate work, this may not be desirable.

4) Getting the keywords organised using Boolean logic

In some research I recently conducted looking at massive online open courses – MOOCs – I used six online databases to search, and used Boolean notation for searching with my keyword lists. There is a nice explanation of the use of Boolean logic on Ithaca College Library website. This in its simplest form uses the words (inputed in capitals – AND, OR, NOT) to combine keywords in order to expand and cross-reference your search accordingly. The Figure 2 summarises this approach.

You can also truncate words to search for all the variants of word endings using an asterisk *

e.g. MOOC MOOCs we can search for MOOC*
e.g. Massive or massively we can search for massiv*

If searching phrases these need to be in quotations otherwise the individual words will be searched for separately and return thousands of results.

e.g. “massiv* online open cours*”

I’ve referred to the use of Boolean notation in another blog article – “Seek and ye shall find” complete with webcasts and instructions. This is following very simple principles and those expert in searching and forming Boolean instruction will be more complex than this. Here are some of the more commonly used ‘operators’ or instructions within the notation.

# means search
OR – this will link together keywords and is used to broaden a search
AND – this will cross-reference two searches (and not expand the search as you might suspect)
NOT – this will exclude terms from the search

Going back to our question whether probiotics are effective for patients with diarrhoea, we could just haphazardly search for the keywords as shown below in Figure 3. However, as shown by the numbers, you will retrieve vast numbers of records and your search will not be specifically addressing your question.

The use of Boolean notation can be illustrated by the formation of a Venn diagram which shows the principles of combining the three separate searches using the word ‘AND’. The ‘OR’ term will enable you to expand out your searches such as for probiotics and lactobacillus. You may also search for humans and adults as a focus, and also the disease of interest. By using the ‘AND’ term you are cross-referencing the three searches to find those papers in the centre of the Venn diagram (Figure 4) – you can see a more manageable number of 1312 papers. These of course can be further limited perhaps by searching just for clinical trials.

5) Running the search and being organised!

Organisation is key and many online databases can set up accounts to save your searches (Medline is great for this) or export your outputs to a reference manager. I prefer to sometimes run the search, save the results as a ‘txt’ file and input into Microsoft Excel for analysis. The analysis steps might be important in a systematic review where you have pre-set what your research question is and your criteria for including studies. You can therefore use a new Excel sheet for each step in the analysis to maintain a good record of your process.

If you are completing a full systematic review, you will wish to refine your search in an iterative manner. That is, you will look at your search results to see if they are retrieving relevant articles, and refine the keywords and Boolean strategy if necessary to produce a more precise result. This step can in my experience take far more time than you might realise. The benefit is, once the search is right, you can save it and use it to update your coursework / project or review in the future.

6) What are study inclusion and exclusion criteria? (Could be optional depending on type of project or research)

For an undergraduate project you may not wish to be so stringent to think about what types of studies you wish to include or exclude. If you are completing a literature review, you may want just to provide an overall evaluation of everything that you have found. If you are being more systematic and wish to generate data for your project, you can follow the steps undertaken by a full systematic review, and record the numbers of studies you include and exclude at each phase. The beauty of this within a project or piece of research is that you are generating legitimate research data that can be displayed in a number of established figures and formats as illustrated below in Figure 5. Here, the results of a literature search and numbers of studies that are excluded during the process are shown.

Some excellent details on how to report systematic review results can be found not he following website, describing the PRISMA statements – preferred reporting items for systematic reviews and meta analyses.

In my studies of ‘education’, I set the entry gate quite wide so not to restrict the numbers of studies based on their design and quality. You will need to decide your inclusion and exclusion criteria at the start when you are writing your research proposal, or planning your essay. If you did wish to consider excluding types of articles, you might for example be doing a medical review and may well wish to only include randomised controlled trials. You might be researching an area of biomedical science and wish only to include animal investigations. If you are interested in systematic reviews in education specifically, this is a subject of development and debate the present time (e.g. Bennett 2005).

When you are analysing the results of your searches you will often soon spot ‘duplicate studies’. You will almost certainly find the same study on a number of databases, so you can use the ‘sort’ function to scan your lists of authors and remove duplicates. Studies can be duplicated in more subtle ways, for example an author might publish an abstract of data in a national journal, and then present the data at international conference. These are strictly duplicate studies because they contain the same data. The duplicate will need to be removed as shown in Figure 5.

7) Data collection and analysis

If you are intending to follow a systematic approach you will need to construct a series of spreadsheets to gather and organise your results. If you are completing a full systematic review you will establish the layout of a data extraction table prior to starting the review. This would include items such as author name, date of publication, methodology, outcome measures, and a host of other details. Again, the Cochrane organisation has further details on data extraction.

Sorting your search results and applying the inclusion and exclusion criteria does take time, but ultimately it will give you the good results you are looking for. In a full-scale systematic review a number of authors would do these steps independently to ensure the process is accurate and to avoid bias introduced by personal choices and preferences. A third author can help discuss any areas of conflict or indecision. Filtering of the papers generally occurs in two phases:

Phase 1 of filtering. You can quickly filter your results often by just looking at the titles and author names to identify duplicates. You may need to review the abstracts at this point to ensure they match your inclusion criteria. Anything that is unclear will need to be checked by reviewing the full paper.

Phase 2 of selecting and filtering. If your inclusion criteria is looking for a specific methodology – e.g. randomised controlled trial, or specific subset of articles – e.g. animal studies, if you cannot glean this information from the abstract you will need the full paper to review.

So you might go through a phase of ordering full papers, and again use a reference manager of file system on your computer to organise yourself. I generally obtain the full paper for every article as I go along.

8) Qualitative versus quantitive analysis

For a full systematic review, if you have identified enough studies you can then extract data for pooling in a meta-analysis to provide quantitative data. As part of a review it is also good practice to provide a brief ‘narrative’ of the papers identified, and also to summarise your results in table form. The extent to which you do all of this will depend on the numbers of papers retrieved, and for the purposes of ‘containing’ an undergraduate project within 5000 words which is often the limit, you may need to restrict the textual explanations of the papers.

Providing the ‘narrative’ is often the part that students struggle to do within project result sections, therefore it is worth gaining a deeper understanding of the approaches and styles that can be undertaken. Popay et al in 2006 wrote a report on narrative synthesis that may be a starting point.

9) Finishing off and identifying themes and conclusions

If you have adhered to your question, keywords and inclusion / exclusion criteria, you should end up with a corpus (body of literature) directly relevant to your question. Depending on the volume of papers retrieved you may be able to look for sub-themes and organise your discussion around these. For example, searching for probiotics and diarrhoeal disease may reveal areas of research focusing on children as opposed to adults for example. The research may focus on different types of bacteria or blends of bacteria. A systematic approach is a great way of organising your research from start to finish!

References

Cochrane Library (2014). About Cochrane Systematic Reviews and Protocols. Available: http://www.thecochranelibrary.com/view/0/AboutCochraneSystematicReviews.html

Popay, J., Roberts, H., Sowden, A., Petticrew, M., Arai, L., Rodgers, M., … & Duffy, S. (2006). Guidance on the conduct of narrative synthesis in systematic reviews. A product from the ESRC methods programme. Version, 1.