How to conduct effective desk-top research?
This article is for any university student about to embark on writing essays or completing dissertations and projects for the first time. I have also run workshops introducing these methods and they do seem to be overwhelmingly useful even to more experienced researchers. This article is also intended to help dissertation supervisors who may want to produce a ‘mini-systematic review’ for an undergraduate or postgraduate research project. This provides a robust methodology for the students to follow and is a much more rewarding and exacting project than a mere literature review. It will also satisfy requirements of those professional bodies who look for an element of ‘data analysis’ within the project.
So, let us embark on an interesting and hopefully informative journey about how to carry out effective desk-top research.
Introducing the systematic review
The word “systematic” in relation to a review involves the use of precise methods to gather and assess the results of research publications that (most importantly) minimises bias within the process. The result should be a robust and reliable assimilation of evidence in order to reach a reliable conclusion. Medical systematic reviews are conducted and published through the Cochrane Library named after Archie Cochrane a Scottish doctor who established the idea of evidence-based medicine. Why do I mention systematic reviews in relation to desk-top research? Well – if you understand the premise and approaches of a systematic review and apply them to your essays, coursework and dissertations, then you will be undertaking a high quality piece of work (or suggesting a high quality assignment if you are setting the work). The steps highlighted below would also provide you with a methodology and the basis of a methods section for a dissertation.
Figure 1 illustrates the systematic approach. The details on the left hand side are the minimum approach that could be undertaken in an essay or piece of desk-top research. For more in-depth undergraduate projects, and certainly for full systematic reviews, the details on the right hand side would need to be fully understood and reported.
Full systematic reviews can be conducted on any subject, not just medical ones. I have written ones on education subjects – and here too, they are useful to pool knowledge about best practice, or to evaluate new innovations in teaching for example. In education, often the methods are more relaxed as generally education papers do not meet the high quality standards of medical papers and their research designs. This is often due to not being able to randomise groups of students / learners due to the constraints of timetabling and classrooms. This isn’t the entire story though, as generally there is a feeling that much medical research and education research is simply not conducted as well as it could be.
Systematic principles – we should all use them!
A full systematic review is a serious piece of research and I like to teach the principles to my university students wherever possible because it provides them with a basis for doing high quality literature reviews, essays and dissertations. In fact I believe that anyone conducting research should know these principles. How many times do we hear that people are just using a Google Search or even Scholar, and they think it is research? The mainstay for any professional research must be the use of peer-reviewed and edited articles, and Scholar will not provide a robust enough search of these, and will also retrieve non-peer-reviewed reports and documents. Interesting as background reading certainly, but not for citation within a professional piece of work.
1) Setting the research question
The formulation of a precise research question is the starting point for any research and can be quite tricky. In medicine the PICO framework is used to define the various elements – population, intervention, comparison and outcome measure. For example I might be interested whether probiotics help people with diarrhoea.
Population – patients with diarrhoea
Intervention – probiotics
Comparison – no treatment
Outcome – alleviation of diarrhoeal symptoms
So a question might be,
In patients with diarrhoea, do probiotics compared to no treatment, alleviate symptoms?
A PICO based question is the starting point of any dissertation student of mine, although not all the categories may apply. Once the question is set, the search strategy evolves and we can start generating keywords around the question categories.
But let’s take an education example. I’m interested in free online learning in the form of massive online open courses – MOOCs and the student experience.
P = learners
I = MOOCs
C = face to face/ traditional learning
O = student experience
The question might be,
Do MOOCs enhance the experience of learners compared to traditional methods?
2) Deciding where to search?
A systematic review will aim to find ALL the articles in the world! This means not just using electronic databases, but hand searching books and journals, and contacting experts for unpublished or ongoing research. This can be quite a time intensive process. Today, the process is greatly helped by being able to save your searches within electronic databases, so once established (e.g. you might run a search at the start of your student project), you can simply run it again at the end to check for recent articles. Be pragmatic with the time you have – you might not be able to search everywhere, and the school of thought is that actually a good search of electronic databases will retrieve you the majority of articles these days, although do take care if you are particularly interested in more historic ones that may not be digitised.
So, where you decide to search will depend on what your organisation or local library has access to. Web of Knowledge and Medline are the mainstays of my research – which is both medical and educational. For my review on MOOCs I also used SCOPUS, IEEE and others. These cover both conference proceedings and workshop proceedings alongside published articles (original research, literature reviews, comments, opinions, letters etc).
3) Building up keyword lists for searching
From our PICO categories, we can start building up lists of keywords on similar themes.
P = learners, students, users
I = MOOCs, xMOOC, cMOOC, massive online open course, free course
C = face to face teaching, traditional teaching
O = student experience, learning gain, knowledge gain
The next step is building up these words further. This is where I recommend using Wikipedia. It is a great keyword generator. I will also run some searches at this point to find relevant studies and look at their keywords to add to the list. If you were carrying out a full systematic review to publish, you would spend some time building up your keywords and then testing the results to ensure you were retrieving relevant articles. This iterative process might go on for some time, although for shorter-time scale projects such as undergraduate work, this may not be desirable.
4) Getting the keywords organised using Boolean logic
In some research I recently conducted looking at massive online open courses – MOOCs – I used six online databases to search, and used Boolean notation for searching with my keyword lists. There is a nice explanation of the use of Boolean logic on Ithaca College Library website. This in its simplest form uses the words (inputed in capitals – AND, OR, NOT) to combine keywords in order to expand and cross-reference your search accordingly. The Figure 2 summarises this approach.
You can also truncate words to search for all the variants of word endings using an asterisk *
e.g. MOOC MOOCs we can search for MOOC*
e.g. Massive or massively we can search for massiv*
If searching phrases these need to be in quotations otherwise the individual words will be searched for separately and return thousands of results.
e.g. “massiv* online open cours*”
I’ve referred to the use of Boolean notation in another blog article – “Seek and ye shall find” complete with webcasts and instructions. This is following very simple principles and those expert in searching and forming Boolean instruction will be more complex than this. Here are some of the more commonly used ‘operators’ or instructions within the notation.
# means search
OR – this will link together keywords and is used to broaden a search
AND – this will cross-reference two searches (and not expand the search as you might suspect)
NOT – this will exclude terms from the search
Going back to our question whether probiotics are effective for patients with diarrhoea, we could just haphazardly search for the keywords as shown below in Figure 3. However, as shown by the numbers, you will retrieve vast numbers of records and your search will not be specifically addressing your question.
The use of Boolean notation can be illustrated by the formation of a Venn diagram which shows the principles of combining the three separate searches using the word ‘AND’. The ‘OR’ term will enable you to expand out your searches such as for probiotics and lactobacillus. You may also search for humans and adults as a focus, and also the disease of interest. By using the ‘AND’ term you are cross-referencing the three searches to find those papers in the centre of the Venn diagram (Figure 4) – you can see a more manageable number of 1312 papers. These of course can be further limited perhaps by searching just for clinical trials.
5) Running the search and being organised!
Organisation is key and many online databases can set up accounts to save your searches (Medline is great for this) or export your outputs to a reference manager. I prefer to sometimes run the search, save the results as a ‘txt’ file and input into Microsoft Excel for analysis. The analysis steps might be important in a systematic review where you have pre-set what your research question is and your criteria for including studies. You can therefore use a new Excel sheet for each step in the analysis to maintain a good record of your process.
If you are completing a full systematic review, you will wish to refine your search in an iterative manner. That is, you will look at your search results to see if they are retrieving relevant articles, and refine the keywords and Boolean strategy if necessary to produce a more precise result. This step can in my experience take far more time than you might realise. The benefit is, once the search is right, you can save it and use it to update your coursework / project or review in the future.
6) What are study inclusion and exclusion criteria? (Could be optional depending on type of project or research)
For an undergraduate project you may not wish to be so stringent to think about what types of studies you wish to include or exclude. If you are completing a literature review, you may want just to provide an overall evaluation of everything that you have found. If you are being more systematic and wish to generate data for your project, you can follow the steps undertaken by a full systematic review, and record the numbers of studies you include and exclude at each phase. The beauty of this within a project or piece of research is that you are generating legitimate research data that can be displayed in a number of established figures and formats as illustrated below in Figure 5. Here, the results of a literature search and numbers of studies that are excluded during the process are shown.
Some excellent details on how to report systematic review results can be found not he following website, describing the PRISMA statements – preferred reporting items for systematic reviews and meta analyses.
In my studies of ‘education’, I set the entry gate quite wide so not to restrict the numbers of studies based on their design and quality. You will need to decide your inclusion and exclusion criteria at the start when you are writing your research proposal, or planning your essay. If you did wish to consider excluding types of articles, you might for example be doing a medical review and may well wish to only include randomised controlled trials. You might be researching an area of biomedical science and wish only to include animal investigations. If you are interested in systematic reviews in education specifically, this is a subject of development and debate the present time (e.g. Bennett 2005).
When you are analysing the results of your searches you will often soon spot ‘duplicate studies’. You will almost certainly find the same study on a number of databases, so you can use the ‘sort’ function to scan your lists of authors and remove duplicates. Studies can be duplicated in more subtle ways, for example an author might publish an abstract of data in a national journal, and then present the data at international conference. These are strictly duplicate studies because they contain the same data. The duplicate will need to be removed as shown in Figure 5.
7) Data collection and analysis
If you are intending to follow a systematic approach you will need to construct a series of spreadsheets to gather and organise your results. If you are completing a full systematic review you will establish the layout of a data extraction table prior to starting the review. This would include items such as author name, date of publication, methodology, outcome measures, and a host of other details. Again, the Cochrane organisation has further details on data extraction.
Sorting your search results and applying the inclusion and exclusion criteria does take time, but ultimately it will give you the good results you are looking for. In a full-scale systematic review a number of authors would do these steps independently to ensure the process is accurate and to avoid bias introduced by personal choices and preferences. A third author can help discuss any areas of conflict or indecision. Filtering of the papers generally occurs in two phases:
Phase 1 of filtering. You can quickly filter your results often by just looking at the titles and author names to identify duplicates. You may need to review the abstracts at this point to ensure they match your inclusion criteria. Anything that is unclear will need to be checked by reviewing the full paper.
Phase 2 of selecting and filtering. If your inclusion criteria is looking for a specific methodology – e.g. randomised controlled trial, or specific subset of articles – e.g. animal studies, if you cannot glean this information from the abstract you will need the full paper to review.
So you might go through a phase of ordering full papers, and again use a reference manager of file system on your computer to organise yourself. I generally obtain the full paper for every article as I go along.
8) Qualitative versus quantitive analysis
For a full systematic review, if you have identified enough studies you can then extract data for pooling in a meta-analysis to provide quantitative data. As part of a review it is also good practice to provide a brief ‘narrative’ of the papers identified, and also to summarise your results in table form. The extent to which you do all of this will depend on the numbers of papers retrieved, and for the purposes of ‘containing’ an undergraduate project within 5000 words which is often the limit, you may need to restrict the textual explanations of the papers.
Providing the ‘narrative’ is often the part that students struggle to do within project result sections, therefore it is worth gaining a deeper understanding of the approaches and styles that can be undertaken. Popay et al in 2006 wrote a report on narrative synthesis that may be a starting point.
9) Finishing off and identifying themes and conclusions
If you have adhered to your question, keywords and inclusion / exclusion criteria, you should end up with a corpus (body of literature) directly relevant to your question. Depending on the volume of papers retrieved you may be able to look for sub-themes and organise your discussion around these. For example, searching for probiotics and diarrhoeal disease may reveal areas of research focusing on children as opposed to adults for example. The research may focus on different types of bacteria or blends of bacteria. A systematic approach is a great way of organising your research from start to finish!
Bennett, J., Fred Lubben , Sylvia Hogarth & Bob Campbell (2005). Systematic reviews of research in science education: rigour or rigidity?, International Journal of Science Education, 27:4, 387-406.
Cochrane Library (2014). About Cochrane Systematic Reviews and Protocols. Available: http://www.thecochranelibrary.com/view/0/AboutCochraneSystematicReviews.html
Popay, J., Roberts, H., Sowden, A., Petticrew, M., Arai, L., Rodgers, M., … & Duffy, S. (2006). Guidance on the conduct of narrative synthesis in systematic reviews. A product from the ESRC methods programme. Version, 1.
Thank you Facebook for doing what many of us have been trying to do for some time, and that is raising awareness of the important and largely un-held ethical debate regarding the use of personal information and data from the internet in research. The Facebook story bought together three important things:
- A commercial organisation claiming that their terms and conditions cover them to do anything they wish when researching human subjects.
- Authors who didn’t make it transparent whether their research had been reviewed at a university ethical committee, what issues emerged and how they were addressed in the study.
- Journal publishers who seem to take leniency in the ethical nature of their papers rather than upholding high standards of research and review.
“participation wasn’t voluntary, it was enforced, and as a result people’s state of mind may have been manipulated, in my view, without consent” (The Reed Diaries).
Peter Reed rightly suggests that advertising companies have been manipulating us since the dawn of time, and Stephen Downes agrees that commercial companies deal and trade in consumer data to bring us the products and services we would like. Companies constantly experiment with consumer data and “ethics regulations are routinely ignored”.
All of the opinions this week miss other important points regarding the clearly unethical nature of the research conducted. And also, having worked in consumer understanding within a multinational blue-chip company, the statement that we routinely ignored ethics, as did many of my collaborators, for me, is entirely garbage.
What is research?
First, let us separate out what is ‘research’ from what is merely the handling of data. What we are then distinguishing between is research ethics and the ethical and moral values that constitute a commercial organisations degree of social and corporate responsibility. This definition is from the US National Commission for the Protection of Human Subjects of Biomedical and Behavioural Research:
“The term ‘research’ designates an activity designed to test an hypothesis, permit conclusions to be drawn, and thereby to develop or contribute to generalizable knowledge (expressed, for example, in theories, principles, and statements of relationships)”.
Researchers have an obligation to ensure:
“Persons are treated in an ethical manner not only by respecting their decisions and protecting them from harm, but also by making efforts to secure their well-being”.
How are research ethics governed?
When conducting research it is the responsibility of the investigators and ethics boards of universities or organisations (commercial or otherwise) to evaluate the risks and benefits of their work. This is an important step on many levels, to protect the researcher, to ensure the work is of the best quality, and to also provide the required documentation to ensure the participant is fully aware of the risks, hence ‘informed’.
How should these systems work? Regardless of the field of research, whether it is medical, psychological, biomedical, sociological, the responsibility of the researcher is to identify and comply with relevant ethical protocols, and in reality this usually means complying with internal protocols alongside codes of conduct from often more than one professional bodies and organisations.
When I worked in market research, we complied with the codes of practice from the UK Market Research Society. Those companies we worked with, such as those processing store card data, also followed research codes of practice. It has amused me the sweeping statements this week about ALL commercial companies being unethical. I would even go so far to say that for commercial companies, there is much more at stake than for an academic institution. Conducting good ethical research would be directly linked to reputation, and if a company is investing half its R&D budget on a product (the rest on packaging and even more on marketing), the research underpinning it needs to be robust. I suggest that with both universities and companies conducting human research, there is a wide range of ethical standards employed, but we cannot tar everyone with the same brush.
How does all this apply to online research?
The Market Research Society defines online research as:
“Collecting information from a social networking service” and “any other collection of personal data in the online environment for the purpose of research”.
And therefore it follows:
“In accordance with the principle of voluntary informed participation, information identifying respondents (personal data) must not be collected from without their consent”, and “researchers conducting social media monitoring, text analytics or sentiment analysis should take steps to avoid collecting personal data. Researcher must ensure that any personal data is not further processed without consent”.
And ethical codes particularly govern working with children and vulnerable individuals:
“In accordance with the principle of voluntary informed participation, information identifying children must not be collected from forums, social networking sites, blogs, etc without their consent and without obtaining the consent of a parent or responsible adult”.
The flaw in the research published this week, and many other studies like it, is that the risks and benefits of the study were not fully weighed up. The methods of ‘informing’ the participants about such risks and the nature of the work was not in place, and there were no means of ensuring each participant was “truly informed?” Many people think that any information on the internet is up for grabs, and the ‘public private’ argument is a bit of a smoke screen. All internet research requires ethical consideration. Fact.
With research on Facebook, Twitter, MOOC platforms and other social networking services, the answer appears far to simple. The Terms and Conditions increasingly seem to out trump the need for any ethical discussion. This is a horrific situation, and if it persists, it gives anyone researching carte blanche to experiment with human subjects, including possibly minors or vulnerable people.
So what were the risks?
So we have established that the study was a deliberate experiment with a hypothesis that had considered the recruitment of individuals into a study. In January 2012, “participants were randomly selected based on their User ID. The experiment manipulated the extent to which people (N = 689,003) were exposed to emotional expressions in their News Feed”. (Kramer et al 2014).
The outcome was they had manipulated people’s emotional thoughts through a social network, not a new finding, but a study done on a large scale none the less. So what might the risks have been to individuals?
The UK Office of National Statistics in their ‘Social Trends 41 for Health’ analysis in 2009/2010 shows that 1 in 10 adults (11%) of England were diagnosed with depression. That is 11% of adults within that year, not the prevalence over a lifetime which is much higher. Over the preceding decade, prescriptions of antidepressants increased by 334% in England. You will find similar figures on the internet elsewhere, so assuming therefore that a fair proportion of study participants might have been depressed, what might the impact on them have been, and what might the impact on non-depressed individuals have been over the research period? Well both negative thoughts, and levels of rumination and dwelling on negative thoughts are measured outcomes in studies of depression; research suggests negativity may or may not contribute to it, and work in this field is generally in its infancy (Pasyugina 2014, Takano 2014).
Within this population there would undoubtedly have been risks. Individuals were not fully informed of the implications and that they were participating in an experiment, with none of the affordances of being free to withdraw at any time. But it doesn’t matter because they gave their consent right?
What do the Facebook terms cover?
The authors claim that signing up to a Facebook account constitutes informed consent for this research. At least they made passing mention of informed consent. In another interesting Facebook clinical study looking at using the service as an enhancement to therapy there was no mention of ethical approval within the publication that had conducted research on vulnerable participants.
So, searching through the “Full Data Use Policy” that they refer to, I couldn’t find any reference to the statement made by the authors in their paper, that creating an account constitutes informed consent. Maybe I just missed it, but this hardly makes for users of Facebook being fully or truly informed that research might be taking place?
In all of the terms I could only find ONE mention of the word ‘research’ and how Facebook uses your information:
“for internal operations, including troubleshooting, data analysis, testing, research and service improvement”.
It also states to what extent they share their data with others, and I’m assuming here that they shared their data with the researchers in order for them to analyse it and write up the study.
“While you are allowing us to use the information we receive about you, you always own all of your information. Your trust is important to us, which is why we don’t share information we receive about you with others unless we have:
received your permission;
given you notice, such as by telling you about it in this policy; or
removed your name and any other personally identifying information from it.”
I suspect the 600,000 odd participants were not invited to provide their permission.
Where does this leave us?
Unfortunately we cannot stop social networks and corporations setting their terms and conditions to suit their commercial needs. I’m not sure whether they can legitimately ‘out trump’ the need for any ethical approval governing research it might undertake because these are legal terms, not philosophical arguments. Facebook can simply amend their terms now to cover research, and make their mechanisms of informed consent more explicit, but academics and publishers can still commit to good quality research? Surely, those researchers who published this week had some moral and ethical conscience, or were they gilded by the prospect of funding and reputation-boosting work?
Academic publishers should surely want to uphold the standards of the articles that they produce, and to uphold the rigours of the peer-review process? Some journals do send papers back that haven’t been ethically considered, and these stances are far more rigours in science and medicine than in other subject areas.
What do the public think?
In all this, nobody knows what the public thinks. We know what opinion makers on social networks think, and we know what some researchers like myself might think. A few conversations with my friends reveals that it is just generally accepted that everyone gathers your personal data anyway. What they didn’t understand were the implications of the research that might have led to risks to the participants.
The situation is that some less ethical corporations allow their socio-ethical boundaries to gradually creep forwards, and it is only when we look back at how far they have been stretched that we get a perspective of the distance of travel. My personal view is if academic researchers and academic publishers do not start contributing to the debate and to govern the quality of research, then who will?
One friend drew parallels with the ‘frog in the pot’ experiment. Don’t worry, no frogs were harmed in the making of the video.
Just completing a research marathon that started last summer looking into open online courses and both the social and ethical considerations for learners.
The work has included:
A systematic review of literature up to 2013 and a subsequent update to June 2014.
Narrative synthesis of opinion articles in the area of study.
Deeper exploration of opinions through a series of interviews.
I’ve just produced a diagram to summarise the findings to date. I’ve used some of the dimensions of Khan’s 2003 eLearning Framework which preceded open courses in the form we know then today. Many of the original dimensions are still relevant, some need extending, and other new areas for ethical consideration emerge from the results of my work. I hope this gives communities, institutions and those involved with open education on the web some ideas for the types of discussions that are worthy of having, if we are truly going to contribute to open education globally in an equitable and fair way.
Ethical Dimensions of MOOCs by Vivien Rolfe is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Based on a work at https://vivrolfe.com/uncategorized/ethical-dimensions-of-moocs/.
Alan just tweeted a tot up of the Second Life education publications profile that came from a paper just out. He made me wonder where we are with MOOCs?
There have been two systematic reviews of MOOC literature – the first by Liyanagunawardena et al (2013) “MOOCS a systematic study” that retrieved literature up to 2012, and the second that I presented at OpenEd “MOOC research on student experience” retrieving literature up to the autumn of 2013.
Search strategy ROLFE – keywords targeting “MOOCS +student experience, social responsibility”.
Search strategy LIYANAGUNAWARDENA – keywords broad “MOOC”
Total publications – for both reviews this included peer-reviewed journals, conference, authoritative reports, comment / opinion articles and case studies. For L’s review it also included magazine articles.
Journals / conf. – total numbers of articles in peer-reviewed journals or conference proceedings. At this point these numbers will include literature reviews and case studies.
You can draw you own conclusions. For me having done systematic reviews on education subjects before the low numbers of empirical studies is never much of a surprise, and certainly within the MOOCature in 2014, more studies have subsequently been published.
Of the empirical studies that I found, all were cross-sectional analyses of student opinions though either questionnaires or interview. Only one was a comparative study looking at the experiences of two groups of participants.
LIYANAGUNAWARDENA, Tharindu Rekha; ADAMS, Andrew Alexandar; WILLIAMS, Shirley Ann. MOOCs: A systematic study of the published literature 2008-2012. The International Review of Research in Open and Distance Learning, [S.l.], v. 14, n. 3, p. 202-227, jun. 2013. ISSN 1492-3831. Available at: <http://www.irrodl.org/index.php/irrodl/article/view/1455/2531>. Date accessed: 19 Jun. 2014.