Blog Education Research

Research Quality’s Scandals April 2014

Research Quality Scandal

(From George White’s Scandals of 1935)

Here we are again with the “Research Quality’s Scandals of 2014“. Those of you awake out there might notice that George White performed his “scandals” or reviews on an annual basis, you will note that on my blog there is ample evidence to write about “research scandals” much more frequently.

Why today?

Well I was anticipating a nice morning writing an abstract for a forthcoming conference (Open Ed 14). As usual, once I started to delve into the literature, practically none of it was accessible at all. Maybe this was a typically bad day, but not really. This experience was after more or less two hours of research and finding numerous interesting papers, and then not being able to openly access a single one of them. Rather ironically the subject was “open learning”.

This really is so bad, but why?

There are many people who dedicate their life to education. Ha. I wouldn’t be sitting here on a Bank Holiday corresponding with students and doing research if I wasn’t one of them. Few of us actually have time to conduct good quality research, and even fewer have the luxury of any funding these days to support their research.

So I find this so deeply annoying because research is so sparse and so precious, particularly anything of good quality, and when you do find it you cannot access it! I do not want to pay $39 and my institution doesn’t subscribe to much of it. Come on, a paper from 1970 surely could be released into the public domain now couldn’t it? Apart from the immorality of it all, that tax payer and public money has funded the work and the livelihoods of the researchers, and then it becomes a commodity to sell on.

And what if you don’t pay for it?

Publication bias is huge – not just from the predominance of publishers and authors only writing about “positive” results, but increasingly these days, people will only write about what they can access for free. So all that stuff hidden behind paywalls is excluded from the discussions and thinking.

Ultimately we are wasting time and re-inventing the wheel, and we are always blaming ourselves for this! I think it is criminal to not learn from previous knowledge. I really do want to know what Roger Lewis wrote about open learning (“What is open learning“?) because there were discussions in the 1970’s about distance learning and the advent of open learning that are important for today’s online learners. This was one of many papers that I found this morning, and I can’t read any of them.

Will the future look brighter?

Well policy is moving slowly in support of open access but that isn’t going to help the back catalogue of papers that are closed off to those that don’t have the money to buy them.

In a recent presentation I talked about changes to the UK research assessment exercise and funding body specifications toward open access.

As for my imminent future, it will be muddy and beer-shaped as I ditch the work and walk across the fields to the pub. 🙂



Blog Education Research

Research Quality’s Scandals 2014

Richard Smith: Medical research—still a scandal!

Richard wrote a great article on the BMJ blog reflecting on a previous editorial by statistician Doug Altman 20 years ago, and how little had changed in the quality of medical research. I often ponder this myself, not just within science and medicine, but education research where I tend to hang out these days.


(From George White’s Scandals of 1934)


Altman’s original reasoning was that researchers feel compelled for career reasons to carry out research that they are ill equipped to perform, and nobody stops them. Ethics committees, who approve research, were ill equipped to detect scientific flaws, and many journals lacked statistical skills and published misleading research. He suggested back then that we needed less but better quality research.

Richard observes that 20 years on, Altman’s article could be published again unchanged, with the observations that poor methodological quality and researchers who chase careers and are under high pressure to publish, are in fact part of the problem today .

I agree with all of that, but also think this.

It is pretty safe to say, if you set a target, then people will work towards that target. Research excellence frameworks are geared around publications, and there is no getting away from the fact that papers are the academic currency for research returns and for promotion. Anyone interviewed for an academic post will be asked if they are REFable? I think the judgment that we are all career-crazed is a bit harsh; most researchers I know are just very passionate (and slightly mad) about their area of work and are just keen to share the results by publishing. Saying that I do know folk who withhold their papers from publishing to meet REF targets.

What else might be part of the problem?

I can understand publishers wanting articles that make great discoveries, but this biases the literature toward positive results (part of publication bias to which we all contribute); but just as important are the things that don’t work. I can think of one of my own which didn’t advance our understanding of fish oil as a therapy for inflammatory bowel disease in children, but did apply novel methods of fluorescence confocal microscopy for studying human intestinal biospies structurally and functionally (based on Naftalin’s work that we adapted for human). The paper was “not interesting enough”.

Colon Surface with BODIPY











Uninteresting intestine – stained for f-actin with BODIPY, Viv Rolfe CC BY SA.

What about quality assurance for research?

I’d be the last to advocate yet another series of measures or frameworks, but there is no quality assurance for research methodologies within universities or hospitals, and that is not the function of the ethical committee I feel. Lordy, I have already known NHS ethics to take up to two years to be granted in some cases, so we don’t want to add to the process. As a researcher if you are lucky, you can be allied to a team where much of your methodology learning can take place, but I can’t remember at any time in my PhD or PostDocs being taught about experimental design or the correct statistics. Some of my best learning has been through journal peer-review – that is, receiving comments from accomplished reviewers, and you soon learn to be a hard nut to take it on the chin.

What about education research?

The problem for me as a novice moving from science to education research is that many of the really informative papers are locked behind paywalls. I am trying to write a research methods paper with bioscience colleagues at the moment, and a number of seminal references are not openly available. Surely, items so fundamentally important to the quality of “our business” should be openly available to be accessed by all? Who in reality these days is inclined to wait 2-3 weeks for an article to be purchased through intra-library loans? We just don’t do it, so our learning is not made easy.

What do we teach to science and medical undergraduates? On my Medical Science degree at De Montfort Univeristy I had modules on “Health Informatics” that looked at data retrieval, search strategies, and appraisal of medical research. A second module on “Evidence Based Medicine” covered clinical trial design, publishing industry, critical appraisal, systematic review and meta analysis. Come to think of it, Medical Sciences also had a “Medical Statistics” module and this culminated in a final year project. I have never come across another EBM module taught to undergraduate scientists.

There is good news for medical research!

The good news for medical research is at least there is a reasonable number of good quality studies that are published. If you were in the field of education research, that would be a different matter. In my recent systematic review of how massive online open courses (MOOCs) support the student experience, I started with 141 potential articles that had investigated the subject, of which only 25 were empirical studies, of which only 1 had a control group so could be deemed of any reasonable quality (Rolfe 2013).

If this were a medical systematic review, none of these studies would have passed the quality checks, although it is acceptable to broaden the entry gates for education research to encompass the practicalities of the classroom (Evans and Benefield 2001), and many other articles have discussed this. In another systematic review by Douglas Gray and myself, 176 studies were identified through the keyword search and were whittled down to just 38 empirical studies of which 21 were excluded because of having no control group, or due to having crucial data being missing. The concept that papers publish even without including numbers of participants, or results of statistical tests, is quite amazing!

Just to convince myself I’m not going mad, here is one further systematic review led by Barbara Means in 2010 at the US Department of Education that looked at literature to compare blended learning with face-to-face approaches, and were there benefits to the learner?

The article states:

The most unexpected finding was that an extensive initial search of the published literature from 1996 through 2006 found no experimental or controlled quasi-experimental studies that both compared the learning effectiveness of online and face-to-face instruction for K–12 students and provided sufficient data for inclusion in a meta-analysis. A subsequent search extended the time frame for studies.


What can we do about it?

“My confidence that things can only get better has largely drained away” says Richard. I don’t think I’m quite at that point although the water might be starting to swirl toward the plug hole!

In our Bioscience community (which will never close), the Higher Education Academy runs research workshops and events that are always well attended. For our subject there is an active community on email. A group of us are working on a guide to getting started in research, although our reviewers suggest we aim for a higher academic and theoretical starting point than originally intended, which is exactly what we didn’t want to do as picking up the language of research is often a difficult starting point.

“OPEN” is the way forward I am sure, with more:

  • Open access articles for basic methodology papers
  • Open educational resources instructing researchers in how to get started
  • Open educational resources and research method modules for our students
  • On-line communities working together

Now there is an idea! I might have to put my money where my mouth is now.

And for some of the real George White’s Scandals of 1935, go to YouTube! Now HERE is what I call a stage entrance.









Blog Education Research Learning & Teaching Study skills

Introduction to ethics for university undergraduate students

An introduction to ethics

Series of open educational resources openly licensed for your use (CC BY SA unless otherwise stated). This page includes:

  • Screencast part 1
  • Screencast part 2
  • PDF of slides
  • MOCK paperwork (participant information sheet, consent form and ethics checklist).

Part 1

Part 2 

Access to PDF of the slides
Introduction to research ethics SLIDES compressed Oct2013

Supplementary material

MOCK Consent Form October2013

MOCK Ethical Reviewer Form October2013

MOCK Participant Info Sheet October2013

Blog Education Research Study skills

Seek and ye shall find.


  • Any student completing coursework essays
  • Students completing research dissertations

Anybody completing more professional research, in depth studies, systematic reviews should seek the help of library services to develop their search strategy. The resources on this page are the types of things I’d teach to new students when tackling an essay for the first time at university, just to give a leg up from looking for stuff on Google.

By watching these tutorials you should be able to:

  • Understand what a search strategy is
  • Compile lists of search keywords
  • Conduct searches using Boolean terms
  • Use PUBMED for conducting your searches (medical and health database)


Conducting the search using the MeSH terms
(Go to >>

This video from NCBI (the PUBMED people) shows how to specifically use the MeSH terms. These are “medical subject headings” and when author’s submit papers to journals, they will provide a list of keywords. As noted in the video above, this introduces some variability. The PUBMED cataloguers will look at every publication and the author keywords, and allocate the paper to the medical subject heading or subheading. You will see how exhaustive these lists are by looking at the MeSH page. Using the MeSH as the search field will be a more precise way of searching and essential for large scale subject areas. If you are not finding many results, I’d switch to “all fields”.

Boolean terms used for searching include these below and also a far more exhaustive list as you become more competent.

OR or +

Web of Knowledge will search PUBMED and a number of other databases (WEB OF SCIENCE, BIOSIS, JOURNAL CITATION REPORTS). The use of keywords and Boolean terms applies in exactly the same way.

Education Research Learning & Teaching

1 Introduction to your research project

Hello from me!

Summary of project ideas

1) Literature and data analytical review of the link between gut dysfunction and manifestation of neurodegenerative disorders. Emerging clinical and laboratory research suggests that for example gut inflammation (cytokines, oxidate stress factors) drive degeneration and accelerate disease progression.
Devos D et al (2013). Colonic inflammation in Parkinson’s disease. Neurobiol Dis (50), 42-48. Available:

2) “Mini”-systematic review and meta analysis of a subject of your choice. Learn more about evidence-based medicine. E.g. is vitamin C really helpful in colds (you will be surprised). Can artichoke leaves reduce cholesterol levels (maybe!). Do antioxidants really do anything clinically? (Hmmm).
Bjelakovic  G et al (2012). Antioxidant supplements for prevention of mortality in healthy participants and patients with various diseases. Cochrane Database of Systematic Reviews  2012 (3). Available:

3) If you are considering teaching as a career you will probably wish to consider a more relevant and education-based project. This will give you an opportunity to develop qualitative and quantitative research skills (hugely useful in every walk of life actually). I’d be happy to discuss and tailor something to your interests.

4) Physiology-laboratory projects – depending on numbers and the availability of the room. This would be of interest to me – looking at the effects of music on physiological parameters, exercise performance and perceptions of exercise.
Jarraya M et al (2012). The effects of music on high-intensity short-term exercise in well trained athletes. Asian J Sports Med 3(4): 233–238. Available:

OR – consider validating or using some of the medical APPS on the iPAD. There is a great one that measures snoring and is used to diagnose sleep apnoea.

5) Your own choice! You may have done a placement and have data already. GREAT

What to do next?

Book an appointment to come and see me! Use the Engage in Research website to understand how to get started and read all about the research process (which is the same regardless of what type of project you are doing. A literature review is not a cop out – it is a serious piece of work with strict methodologies and ways of analysing data too 🙂 )
As the video says, try and think of an idea and come up with a RESEARCH QUESTION as the first step.