Introduction to critical appraisal!
Welcome to all students studying the MSc in Forensic Analysis. This blog post is part of your Research Methods and Practical Skills module led by Helen Green (USSKM3-30-M). USSKM3 = the module code; 30 = indicates this is a 30 credit module; M = indicates master’s level).
I’m Viv Rolfe and we have a number of 2 hour sessions together as follows:
Week 9 (20 September)
Lecture and discussion –> Notes here lecture-1-19sept2016_online
Task 1 – writing critically
Week 11 (4th October)
Week 15 (Independent learning)
Week 16 (8th November)
Week 17 (15th November)
Week 18 (22nd November)
My goal for you is of course to pass the January 2017 examination in which you will write a critical appraisal of a forensic science journal article. I also hope we have a constructive and fun time in these sessions and that you will also develop valuable skills in critical thinking and critical writing.
I think we take for granted our ability to read scientific articles, and write about them, but do we ever stop to question whether we are being really effective? How are your critical thinking skills? Do you sometimes think critically about the scientific world around you, or are you too rushed to stop and do so? Do you consider yourself a fair person, unbiased, in the way you think and communicate your ideas with others?
I hope these sessions help you grow as critical thinkers and writers. You might wish to watch this introductory video to critical thinking, which references the work of Richard Paul who was a leading proponent in this area.
How are these sessions structured?
You will need to bring a pen and paper to these sessions, as a big part of them will be you developing your thinking and writing skills. We shall be forming pairs and groups to discuss aspects of forensic science research and court case studies. Each week there will be a ‘task’ or homework which I very much hope you will all take part in; these will include the opportunity for you to complete small writing tasks for me to help you develop your talents!
Some preliminary reading for weeks 9 – 11
I am hoping you’ll find these sessions a bit of an ‘eye opener’ and we will be challenging some of the established doctrines that surround our research industry – from experimenting, interpretation, communication and publication. Here is a blog post that I wrote reflecting on the quality of medical research – or often, lack of it.
We are going to base some of this module work on the writing of Trish Greenhalgh. She has perfected the art of ‘trashing a paper’, and there are a number of articles that you can refer to, all freely available here. I’d focus on two at the start of this module:
Getting Your Bearings
Assessing the Methodological Quality
We’ll work toward understanding these papers toward the end:
Papers that Report Drugs Trials
Papers that Report Diagnostic Screening
Greenhalgh T (-) How to read a paper. The BMJ. Available: http://www.bmj.com/about-bmj/resources-readers/publications/how-read-paper
The do’s and don’ts of publishing.
American Association of Immunologists (2010). Dos and Don’ts
for Authors and Reviewers. Available: http://vivrolfe.com/research-methods/Assets/Scientific%20Publishing_Dos_Donts.pdf
These items may also be helpful:
A blog post on basic journal searching and some openly licensed (again free) learning resources coving all basic skills for students.
Let’s get started!
You have my UWE email address and contact details on the Blackboard Module Page, but you can also contact and chat with me via Twitter – in fact, I would love for you to share any interesting articles or videos relating to our studies.
How to think about assessment.
A starting point – or refresher – for anyone wishing to think about assessment and feedback is Phil Race’s textbook (available as a free eBook via our UWE library):
“The lecturer’s toolkit: A practical guide to assessment, learning and teaching”. (Latest UWE edition, 2015).
Chapter 2 gets us thinking about how to make assessments (and feedback) fair, valid, authentic and reliable. I think in recent years we have become confused regarding the purposes of formative and summative assessment. Our summative assessments come with the expectation of providing feedback, and programme assessment strategies tend to lose the scope for incremental development. Here are the four main purposes of assessment:
- Formative assessment – assessment ‘for’ learning uses assessment to raise student achievement.
- Summative assessment – assessment ‘of’ learning judges work against standards or criteria.
- Diagnostic – identification of needs/knowledge at the start of a session/course.
- Ipsative – self-assessment based on student previous levels of achievement.
Based on Phil’s toolkit, and some of my ideas, here are some important principles, with Phil’s highlighted in bold:
- Validity – how does the assessment link to the learning outcomes?(i.e. follow the principles of constructive alignment)?
- Reliability – how to ensure consistency between markers / quality control?
- Authenticity – is the assessment relevant to the discipline, or more broadly provide skills for employability?
- Fairness – do your students have a practice run or feedback on a draft piece of work? Are the assessment and marking criteria clear, and is the availability of support clear to learners?
- Engagement – is there evidence of the work in progress (e.g. blog or progress log, or submission of drafts to Turnitin)?
- Innovation – are you inviting originality from the students, or even better foster co-creation?
- Inclusivity – does your approach support all students on an equal basis, or are you testing their ability to take the assessment? Are you offering assessments in other formats? Is your overall curriculum design for assessments diverse to offer equity in opportunity overall?
When you are designing assessments – ideally as a team – you need to consider the individual student as well as a programmatic approach. I believe that if you get many of these elements right – engagement, innovation and inclusivity, then you will achieve the goal of ‘designing out’ academic offences in the holistic approach described by Macdonald and Carroll (2006).
Academic offences – and how we should be thinking about them.
The paper “Plagiarism detection and prevention” by Ursula McGowan (2005) always struck a chord with me. Ursula describes how the introduction of so-called ‘plagiarism detection software’ and academic offence policies have taken precedent over ensuring students have the right skills and understanding in the first place. She rightly argues we have put ‘the cart before the horse’, and a number of studies have verified how students new to university more often do not know what academic offences are, and international students may be particularly disadvantaged as they are not used to the regulations and cultural norms of our education system. Are we helping our students learn, or are we just trying to stop them cheat? Review the language in your module handbook? Are you policing students or building support and a culture of honesty and academic integrity?
What causes academic offences?
Academic offences can occur for many reasons. They may be inadvertent due to lack of student skill or understanding, they may of course be purposeful, or they quite often can be encouraged by poor assessment design. The main offences we are dealing with here include plagiarism, collusion and contract cheating (the purchasing of assignments from essay mill sites, or exchanges through social media or forums). In most of these sites where work can be purchased – the work is run through text-matching software and occasional words altered to break the text string. Therefore the work purchased won’t be detected by the digital tools often used in institutions (see sections below). BUT BUT BUT the well trained eye of a teacher or tutor will spot such work a mile off. And of course, the quickest and simplest thing to do is to copy and paste any suspected text into Google to see what comes up.
Are we really sure what these offences are?
What is plagiarism? “The action or practice of taking someone else’s work, idea, etc., and passing it off as one’s own” (Oxford English Dictionary 2007). This can be done in number of ways or combinations of these:
- Copying words verbatim without acknowledging the source
- Poor paraphrasing – a poor attempt at writing in student’s own words – without acknowledging the source
- Using other media without acknowledgement (e.g. photographs/diagrams – and of course the acquisition of additional permissions may also be required here)
It is always worth checking your own university definitions and regulations around academic offences as they may differ slightly. You can access the UWE student study skills guide on plagiarism here –> plagiarism.
For the correct acknowledgement we would look for a citation and reference in the correct format. Work can also be ‘self-plagiarised’ if previously submitted work is reused but not cited (i.e. acknowledging this is their own previous work). If a full assignment is resubmitted without citation, then this would be self-plagiarism alongside a fuller consideration of poor academic practice. Plagiarism is all about lack of acknowledgement of the ownership of the original work. Broader aspects of cheating could include altering or inventing data for example.
What is collusion? Another form of academic misconduct that can be common in group working where the assignment brief is not made clear to students. In this, students submit assignments that have been completed with other people.
Contract cheating? The purchasing of coursework online could not be simpler, either through essay mill websites, or through easy enough contact with other students. The only way of avoiding this is using the steps below to ensure the originality and individuality of work submitted.
How do we design out plagiarism?
Going back to our assessment principles, and in particular the ones that can help ‘design out’ academic offences:
I do believe there should be no such thing as an academic offence, or that it should very rarely occur. Students are often accused of collusion because for example the requirements and allowances of group working, say, for writing up laboratory practicals, is not clearly explained. (Is this a fair assessment?) We set them high stakes assignment (e.g. a dissertation write up) often with no evidence of work in progress. (Can we encourage engagement and therefore verify their work?) Research suggests students are more tempted to cheat when they are under stress – look at your programme assessment strategy. Is coursework bunched up at the end of term encouraging them to cut corners? Is the assessment blend diverse and flexible to accommodate learner abilities? (Inclusive?) Are students getting adequate formative feedback to develop their writing, and is this incremental helping them to develop across years of study? (Fairness)? Are you setting the same essay titles year on year? (Innovation)?
We must look at our individual practice and the assessments that we set, and we must also look at our overall programme curriculum design. Macdonald and Carol (2006) talk about a holistic approach from policy to curriculum design and student skills needs, and to balance “low-stakes formative assessment for learning and using high-stakes assessment sparingly to genuinely measure student learning“.
So ask yourselves some questions about your own assessments?
- Do you set the same essay title year on year?
- If students are retaking a module or the year, do you set the same assignment (tempting them to resubmit old work and self-plagiarise)?
- Do you provide an obvious title “describe the electrical conductance system of the heart” that can simply be Googled and easily ‘patchwork written’ or purchased?
- Is the student answering your question or establishing one for themselves? Or can you include a reflective question to ask “how did you approach this assignment”?(Innovation and originality?)
- Are you monitoring progress by introducing iterative steps? e.g if completing a 2000 word essay, why not include a search strategy and/or annotated bibliography that can be submitted separately for quick feedback (or peer-feedback)?
- Do you encourage independent critical thinking or do you slog through providing all the feedback yourself? Why not use Turnitin or similar for the submissions and self-review of drafts? (Evidence of engagement).
- Why not go the whole hog and establish the practice of students as ‘co-creators’ of their curricula and assessments? (Innovative and original; fairness; engagement).
‘Plagiarism detection software’ klaxon!
The article already mentions the use of digital tools for the detection of plagiarism. First of all this is a misnomer. Software such as Turnitin(R) or SafeAssign(R) do not detect plagiarism – you do! Or if the student is using services to check their work, then they are forming a judgement upon it. These software use text-matching algorithms to identify strings of words similar to other sources. They make the matches visible by often highlighting them in colour, and coming up with a percentage value of matched-text. Do not fall into the trap of using this % as a cut off for plagiarism or not, as I have seen in some university regulations. These commercially available plug-ins such as Turnitin can be used with virtual learning environments (e.g. commonly Moodle or Blackboard).
Turnitin as an example works by scanning submitted work against the company’s database of submitted work, articles and documents available from publishers, and internet pages. Not all publishers (especially some journals) have their text available for comparison, and the matches provided do not necessarily equate to the original source of the material. Whilst I do think they are useful tools, there is some work to be done to use them effectively.
What to think about before implementing the use of such software?
In some of my previous work, we were about to roll out the use of Turnitin for all student assignments in our university. One thing was clear, it was very important to proceed carefully with students and staff when doing this. Think about the message your are presenting when you make it compulsory for students Ito submit work to something that is essentially for the detection of cheating? Ask yourself also, are you also providing a robust programme of skills development for new students? Also, what impact is this extra assessment step going to have on staff marking time? In this paper I used Turnitin to provide instant formative feedback for students – but the process for getting there involved student and staff interviews. (Rolfe 2010).
Using Turnitin formatively = assessment for learning.
So in rolling out Turnitin, I worked with students and staff through interviews and questionnaires to come up with the best solution. The end result was brilliant – students helped us evolve a ‘self-service’ approach to enable them to check their own work. They could submit draft assignments to Turnitin and review their own reports. Any passages of their writing that was matched (highlighted in the software’s report) would have to be rewritten. Of course, references and text in quotations WOULD be highlighted, so it was also a way of crudely checking that references were correct!
The added bonus of this idea was that students would be regularly submitting DRAFTS of work in progress. This is an important step for verifying the ownership of work and engagement in the assessment process. For Turnitin and SafeAssign you can alter the submission and report settings to facilitate their use in a variety of ways, i.e. ensuring student draft work was not submitted to the database and would not therefore be a match for their final work.
I hope this post is useful and can help inject some creativity and fun back into assessments. As Macdonald and Carol (2006) conclude – we need to look at the causes and not just the symptoms. And that means also looking at our own teaching and assessment practice.
Presentation from 2009
Macdonald, R. and Carroll, J. (2006). Plagiarism – a complex issue requiring a holistic institutional approach. Assessment and Evaluation in Higher Education, 31 (2), 233-245.
McGowan, Ursula. (2005). Plagiarism detection and prevention: Are we putting the cart before the horse.” Proceedings of the HERDSA conference.
Rolfe V. 2008. Powerpoint guide for staff and students to understand Turnitin reports. (The software version will have changed, but the principles may be helpful to you). PPT slides –> Understanding Turnitin Reports
Rolfe V. 2011. Can Turnitin be used to provide instant formative feedback?. British Journal of Educational Technology, 42(4), pp.701-710. –> 10-108002602930500262536
Rolfe V. 2016. Powerpoint slides from a staff plagiarism workshop.
(Happy to run with your team!) PPT slides –> Plagiarism Workshop_Feb2016
- 29th March The HEA published a report on ‘fellowships and student engagement’, unpicking the impact on fellowships and teaching which is a nice read. However, a line is included that students are more able to contact and interact with staff “outside of formal class hours/work on activities other than coursework”. This casual comment worried me as it massages the expectation that staff are always available.
- 30th March Dave Cormier wrote about the resilience that students need to demonstrate to become successful learners. This seems a combination of emotional and academic resilience, and what we might thinking of in the UK more in terms of transitions? This reminds me of Helen Beetham’s work on digital wellbeing.
- 31st March Times Higher Education article suggesting stress is not always a bad thing. There is no mention of institutional responsibility or support.
- 2nd April Frances Bell discussed the idea of institutional fragility and links to some further excellent writing on resilience and well being.
- Article on White Fragility
Keep sharing folks.
RefWorks is a simple to use on-line application for organising references to articles that you may wish to use as part of a project or piece of research. It is very simple and intuitive to use, but often some of the difficult bits are exporting references or getting the information that you require. This is where Google Scholar can come in very handy, and the short video shows how I use Scholar and also another bibliographic database PubMed to get my ‘stuff’ into RefWorks.