August 12th and all ptarmigans and teaching teams run for cover

Today is August 12th and marks the start of the shooting season for Grouse, Ptarmigan and the Common Snipe.

It also, entirely uncoincidently, marks the release day of the 2015 National Student Survey results in the UK. With much discussion around the introduction of new metrics and outcome criteria for the proposed Teaching Excellence Framework (TEF), and with HEFCE planning a review of NSS questions in 2016/17 to possibly include student engagement, it is certainly worth taking a step back to think about the mathematics of it all.

Are metrics losing the plot?

Myself and more notable others have been concerned about the gamification of these metrics and the emphasis on strategies used to encourage students to participate. In my blog post last year “NSS is the name of the game” I looked at some of the satisfaction data and pondered on the overall usefulness (“add the shoe sizes of VC’s into league tables! Would be just as accurate“) and questioned the ethics of some of the approaches to gathering the data (“we were pressed by tutors to answer certain questions in a particular way“). I’ve heard myself someone comment to a student that if they don’t give a good NSS result, then employers won’t consider recruiting students from a particular university.

As I concluded in my blog last year after applying some statistical tests, is what we see a genuine annual increase in student claimed level of satisfaction or is it a result of a carefully honed precess for gathering data?

What about this year?

The 2015 NSS results were published by HEFCE this morning. There were no changes to the benchmarking this year, and only minor change was the cut off for inclusion of low datasets from 23 to 10 respondents.

The data above arranges English HEIs alphabetically. What we see is in 2015 (orange), the data very closely aligns with 2014 (turquoise). In 2014 the benchmarking was altered and brakes put onto the system, and this was the first year where no significant increase in overall satisfaction across the English university sector (ANCOVA tests previously reported).

By looking at Q22 in terms of annual means and standard deviations (below), a yearly increase in overall satisfaction is apparent across the 127 English HEIs. What is interesting is the reduction in variation across the institutions, and one has to question whether the NSS is becoming less discerning?

NSS Q22 Mean — FIGURE 2: Q22 Mean NSS results 2005-2015. CC BY Viv Rolfe

Results by mission group?

It is interesting to split the analysis by mission group, separating out the 19 Russell Group and 18 University Alliance institutions from the others. By extrapolating the data, in 2021 there will be a big party as universities outstrip the performance of the Russell Group should the survey remain unchanged. But that is unlikely.

NSS Mission Group — FIGURE 3: Trended Data for Q22 By Mission Group. CC BY Viv Rolfe

Commentary

Hello – David here. You may remember me from such blogs as “Followers of the Apocalypse” where I write primarily about UK HE policy making. When Viv showed me this data set I was fascinated to see trends in the NSS, and immediately started to think about implications for the proposed “Teaching Excellence Framework” (TEF).

Figure 2, above, shows the variability in NSS scores between institutions decreasing with each NSS iteration. There could be a number of drivers for this, I would suggest that it perhaps shows institutions getting better at running the process, and getting the message out to students that a good institutional NSS score is good for the perceived value of their degree from that institution. Manifest nonsense, obviously – but if metrics are good for one thing it would be for developing faith-based belief systems!

When the NSS was originally developed the scores were primarily used at a course (or at worst, subject area) level. This allowed prospective students to compare the attitudes of students doing a similar course at different institutions. Anyone that works in a HEI will tell you that variability between departments and subject areas is huge, and indeed most of the pain experienced by academic staff on the “glorious twelfth” will be concerning this intra-institutional variation.

Johnson Minor’s TEF would (we are led to believe) be at an institutional level, and Osborne announced that it would affect an institutional ability to increase student fees to match inflation (as if inflation was some kind of an optional extra rather than reflecting the reality of rising costs). If NSS results at an institutional level are included in the proposed “basket” of metrics within the TEF, the decreasing inter-institutional variance shown in Figure 2 implies that this will have the effect of making it harder to discriminate between institutions.

Of course, this may be what BIS want (so all institutions can increase fees with the figleaf of independent oversight justifying it – see also OFFA!), but in this case it seems like a very expensive way to pretend that you are not making HE more expensive for the taxpayer. But I suppose BIS are used to spending lots of money to do that kind of thing. Such is politics.

So that’s why I think this analysis is important.

[declaration of interest: I received 1 pint of beer for writing the above]

Thank you David for your wisdom and insight. The discipline variation and impact on teaching teams also concerns me as often we are held to account for much of what goes on behind the scenes of successful teaching (timetables, IT, efficiency of academic administration systems) which is not reflected in the survey.

It just remains to say, and I know all the readers are dying to know, that a ptarmigan is a slightly plump bird with a beautiful plumage. I hope that like most teaching teams today, it manages to dodge any bullets and experiences nothing but a mild ruffling of feathers.

Rock Ptarmigan — By Jan Frode Haugseth (Own work) [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0) or GFDL (http://www.gnu.org/copyleft/fdl.html)], via Wikimedia Commons

Process:

Download summary data from HEFCE.
Data for English HEIs cleaned – aligning university names with 2014 recorded names (e.g. The University of Bath was University of Bath in 2014). Data was then sorted.
Registered data was used – that is the data represented the institution where the student was registered (as opposed to Taught – where sutdents do majoriy of year 1 study).
Data is all full time and part time students.
In 2010 the data benchmarking changed and was adjusted for ethnicity. Interetingly the data is not adjusted for socio-economic background (http://www.bristol.ac.uk/academic-quality/ug/nss/research.html).

Other articles that week:

http://www.theguardian.com/higher-education-network/2015/aug/13/the-national-student-survey-should-be-abolished-before-it-does-any-more-harm?CMP=share_btn_tw

Chris Hanretty (2015). When communicating uncertainty goes wrong. Available: https://medium.com/@chrishanretty/when-communicating-uncertainty-goes-wrong-cdc5b7ae226b

Keith Burnett (2015). Available: https://www.timeshighereducation.co.uk/blog/want-raise-quality-teaching-begin-academic-freedom

June 7, 2015June 7, 2015

The pic-a-mix of open education

“Fale – Barcellona – 194” by Fabio Alessandro Locati CC BY-SA 3.0 via Wikimedia Commons – http://commons.wikimedia.org/wiki/File:Fale_-_Barcellona_-_194.jpg

Week 1 thoughts:
An awesome and informative week learning about open pedagogy with details of the course right here:
http://blogs.ubc.ca/teachwordpress/week-1-discussion/

The webinar by Amanda Coolidge and Tracy Kelly was super and made me think about open pedagogy really for the first time really. I’ve been working on open educational resource (OER) projects for around ten years with rather a ‘bull in a china shop’ approach, and this week is really making me think through more of a structure that would be useful.

How do we define open?

The webinar started off assisting us in thinking about defining open pedagogy. I really have struggled recently with how on earth the open movement can move forward unless we start defining the ethical boundaries in which we operate? Open ultimately is a free-for-all on the web and we are all familiar with the more negative aspects of this. We need ethical common ground for sure. Hopefully his community will help define it. (Here is a pitch for funding that was unsuccessful but shares my thoughts…).

David Wiley’s definition of ‘open’ and the 5 Rs of openness

The 5 Rs of openness provide a useful framework for us to think of in terms of sharing learning materials openly – reuse, revise, remix, redistribute and retain (the control of the content produced). I would add a 6 category of course – to participate in the spirit of openness in an ethically appropriate manner.

(Just thinking the 6th R might be – responsibly or responsibility).

Back to the show – many of the definitions of open do focus on resources and as the webinar presenters highlighted there is a whole field of open practice, behaviour and activity that also apply. I know in my own institutions there are discussions about open data, open science and open research. We all increasingly operate a ‘pick-a-mix’ approach to education, dipping in and out of being open, although many learners and teachers find it more easy to just adopt a philosophical stance toward openly working, and apply it to all they do.

So how do I define open pedagogy?

Going back to some of the clearest thinking about education and what it is, Richard Peters’ describes the ‘matter’ and ‘manner’ of education, and discussions about open pedagogy for me cannot isolate the content from the learner-teacher relationship, and the manner in which we all engage on the web. His book “Ethics and Education” is a must-read. (R. S. Peters: Ethics and education. 5th edn, George Allen & Unwin Ltd, London 1968).

Also, going back to the definitions of pedagogy (and andragogy) that are the science and practice of teaching, we absolutely do not want to lose sight of the need to evaluate what we do either to produce well informed approaches. I personally think we have all become a little lousy at that.

How can we reflect on our approaches now?

I really liked this next bit of the webinar. Amanda and Tracy have constructed a matrix to help us position our practice. I might challenge and say there could be a new row on co-creation. The matrix I think identifies a learning journey that we must encourage students along. Obviously, students new to university or anyone making the transition back to education are vulnerable and need support, so may start off in box 4. As they become accustomed to open licensing, use of technology, working openly on the web – as they may not have ever done before – we can gently nurture them toward box 1. It would be good if the journey to 2 and 1 were a quick one as we don’t want to dwell to long on dinosaur methods of education.

Amanda Coolridge and Tracy Kelly. Open Pedagogy Matrix. CC BY. (BCU week 1 webinar screen grab).

Unfortunately we may be based in institutions that are non-open, and have no policy or inclination to work openly. We may therefore be firmly rooted in 4 and 3, using dinosaur methods, locking learning behind VLEs, and assessing student knowledge regurgitation in examinations, obsessing with providing them feedback, all of which does nothing to develop individuals.

Summary of week 1

So I have formed a definition of open pedagogy in my mind, embracing the pic-a-mix of open practices and approaches we all use these days.
I have reflected on my own practice and that we need to encourage students (and staff) through the matrix up to box 1 and perhaps beyond as co-producers.
I believe in the 5Rs of open plus a 6th category of embracing the spirit of openness in an ethical manner.
My question that I hope to explore through this course is – how can I work openly as an educator on the web – because I chose to do so outside of work hours – but operate ethically in terms of copyright (OK, using what is mine), in terms of privacy (OK, using the privacy terms that come with WP blogging) but also ethically in terms of the manner in which we conduct ourselves?

June 1, 2015

Initial thoughts and why am I doing this?

Open Pedagogy with UBC.

Week 1

I’m very much looking forward to this course by the University of British Columbia (UBC). It is strange that anything with pedagogy in the title always strikes the fear of god in me. I remember my first lecture role at Nottingham University where I kept hearing this strange word and wondering what the hell it was. Most professions are strange and you drift into roles with no formal training. I’ve drifted from science to management to teaching and now vaguely drift between all three. There was never any formal training about pedagogy at any point.

So for my own benefit, “pedagogy is the discipline that deals with the theory and practice of education” (thank you Wikipedia).

But aren’t we lucky today with open courses such as this. We have such amazing knowledge and learning at our finger tips which I guess is the whole point of open pedagogy – to effortlessly share your materials and learning with others to benefit the sum of the whole. And this is underpinned by the most recent theories, thinking and ideas. How cool is that.

The course structures some topics around pedagogy, practice and design, again all separate but obviously inter-related. I’m planning an open course of my own later in the year so all of this will be invaluable.

I’m certainly looking forward not just to the design aspects but learning about ethics for working out in the open. This is an area in need of attention and I look forward to the conversations that will take place. Our technology has out-paced our pedagogy which has certainly out-paced our ethical thinking, and I suppose tech always challenges us in these ways. I’ve talked and written about this a little, and also the bredth of the ethical challenges are wide ranging. My point now is well, what are we going to do about it?

Anyway I need to stop the open rambling and get open learning with UBC. By the way, open rambling is not a ‘thing’ yet. Just remember you heard it hear first.