The power of reflective writing. And parma violets.

I feel twitchy because I haven’t written anything (sensible) in a long time. Work has been so stressful I don’t think I’ve had the brain capacity. Suddenly now this year my other work – open education – my outside of office hours work – the work I get no time or recognition for in my institution – is bursting with activity. I was bored on Twitter just now, and as always the wonderful Sally was poised with a challenge: So what shall I write about?

I don’t know yet but I’ll let you know by the end of this blog post. It might be:

  • Parma violets are disgusting but I’ve eaten the whole packet anyway.
  • I’ve bought daffodils and Haribo for my team meeting tomorrow.
  • I’ve seven invited talks and conference presentations between now and the end of June!
  • We buy our first house and move. Soon!
  • Yoiks.

Actually I know what I’m going to write about and it is going to be ugly. Because blogging is a great way of getting things off yer chest and solving problems with a wider community. The trouble is I’ve national awards for my work – I’m a National Teaching Fellow and a Principal Fellow of the Higher Education Academy. That is fine, but as part of that work I have to mentor others, promote the scheme. Don’t get me wrong, I’m a big fan of the Higher Education Academy and wish them well in their transition. But I sadly think learning and teaching in institutions is moving away from that community of teachers, the learning and teaching culture that is the heart of your university, to being top-down strategically decided. I see this everywhere and I’m not picking on any university in particular. So I’m struggling to attend workshops to mentor folk through their applications, to sit on reviewer panels, to read applications, because if I’m totally honest, I’m thinking it is an absolute waste of time (apart from getting to hang out with fantastic people).

The work of the wonderful Annette Cashmore and colleagues from their 2013 report always comes to mind: Rebalancing promotion in the HE sector: is teaching excellence being rewarded? They talk of a two tier system. Career cul-de-sacs. Others have referred to career suicide. As commented at the time – the students are at the heart of the system. For sure! But we must “value and properly reward those who teach them”. In a 2016 communication – you can see how grumpy I was getting then – I reflected upon the fact that as a sector there is still a fair bit of inertia in recognising great teaching. As stated on the poster below, Jo Johnson in all of the papers said that teaching should not be the poor cousin of research. Unfortunately the teaching excellence framework (#TEF) will be about as useful as a chocolate teapot, and I suspect a fair amount of fabrication and spin has already gone into TEF submissions. Care for amazing National Teaching Fellows and care for fantastic research and evaluation to improve learning and teaching practice will have not.

But surely we recognise excellence?

 I presented this poster last year because I’d been declined from a job. An essential criterion was ‘postgraduate teaching qualification’ and based upon the HEA, HEFCE and HESA definitions, verified by our lovely HEA chums, I did indeed have some of them. I was rejected from the selection process. In the end it wasn’t because I din’t have ‘postgraduate teaching qualifications’ but ‘I didn’t have the right ones’. OK I’m from Romford, I can take shit on the chin.

Poster_VRolfe_TeachingExcellence

What can we learn from elsewhere?

A really interesting conversation on the Association of Learning Technology (ALT) mailing list echoes some of these points. Our wonderful community of learning technologists in the UK I do feel is on a similar trajectory. In the US and Canada, techs and library staff are the power houses of university and college innovation. In the UK they are sequestered into dark basement rooms and never see the light of day. (I love them all). Then, considering about 90% of learning /teaching and assessment is digital these days, we need to listen to ALT conversations and the likes of the awesome Tony Bates who talked about the digital in this entirely free and openly licensed textbook: A MUST READ FOR ANY PGERTHE STUDENT: That is free to read. And openly licensed. To share. Teaching in a digital age. The email thread (of about 20+ responses) asked about the use of evidence-informed decision making. I think the same applies for learning and teaching these days:

  • Evidence is slow – the research model does not work
  • A focus on particular technologies / assessment and feedback strategies (not the learning)
  • Long lists of journals presenting positive impacts of TEL (and less so of pedagogy) on learning (which is perhaps part of the problem – publication bias)
  • I’m pretty sure James Clay won’t mind me quoting (cos I can’t get onto his blog article) “when an academic asks “for the evidence to show technology can make a difference” the problem is not lack of evidence, but one of resistance to change, culture, rhetoric and motivation”. Sure. I also think academia has expanded like layers of filo pastry in Mary Berry’s oven. Do we think about students, ourselves, our league tables, metrics…WHAAAT? (Come on, I joined this gaff because I just wanted to teach).

There was then on the ALT list a general melching of agreement that in terms of learning and teaching versus technology (still separated at birth – why) implementation, implementation of processes and technology, implementation of new assessment policy happens from the top-down all the time, but is rarely backed up by evidence or research. And certainly never published. As for being in a university with a community of librarians – technologists – and passionate teachers, this is long gone. How bloody sad. I’m trying to think of a metaphor here but failing. What is tragic is that this system – the omnipotent – is failing some students every day.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

So welcome to the end of the blog post.

Where next? Well let’s try and influence our friends in influential positions. Wait! That is many National Teaching Fellows and Principal Fellows! You are heads of learning and teaching, pro-vice chancellors.

 

So let’s hopefully have some honest discussion about what we are doing. And for god sake. Parma violets are bloody disgusting.

 

 

Want quality in HE – more open education then please!

Assessing quality in Higher Education inquiry – publications

Well along with the 79 more eloquent responses on the 30th October 2015 HE quality and teaching parliamentary inquiry was little old me. It was my first time and probably not very good, but hopefully some of the sentiments about the importance of open education will influence someone somewhere.

Go to –> List of responses

It was good to see the Association of National Teaching Fellows, Terry McAndrew and other excellent responses on the list.

For what it is worth, here is what I said.

Open Sorry We're Closed

Alan Levine Flickr https://www.flickr.com/photos/cogdog/ CC BY 2.0

I am an academic in the UK and have worked strategically to lead open education projects for 10 years. The Global Open Education movement is now 15 years well established and is probably one of the most significant education initiatives of our time in terms of enhancing educational opportunities for all learners, offering a high-quality and flexible means of gaining training or embarking on one’s educational journey.

The use of Creative Commons license [1] has been the ‘key to the door’ for a range of activities – providing an academic ecosystem of open educational resources (OERs), providing free or low cost openly licensed textbooks for students, enabling open access to journal articles, and has led to more progressive thinking about the sharing of research data, and building open science and innovation communities that extend beyond the walls of institutions.

I wish to comment on points 3-5. Open education is paramount to each of these points as engaging with open education (having the confidence to share your stuff) is one of the most powerful professional development tools that an academic member of staff or teacher can engage in.

The intention of the Government proposal as outlined in the chair’s comments “to strengthen the UK’s world-leading university brand” can only be delivered through policy that enables universities to compete through global open education enterprise.

England as a nation is sadly lacking behind others in the UK, and needs to join the initiatives and level of national interest in open education as seen in Scotland [2, 3] and Wales [4, 5]. Both nations have made declarations of intent, and the UK as a whole now needs to embed ‘open’ within education policy.

3) What should be the objectives of a Teaching Excellence Framework (‘TEF’)?

a) How should a TEF benefit students? Academics? Universities?

The TEF has to lever open education practice. Students today need to develop global perspectives and graduate as citizens with strong global consciousness. Open education – for example, engaging with openly licensed learning materials from Africa, through participating in massive online open courses from India, through the UK driving the use of open textbooks to lower costs of study in universities, can provide our graduate body with global knowledge, connections and cultural understanding and tolerance.

For academics and universities, the TEF needs to lever open practice through embedding openness in UK Professional Skills Framework criterion (UKPSF as run by the HEA and SEDA), and National Teaching Fellowship schemes. These criteria are some 10 years out of date and there is no reflection of the digital ability and social media literacy required by staff (and students) to communicate and operate in the world today. There is already an established teaching excellence award and the TEF can reflect upon numbers of fellows and case studies from the National Teaching Fellow scheme (NTFS), and those who achieve the highest levels of professional teaching standards of the UKPSF scheme.

Universities will strengthen their world-class brand by the TEF providing a window to these already outstanding individuals. However we need Government policy in order for the UK to engage in global open education enterprise.

b) What are the institutional behaviours a TEF should drive? How can a system be designed to avoid unintended consequences?

The TEF has to drive this. All publically-funded educational teaching and research materials should be freely available through adopting open licenses. On the 29th October 2015 the US Department of Education announced a new regulation in this fashion [6]. The US Government are committed to opening up education opportunities to all learners, and open licenses reduce the costs of text books, provide pools of high-quality learning materials for schools, colleges and universities to share, and stimulate innovation and creativity across communities of practice [7]. President Obama recognises education as “a cornerstone for progress” and a number of new open government initiatives in 2014/2015. We cannot be blind to the inequalities within our UK education system, and openness can drive behavioural changes and wholesale institutional change as it has done across the education system in the US.

Unintended consequences? The TEF has to support outstanding and deeply committed educators and teachers in our university system. It must not provide another layer of burocracy for teaching staff which will detract more of their time away from the classroom. Universities need to decide whether they wish to deliver world class education or world class administration.

c) How should the effectiveness of the TEF be judged?

The TEF will be deemed successful to staff and institutions short-term if it provides reward and recognition for those outstanding teachers, if it drives equality and investment in teaching and pedagogic research in universities, and if it does not provide another burdensome set of metrics that distract from educating young people. I see this as more important as the long-term and absolute goals of fostering competition and enhancing student choice.

4) How should the proposed TEF and new quality assurance regime fit together? 

The two are distinct. The present QA scheme is vital to ensure that all publically and privately funded institutions deliver a baseline standard of education, and the process must continue to offer developmental opportunities to institutions and to ensure parity across the sector. The QA focus on ‘enhancement’ has never come to light, and the TEF scheme is a vehicle for this. Both work together to delivery ‘quality and excellence’. Again, the regimes need to drive genuine change and not become another set of burdensome data-gathering schemes.

5) What do you think will be the main challenges in implementing a TEF? 

That Universities will focus on the process and infrastructure to deliver the TEF and not teaching at all. A ‘light touch’ process, driving excellence through open education, but a process that is politically important, is what is needed. Excellence can be linked to increases in fees, but there should be matched-funding to ensure education does not become exclusive. It needs to encourage all levels of wealth, gender, ethnic diversity and offer flexible study options for part-time and mature students. Open education is one proven solution to this in the US, opening up different educational business models.

 

[1] Creative Commons http://creativecommons.org

[2] Open Scotland http://openscot.net/

[3] Scottish Open Education Declaration http://declaration.openscot.net/

[4] Open Wales Cymru http://www.oerwales.ac.uk

[5] Open Wales Cymru Declaration http://www.oerwales.ac.uk/?page_id=4

[6] USA Department of Education – Office of Educational Technology

http://tech.ed.gov/open-education/

[7] The White House https://www.whitehouse.gov/blog/2014/09/26/promoting-open-education-help-teachers-and-students-around-world

 

 

 

 

Yours faithfully

 

August 12th and all ptarmigans and teaching teams run for cover

Today is August 12th and marks the start of the shooting season for Grouse, Ptarmigan and the Common Snipe.

It also, entirely uncoincidently, marks the release day of the 2015 National Student Survey results in the UK. With much discussion around the introduction of new metrics and outcome criteria for the proposed Teaching Excellence Framework (TEF), and with HEFCE planning a review of NSS questions in 2016/17 to possibly include student engagement, it is certainly worth taking a step back to think about the mathematics of it all.

Are metrics losing the plot?

Myself and more notable others have been concerned about the gamification of these metrics and the emphasis on strategies used to encourage students to participate. In my blog post last year “NSS is the name of the game” I looked at some of the satisfaction data and pondered on the overall usefulness (“add the shoe sizes of VC’s into league tables! Would be just as accurate“) and questioned the ethics of some of the approaches to gathering the data (“we were pressed by tutors to answer certain questions in a particular way“).  I’ve heard myself someone comment to a student that if they don’t give a good NSS result, then employers won’t consider recruiting students from a particular university.

As I concluded in my blog last year after applying some statistical tests, is what we see a genuine annual increase in student claimed level of satisfaction or is it a result of a carefully honed precess for gathering data?

What about this year?

The 2015 NSS results were published by HEFCE this morning. There were no changes to the benchmarking this year, and only minor change was the cut off for inclusion of low datasets from 23 to 10 respondents.

NSS Q22 2015

FIGURE 1: Q22 NSS results 2010-2015. CC BY Viv Rolfe

 

The data above arranges English HEIs alphabetically. What we see is in 2015 (orange), the data very closely aligns with 2014 (turquoise). In 2014 the benchmarking was altered and brakes put onto the system, and this was the first year where no significant increase in overall satisfaction across the English university sector (ANCOVA tests previously reported).

By looking at Q22 in terms of annual means and standard deviations (below), a yearly increase in overall satisfaction is apparent across the 127 English HEIs. What is interesting is the reduction in variation across the institutions, and one has to question whether the NSS is becoming less discerning?

NSS Q22 Mean

FIGURE 2: Q22 Mean NSS results 2005-2015. CC BY Viv Rolfe

 

Results by mission group?

It is interesting to split the analysis by mission group, separating out the 19 Russell Group and 18 University Alliance institutions from the others. By extrapolating the data, in 2021 there will be a big party as universities outstrip the performance of the Russell Group should the survey remain unchanged. But that is unlikely.

NSS Mission Group

FIGURE 3: Trended Data for Q22 By Mission Group. CC BY Viv Rolfe

 

Commentary

Hello – David here. You may remember me from such blogs as “Followers of the Apocalypse” where I write primarily about UK HE policy making. When Viv showed me this data set I was fascinated to see trends in the NSS, and immediately started to think about implications for the proposed “Teaching Excellence Framework” (TEF). 

Figure 2, above, shows the variability in NSS scores between institutions decreasing with each NSS iteration. There could be a number of drivers for this, I would suggest that it perhaps shows institutions getting better at running the process, and getting the message out to students that a good institutional NSS score is good for the perceived value of their degree from that institution. Manifest nonsense, obviously – but if metrics are good for one thing it would be for developing faith-based belief systems!

When the NSS was originally developed the scores were primarily used at a course (or at worst, subject area) level. This allowed prospective students to compare the attitudes of students doing a similar course at different institutions. Anyone that works in a HEI will tell you that variability between departments and subject areas is huge, and indeed most of the pain experienced by academic staff on the “glorious twelfth” will be concerning this intra-institutional variation.

Johnson Minor’s TEF would (we are led to believe) be at an institutional level, and Osborne announced that it would affect an institutional ability to increase student fees to match inflation (as if inflation was some kind of an optional extra rather than reflecting the reality of rising costs). If NSS results at an institutional level are included in the proposed “basket” of metrics within the TEF, the decreasing inter-institutional variance shown in Figure 2 implies that this will have the effect of making it harder to discriminate between institutions.

Of course, this may be what BIS want (so all institutions can increase fees with the figleaf of independent oversight justifying it – see also OFFA!), but in this case it seems like a very expensive way to pretend that you are not making HE more expensive for the taxpayer. But I suppose BIS are used to spending lots of money to do that kind of thing. Such is politics.

So that’s why I think this analysis is important.

[declaration of interest: I received 1 pint of beer for writing the above]

 

Thank you David for your wisdom and insight. The discipline variation and impact on teaching teams also concerns me as often we are  held to account for much of what goes on behind the scenes of successful teaching (timetables, IT, efficiency of academic administration systems) which is not reflected in the survey.

It just remains to say, and I know all the readers are dying to know, that a ptarmigan is a slightly plump bird with a beautiful plumage. I hope that like most teaching teams today, it manages to dodge any bullets and experiences nothing but a mild ruffling of feathers.

Rock Ptarmigan

By Jan Frode Haugseth (Own work) [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0) or GFDL (http://www.gnu.org/copyleft/fdl.html)], via Wikimedia Commons

 

Process:

  • Download summary data from HEFCE.
  • Data for English HEIs cleaned – aligning university names with 2014 recorded names (e.g. The University of Bath was University of Bath in 2014). Data was then sorted.
  • Registered data was used – that is the data represented the institution where the student was registered (as opposed to Taught – where sutdents do majoriy of year 1 study).
  • Data is all full time and part time students.
  • In 2010 the data benchmarking changed and was adjusted for ethnicity. Interetingly the data is not adjusted for socio-economic background (http://www.bristol.ac.uk/academic-quality/ug/nss/research.html).

Other articles that week:

http://www.theguardian.com/higher-education-network/2015/aug/13/the-national-student-survey-should-be-abolished-before-it-does-any-more-harm?CMP=share_btn_tw

Chris Hanretty (2015). When communicating uncertainty goes wrong. Available: https://medium.com/@chrishanretty/when-communicating-uncertainty-goes-wrong-cdc5b7ae226b

Keith Burnett (2015). Available: https://www.timeshighereducation.co.uk/blog/want-raise-quality-teaching-begin-academic-freedom