Want quality in HE – more open education then please!

Assessing quality in Higher Education inquiry – publications

Well along with the 79 more eloquent responses on the 30th October 2015 HE quality and teaching parliamentary inquiry was little old me. It was my first time and probably not very good, but hopefully some of the sentiments about the importance of open education will influence someone somewhere.

Go to –> List of responses

It was good to see the Association of National Teaching Fellows, Terry McAndrew and other excellent responses on the list.

For what it is worth, here is what I said.

Open Sorry We're Closed

Alan Levine Flickr https://www.flickr.com/photos/cogdog/ CC BY 2.0

I am an academic in the UK and have worked strategically to lead open education projects for 10 years. The Global Open Education movement is now 15 years well established and is probably one of the most significant education initiatives of our time in terms of enhancing educational opportunities for all learners, offering a high-quality and flexible means of gaining training or embarking on one’s educational journey.

The use of Creative Commons license [1] has been the ‘key to the door’ for a range of activities – providing an academic ecosystem of open educational resources (OERs), providing free or low cost openly licensed textbooks for students, enabling open access to journal articles, and has led to more progressive thinking about the sharing of research data, and building open science and innovation communities that extend beyond the walls of institutions.

I wish to comment on points 3-5. Open education is paramount to each of these points as engaging with open education (having the confidence to share your stuff) is one of the most powerful professional development tools that an academic member of staff or teacher can engage in.

The intention of the Government proposal as outlined in the chair’s comments “to strengthen the UK’s world-leading university brand” can only be delivered through policy that enables universities to compete through global open education enterprise.

England as a nation is sadly lacking behind others in the UK, and needs to join the initiatives and level of national interest in open education as seen in Scotland [2, 3] and Wales [4, 5]. Both nations have made declarations of intent, and the UK as a whole now needs to embed ‘open’ within education policy.

3) What should be the objectives of a Teaching Excellence Framework (‘TEF’)?

a) How should a TEF benefit students? Academics? Universities?

The TEF has to lever open education practice. Students today need to develop global perspectives and graduate as citizens with strong global consciousness. Open education – for example, engaging with openly licensed learning materials from Africa, through participating in massive online open courses from India, through the UK driving the use of open textbooks to lower costs of study in universities, can provide our graduate body with global knowledge, connections and cultural understanding and tolerance.

For academics and universities, the TEF needs to lever open practice through embedding openness in UK Professional Skills Framework criterion (UKPSF as run by the HEA and SEDA), and National Teaching Fellowship schemes. These criteria are some 10 years out of date and there is no reflection of the digital ability and social media literacy required by staff (and students) to communicate and operate in the world today. There is already an established teaching excellence award and the TEF can reflect upon numbers of fellows and case studies from the National Teaching Fellow scheme (NTFS), and those who achieve the highest levels of professional teaching standards of the UKPSF scheme.

Universities will strengthen their world-class brand by the TEF providing a window to these already outstanding individuals. However we need Government policy in order for the UK to engage in global open education enterprise.

b) What are the institutional behaviours a TEF should drive? How can a system be designed to avoid unintended consequences?

The TEF has to drive this. All publically-funded educational teaching and research materials should be freely available through adopting open licenses. On the 29th October 2015 the US Department of Education announced a new regulation in this fashion [6]. The US Government are committed to opening up education opportunities to all learners, and open licenses reduce the costs of text books, provide pools of high-quality learning materials for schools, colleges and universities to share, and stimulate innovation and creativity across communities of practice [7]. President Obama recognises education as “a cornerstone for progress” and a number of new open government initiatives in 2014/2015. We cannot be blind to the inequalities within our UK education system, and openness can drive behavioural changes and wholesale institutional change as it has done across the education system in the US.

Unintended consequences? The TEF has to support outstanding and deeply committed educators and teachers in our university system. It must not provide another layer of burocracy for teaching staff which will detract more of their time away from the classroom. Universities need to decide whether they wish to deliver world class education or world class administration.

c) How should the effectiveness of the TEF be judged?

The TEF will be deemed successful to staff and institutions short-term if it provides reward and recognition for those outstanding teachers, if it drives equality and investment in teaching and pedagogic research in universities, and if it does not provide another burdensome set of metrics that distract from educating young people. I see this as more important as the long-term and absolute goals of fostering competition and enhancing student choice.

4) How should the proposed TEF and new quality assurance regime fit together? 

The two are distinct. The present QA scheme is vital to ensure that all publically and privately funded institutions deliver a baseline standard of education, and the process must continue to offer developmental opportunities to institutions and to ensure parity across the sector. The QA focus on ‘enhancement’ has never come to light, and the TEF scheme is a vehicle for this. Both work together to delivery ‘quality and excellence’. Again, the regimes need to drive genuine change and not become another set of burdensome data-gathering schemes.

5) What do you think will be the main challenges in implementing a TEF? 

That Universities will focus on the process and infrastructure to deliver the TEF and not teaching at all. A ‘light touch’ process, driving excellence through open education, but a process that is politically important, is what is needed. Excellence can be linked to increases in fees, but there should be matched-funding to ensure education does not become exclusive. It needs to encourage all levels of wealth, gender, ethnic diversity and offer flexible study options for part-time and mature students. Open education is one proven solution to this in the US, opening up different educational business models.

 

[1] Creative Commons http://creativecommons.org

[2] Open Scotland http://openscot.net/

[3] Scottish Open Education Declaration http://declaration.openscot.net/

[4] Open Wales Cymru http://www.oerwales.ac.uk

[5] Open Wales Cymru Declaration http://www.oerwales.ac.uk/?page_id=4

[6] USA Department of Education – Office of Educational Technology

http://tech.ed.gov/open-education/

[7] The White House https://www.whitehouse.gov/blog/2014/09/26/promoting-open-education-help-teachers-and-students-around-world

 

 

 

 

Yours faithfully

 

August 12th and all ptarmigans and teaching teams run for cover

Today is August 12th and marks the start of the shooting season for Grouse, Ptarmigan and the Common Snipe.

It also, entirely uncoincidently, marks the release day of the 2015 National Student Survey results in the UK. With much discussion around the introduction of new metrics and outcome criteria for the proposed Teaching Excellence Framework (TEF), and with HEFCE planning a review of NSS questions in 2016/17 to possibly include student engagement, it is certainly worth taking a step back to think about the mathematics of it all.

Are metrics losing the plot?

Myself and more notable others have been concerned about the gamification of these metrics and the emphasis on strategies used to encourage students to participate. In my blog post last year “NSS is the name of the game” I looked at some of the satisfaction data and pondered on the overall usefulness (“add the shoe sizes of VC’s into league tables! Would be just as accurate“) and questioned the ethics of some of the approaches to gathering the data (“we were pressed by tutors to answer certain questions in a particular way“).  I’ve heard myself someone comment to a student that if they don’t give a good NSS result, then employers won’t consider recruiting students from a particular university.

As I concluded in my blog last year after applying some statistical tests, is what we see a genuine annual increase in student claimed level of satisfaction or is it a result of a carefully honed precess for gathering data?

What about this year?

The 2015 NSS results were published by HEFCE this morning. There were no changes to the benchmarking this year, and only minor change was the cut off for inclusion of low datasets from 23 to 10 respondents.

NSS Q22 2015

FIGURE 1: Q22 NSS results 2010-2015. CC BY Viv Rolfe

 

The data above arranges English HEIs alphabetically. What we see is in 2015 (orange), the data very closely aligns with 2014 (turquoise). In 2014 the benchmarking was altered and brakes put onto the system, and this was the first year where no significant increase in overall satisfaction across the English university sector (ANCOVA tests previously reported).

By looking at Q22 in terms of annual means and standard deviations (below), a yearly increase in overall satisfaction is apparent across the 127 English HEIs. What is interesting is the reduction in variation across the institutions, and one has to question whether the NSS is becoming less discerning?

NSS Q22 Mean

FIGURE 2: Q22 Mean NSS results 2005-2015. CC BY Viv Rolfe

 

Results by mission group?

It is interesting to split the analysis by mission group, separating out the 19 Russell Group and 18 University Alliance institutions from the others. By extrapolating the data, in 2021 there will be a big party as universities outstrip the performance of the Russell Group should the survey remain unchanged. But that is unlikely.

NSS Mission Group

FIGURE 3: Trended Data for Q22 By Mission Group. CC BY Viv Rolfe

 

Commentary

Hello – David here. You may remember me from such blogs as “Followers of the Apocalypse” where I write primarily about UK HE policy making. When Viv showed me this data set I was fascinated to see trends in the NSS, and immediately started to think about implications for the proposed “Teaching Excellence Framework” (TEF). 

Figure 2, above, shows the variability in NSS scores between institutions decreasing with each NSS iteration. There could be a number of drivers for this, I would suggest that it perhaps shows institutions getting better at running the process, and getting the message out to students that a good institutional NSS score is good for the perceived value of their degree from that institution. Manifest nonsense, obviously – but if metrics are good for one thing it would be for developing faith-based belief systems!

When the NSS was originally developed the scores were primarily used at a course (or at worst, subject area) level. This allowed prospective students to compare the attitudes of students doing a similar course at different institutions. Anyone that works in a HEI will tell you that variability between departments and subject areas is huge, and indeed most of the pain experienced by academic staff on the “glorious twelfth” will be concerning this intra-institutional variation.

Johnson Minor’s TEF would (we are led to believe) be at an institutional level, and Osborne announced that it would affect an institutional ability to increase student fees to match inflation (as if inflation was some kind of an optional extra rather than reflecting the reality of rising costs). If NSS results at an institutional level are included in the proposed “basket” of metrics within the TEF, the decreasing inter-institutional variance shown in Figure 2 implies that this will have the effect of making it harder to discriminate between institutions.

Of course, this may be what BIS want (so all institutions can increase fees with the figleaf of independent oversight justifying it – see also OFFA!), but in this case it seems like a very expensive way to pretend that you are not making HE more expensive for the taxpayer. But I suppose BIS are used to spending lots of money to do that kind of thing. Such is politics.

So that’s why I think this analysis is important.

[declaration of interest: I received 1 pint of beer for writing the above]

 

Thank you David for your wisdom and insight. The discipline variation and impact on teaching teams also concerns me as often we are  held to account for much of what goes on behind the scenes of successful teaching (timetables, IT, efficiency of academic administration systems) which is not reflected in the survey.

It just remains to say, and I know all the readers are dying to know, that a ptarmigan is a slightly plump bird with a beautiful plumage. I hope that like most teaching teams today, it manages to dodge any bullets and experiences nothing but a mild ruffling of feathers.

Rock Ptarmigan

By Jan Frode Haugseth (Own work) [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0) or GFDL (http://www.gnu.org/copyleft/fdl.html)], via Wikimedia Commons

 

Process:

  • Download summary data from HEFCE.
  • Data for English HEIs cleaned – aligning university names with 2014 recorded names (e.g. The University of Bath was University of Bath in 2014). Data was then sorted.
  • Registered data was used – that is the data represented the institution where the student was registered (as opposed to Taught – where sutdents do majoriy of year 1 study).
  • Data is all full time and part time students.
  • In 2010 the data benchmarking changed and was adjusted for ethnicity. Interetingly the data is not adjusted for socio-economic background (http://www.bristol.ac.uk/academic-quality/ug/nss/research.html).

Other articles that week:

http://www.theguardian.com/higher-education-network/2015/aug/13/the-national-student-survey-should-be-abolished-before-it-does-any-more-harm?CMP=share_btn_tw

Chris Hanretty (2015). When communicating uncertainty goes wrong. Available: https://medium.com/@chrishanretty/when-communicating-uncertainty-goes-wrong-cdc5b7ae226b

Keith Burnett (2015). Available: https://www.timeshighereducation.co.uk/blog/want-raise-quality-teaching-begin-academic-freedom