Paul Kleiman

November

November Landscape

Only a lone crow

breaks the chill silence

sensing perhaps

as autumn fades

that winter’s wait is over

its time has come.

Beyond Teaching Excellence

(On not filling and measuring the pail, but lighting the fire)

[This is the transcript of a ‘provocation’ I presented at the joint conference of the three subject associations for dance (DanceHE), drama (SCUDD) and music (NAMHE) as part of a debate about the Teaching Excellence Framework (TEF), University of Huddersfield, 7th April 2017]

So…..
We didn’t vote for it. It’s far more complicated than those who envisaged it thought it might be. There are serious doubts about whether it’s going to achieve what it set out to do. But it’s here, it’s happening, and we’re told we’ve got to make the best of it.

And that’s just the TEF.

So what are the “known knowns”, which in some cases turn out to be “unknowns” anyway.

Take ‘excellence’ for example. In the original government Green Paper, there’s a very helpful section headed:

What do we mean by excellence?

(At this point you have to imagine a scene from Yes Minister, in which Sir Humphrey, in his inimitable way, explains ‘excellence’ to a rather bewildered Jim Hacker)

Humphrey: Well, Minister…..

“There is no one broadly accepted definition of “teaching excellence”. In practice it has many interpretations and there are likely to be different ways of measuring it. The Government does not intend to stifle innovation in the sector or restrict institutions’ freedom to choose what is in the best interests of their students. But we do think there is a need to provide greater clarity about what we are looking for and how we intend to measure it in relation to the TEF. Our thinking has been informed by the following principles:

excellence must incorporate and reflect the diversity of the sector, disciplines and missions – not all students will achieve their best within the same model of teaching;
excellence is the sum of many factors – focussing on metrics gives an overview, but not the whole picture;
perceptions of excellence vary between students, institutions and employers;
excellence is not something achieved easily or without focus, time, challenge and change.”

Hacker: Really? Is that it?
Humphrey: Yes, Minister.

So that’s much clearer isn’t it? The Government clearly believes that excellent teaching can occur in many different forms, in a wide variety of institutions, and it is not the intention of the TEF to constrain or prescribe the form that excellence must take. What we should expect though, is that excellent teaching, whatever its form, delivers excellent outcomes.

Well, for a start, the TEF has criteria, and metrics, so the notion and form of excellence in regard to the TEF is already proscribed or constrained. In fact, what will – and no doubt is happening – is that institutions are aligning their priorities precisely to those criteria – which, as almost everyone admits – are in any case based on the proxy ‘outcomes’ of NSS scores, retention and most importantly graduate salaries which are high enough to pay back all the money the government has lost in its ill-advised restructuring of HE finance.

What about some “known unknowns”?

What we do know is that the TEF puts the so-called ‘elite’ institutions, whose excellence is hitherto self-evident and uncontested, under some pressure. It’s been noticeable that while there has been a great deal of loud and insistent criticism of the NSS metrics from that quarter, there has been far less directed at the stats on employment and earnings. Surely that has nothing to do with the fact that, in regard to employment and earnings, Russell Group graduates do rather well…..but the evidence shows that has more to do with the social, educational and cultural capital of Russell Group students than the quality of the teaching

So, while we know that it’s hard to fall once you’ve been at the top for a long time, the “unknown” is how the politics of this will play out in practice when our much vaunted and excellent ‘elite’ institutions don’t appear at the top of whatever league table appears for Teaching Excellence. Paul Blackmore, at Kings, argues that long-held prestige – primarily based on research outcomes – will probably still trump the TEF outcomes in the short-term.

Some years ago, way before TEF even glimmered on the horizon, I was asked to contribute to a special ‘On Excellence’ issue of the HEA’s ‘Exchange’ magazine. I was writing at a time when I’d just been at a large academic conference where there had been a major debate about excellence in higher education.

Even back then there was a clear majority who felt that the term has lost credibility and value. When all institutions are either ‘excellent’ or, at the very least, ‘striving for excellence’ then we are witnessing a lot of sound (but hopefully not fury) signifying nothing. Excellence, in the memorable term coined by Bill Readings’ has become ‘de-referentialised’.

Readings’ extraordinarily prescient book – The University in Ruins – appeared 21 years ago. He described the university, forced to abandon its historical raison d’etre as a bastion of knowledge and culture, now enmeshed in corporatist, consumerist ideology, and obsessed by the ubiquitous, but empty, quest for excellence. Everywhere one looks, one sees mission statements and vision statements aspiring to excellence. There are no modifiers to the word, and thus the excellence can seem empty. It is unspecific. Excellence in teaching, excellence in research, and not forgetting excellence in parking (I kid you not). Everybody’s striving for excellence, because who wants to strive for just being ‘pretty good’.

The “excellence in parking” demonstrate the vacuity of the term. It was actually awarded to a university’s Parking Services for their success in restricting vehicle access to the university, and significantly reducing the number of parking spaces. Excellence could just as well have meant making people’s lives easier by increasing the number of parking spaces available. The issue here is not the merits of either option but the fact that excellence can function equally well as an evaluative criterion on either side of the issue of what constitutes “excellence in parking”, because excellence has no content to call its own.

So beyond the dodgy but unavoidable metrics, and the slipperiness of the notion of excellence, how do we demonstrate our ‘excellence’?

Well, there is of course the Provider Submission to supply a richer, deeper more meaningful narrative. Well having seen a few, there’s a clear M.O. (as the police say at a crime scene):

Use the word excellence or excellent a lot, particular if you can quote a reputable source e.g. QAA or External Examiner or provide some evidence where you can.
Write things like….
“We recognise and reward excellent teaching.”
“Team teaching and team meetings all support teaching excellence”
“Our eternal examiner comments that students’ standards of analysis are at times excellent”
“Our accrediting body commented on our ‘Excellent interaction with industry bodies and exemplary experiential learning practices’”

So we do what we’ve always done which is to play the game, and play it as well, or as excellently, as we can.

But…thinking back to that article on excellence I wrote for the HEA: my starting point for the article was the thought that we tend to avoid the ‘E’ word when it comes to talking about our art and performance practices. I know it’s a cliché, but when we greet the performers after a great show, we rarely, if ever, say “That was excellent”.

We say “That was wonderful” (the ‘darling’ is an optional extra), and I do think ‘wonderful’ is truly a much better word than ‘excellent’. Rather than ‘excellent’s’ rather hard-edged, triumphalist implication of being better than others, ‘wonderful’, i.e. full of wonder, has a sense of the remarkable, the extraordinary, the inspirational, the truly successful that is the mark of the highest quality work.

There is that famous quote ‘education is not the filling of a pail, but the lighting of a fire‘ (which is often misattributed to W.B.Yeats). That ‘lighting of the fire’ is what I know we truly strive for, and that we need to look beyond our obsession with trying to define, achieve, assess and reward excellence. What the TEF forces us to do is to focus on the filling of the pail, and then tick off when it is full, when we should be focusing on the fire.

If you enjoyed this, you might also enjoy The Emperor’s Folderol or Tales of the TEF: https://stumblingwithconfidence.wordpress.com/2015/11/12/the-emperors-folderol-or-tales-of-the-tef/

Remembering Bill Mitchell

Tributes have been pouring in for Bill Mitchell the director, designer, theatre-maker and inspirational founder of the unique landscape theatre company Wildworks, who passed away on the 14th April 2017 at the age of 65. Many people in the U.K. and around the world know of Bill through his extraordinary work with Kneehigh and then, particularly, Wildworks, and there is a wonderful obituary by Lyn Gardner in The Guardian.

I first encountered Bill when we were both studying theatre design at Wimbledon School of Art in the early 1970s. Although I was a year ahead of him, students often worked together across the years, and I was fortunate to work with Bill. His creativity and imagination, as well as his great sense of fun and play, were much in evidence even then. A few years later, in 1979, I joined Bill and his partner Sue Hill in the TIE/Community theatre company Key Perspectives, based in Peterborough. (One the founder members of the company – who had left by the time I joined – was Colin Hicks, who now sits on the Wildworks board).

Key Perspectives was operating in what was TIE’s heyday, and we were just one of many such companies operating around the UK, supported by local authorities and the Arts Council. We were committed to creating and producing high quality theatre and drama-based educational programmes in schools and commmunities, working closely with teachers and students and with the communities in and around Peterborough in which they lived. Bill’s personal and creative engagement with people, place and community that became such a distinctive characteristic of his later work, can be traced back to that early work.

One my clearest and fondest memories of Bill was when we were working on a children’s Christmas show called ‘Oddbod’, which we were creating for the main house at the Key Theatre. Peterborough was then an ancient small city attached to a fast growing new town, and the infant and primary schools, particularly in and around the newly built estates, were full of children who had recently arrived from other towns and cities. Though we worked collectively, Oddbod was very much driven by Bill’s directorial and creative vision and his passion to reflect, truthfully and imaginatively, the experiences of those very young people.

We visited a number of schools, and we listened to and collected the stories that the children told and painted about ‘Oddbod’, about being a ‘stranger in a strange land’, about displacement and arrival, about feeling alone, about making friends, etc. Under Bill’s directorial and visual eye we took all the drawings and pinned them around the walls of our rehearsal space, which we filled with as many pieces of costume, materials and objects that we could find. We then started to use the drawings as starting points with which to create characters and improvise situations.

I remember that Sue Hill was attracted to a particular ‘Oddbod’ painting which consisted of a large oval black blob with a head-like smaller blob attached to it. Above it hovered another large and rather ominous looking black blob. Wrapping herself in a large black blanket, with a large black hat, Sue created a brilliant, funny character that was terrified of everything and anything, whose only utterance was “Any minute! Ooh, it’s goin’ to happen! Any minute!”. Sue reduced Bill and the rest of us to helpless hysteria and, needless to say, Any Minute became one of the ‘stars of the show’.

My particular memory of Bill was when, with our designer hats on, he and I went to buy some large fishing weights which we were going to use to assist in lowering a huge painted backcloth for the final scene. Fishing was ‘big’ in Peterborough and in the surrounding Fens, and Woolworths, in the town centre, had a large fishing section, but the particular half-pound conical weights we needed weren’t on display. So we approached one of the sales staff and asked if they still had any in stock. He went off to the storeroom and then came back saying “Yes, we have some, but we can’t sell them to you because it’s out of season”.

Bill: “But we don’t want them for fishing, we want them for a children’s Christmas show!”.

Salesperson: “You’ll have to speak to the Manager. I’ll get him.”

The manager arrived and Bill explained what we wanted, and the manager repeated that while they had them in stock they couldn’t sell them to us because it was out of season and that it was Woolworth’s policy.

At which point Bill, who was over 6ft and who, with his gold earring and gold tooth, could appear quite fearsome, went into full-on, John Cleese, dead parrot mode:

Bill: “This is a shop, yes? A shop that sells things to customers, yes? I am a customer, yes? I wish to purchase something with this [waves money in manager’ face] that you normally sell and which you currently have stored in a box in your storeroom. I understand that it is not the fishing season, but I don’t want them for bloody fishing, I want them for a Christmas show for the children of this city. Now are you going to allow me to purchase them or not?”

Manager: “How many do you need?”

* * * * * * * * * * *

Wanderer, your footsteps are the road,
and nothing more;
wanderer, there is no road,
the road is made by walking.
By walking one makes the road,
and upon glancing behind
one sees the path that never will be trod again.
Wanderer, there is no road –
only wakes upon the sea.

Antonio Machado

Assessment at the Edge 2: Clash of the Paradigms

(or Rumble in the Epistemological Jungle)

clashoftheparadigms

We operate, on the whole, within education systems that are based on a traditional linear, positivist, computational paradigm that has been the dominant scientific paradigm since Newton et al back in the 17th century. It is a paradigm in which education is perceived as a form of industrial or mechanised process.

It is, essentially, a closed system, which is the sum of its parts (learners, teachers, curriculum, content, delivery, technology, etc.). By controlling these parts, we can regulate the performance of the whole system. Educational systems design is the process of regulating these closed systems. It is a system in which human behaviour and performance are assumed to be predictable within known circumstances, and in which knowledge is assumed to be an external, quantifiable object that can be transmitted to and acquired by learners. This enables patterns of behaviour to be analysed and used to make judgements about how learners are thinking or what they have learned.

It is a system in which a ‘line of determination’ is assumed between cause and effect: for example – teaching predictably causes learning. These assumptions over-simplify the world and tend to reduce human learning, performance and achievement to a repertoire of manipulable behaviours. But learning is far more complex and much less certain than these assumptions infer.

In one corner we have the dominant Positivist or Quantitative Paradigm which is based on the epistemological belief that all true knowledge is ‘scientific’ knowledge. In this paradigm there is a single objective reality ‘out there’ that is orderly, predictable, and can be studied, captured and understood by amassing data and triangulating it (I shall return to the triangle).

The overarching aim is to achieve explanation and control, which is possible because knowledge is objective, measurable, value-free and a quantifiable object that is transmitted by the ‘teacher as expert’ to, and acquired by, learners. Rigour is achieved via the ‘holy trinity’ of validity, reliability and generalisability.

In the other corner we have the Interpretive or Qualitative Paradigm in which subjectivity is inherent and should be acknowledged because complete or pure objectivity is impossible and should never be claimed. For those in this corner ‘truth’ is a matter of consensus amongst informed and sophisticated constructors, not of correspondence with an objective reality. Furthermore, because all measurement is fallible, there is great emphasis on multiple measures and observations in order to able to claim authenticity, and for that authenticity to be recognised.

Those who operate within this paradigm understand that there are multiple realities and that knowledge is subjective, contextualised and value-dependent. They aim for understanding in order to enhance learning, they are openly self-questioning and self-critical, and they welcome scrutiny and debate. Importantly, they view students as co-constructors of their learning, and perceive themselves to be partners and participants in learning as well as guides and mentors. (That position, by the way, does not prevent them from also being experts!).

In order to find a way to deal with all of this epistemological complexity in relation to how we approach assessment, I’m suggesting that one way – and of course there are and will be others – is to approach assessment as a form a qualitative research instead of a quasi-scientific investigation. If we choose to follow the interpretive paradigm in relation to assessment then we need adjust our thinking and our language. Essentially we need to do a form of ‘Find and Replace’.

We need to replace :

Validity with Credibility, Coherence, Consistency, Trustworthiness, Authenticity
Certainty with Relativity
Generalised Explanation with Local Understanding
Source Data with Empirical Materials
“Is it true?” with “Does it work?”
Single Point Perspective with Multiple Perspectives
the Triangle with the Crystal

To be continued……

Next instalment coming soon: Assessment at the Edge 3: Triangles and Crystals

Back to Assessment at the Edge 1

Assessment at the Edge 1: Faultlines

I know I’m not alone in feeling – increasingly as the years roll by – that all too often the way we assess is at odds with the way our students (and we ourselves) actually learn and experience learning. While I and everyone else round the assessment board table is doing their very best to be professional, to ensure that procedures and regulations are followed, and taking great care to ensure that students are treated fairly and reliably….a bit of my brain is suffering from a form of cognitive dissonance and saying ‘This is nuts!’

There used to be a one of those car stickers that went something like ‘Do Not Adjust Your Mind…There Is A Fault With Reality’. And that’s how it feels. There seems to be a serious disjunction or faultline between what appears on the hundreds of assessment print outs – actual or virtual – and the actual day-to-day experience of learning and teaching, of creating work, of pursuing ideas, of encouraging and enabling students to really stretch themselves, to try out new things, to fail gloriously, to boldly go.

As teachers we need to – and are required to – ascertain, with as much validity, reliability and fairness as possible, what our students know and understand. For most of us, learning, teaching and assessment is a form of journey along the highways and byways of a particular subject. We, the guided and the guides, explore the landscape of the discipline. Our role as guides, more often than not, is to enable those we guide to understand the meaning and significance of what is seen, what is heard, what is felt, what is experienced.

Occasionally, because as guides we take our work seriously, and there are matters of accountability and responsibility that need to be attended to, we stop and check to see how much those who have entrusted their education to us know and understand, and what they can do.

In order to assess our students we stop acting as guides and essentially become researchers or purposeful explorers. We set out to discover what they know and understand, and what skills they possess. We ask them, demand of them, to demonstrate their knowledge, understanding and skills. We assess them, evaluate them, judge them, measure them against a set of standards or criteria.

If it’s a relatively simple matter of fact or basic competence then it is relatively straightforward to test it. The student either knows who, or what, or when or how…or they don’t. But the landscapes we explore in education are highly complex, intricate, shifting, multi-layered, multi-faceted. Simple straightforward answers and simple straightforward questions are hard to come by. The terrain does not reveal itself easily. Nor should it. In such a landscape meaningful assessment is also highly complex, intricate, shifting, multi-layered and multi-faceted.

If we consider the types of assessment that dot the landscape, we can see a veritable bio-diversity of assessment. But this diversity is also a challenge, and it is worth noting just how many of these types of assessment result in assessment ‘data’ that is qualitative rather than quantative in nature.

But there is may be a problem with this: the more assessment involves qualitative information, the more subjectivity is involved. Now this would be mitigated and we would have improved reliability if we had strict or stricter assessment criteria and also more structured and proscribed content. But, and this is a big ‘but’, if we had those it would obliterate the essence of qualitative assessment in terms of flexibility, personal orientation and authenticity. Which brings us, eventually, to the question of assessment paradigms and to the Clash of the Paradigms.

Next instalment: Assessment at the Edge 2: Clash of the Paradigms