Assessment at the Edge 2: Clash of the Paradigms

(or Rumble in the Epistemological Jungle)

clashoftheparadigms

We operate, on the whole, within education systems that are based on a traditional linear, positivist, computational paradigm that has been the dominant scientific paradigm since Newton et al back in the 17th century. It is a paradigm in which education is perceived as a form of industrial or mechanised process.

It is, essentially, a closed system, which is the sum of its parts (learners, teachers, curriculum, content, delivery, technology, etc.). By controlling these parts, we can regulate the performance of the whole system. Educational systems design is the process of regulating these closed systems. It is a system in which human behaviour and performance are assumed to be predictable within known circumstances, and in which knowledge is assumed to be an external, quantifiable object that can be transmitted to and acquired by learners. This enables patterns of behaviour to be analysed and used to make judgements about how learners are thinking or what they have learned.

It is a system in which a ‘line of determination’ is assumed between cause and effect: for example – teaching predictably causes learning. These assumptions over-simplify the world and tend to reduce human learning, performance and achievement to a repertoire of manipulable behaviours. But learning is far more complex and much less certain than these assumptions infer.

In one corner we have the dominant Positivist or Quantitative Paradigm which is based on the epistemological belief that all true knowledge is ‘scientific’ knowledge. In this paradigm there is a single objective reality ‘out there’ that is orderly, predictable, and can be studied, captured and understood by amassing data and triangulating it (I shall return to the triangle).

The overarching aim is to achieve explanation and control, which is possible because knowledge is objective, measurable, value-free and a quantifiable object that is transmitted by the ‘teacher as expert’ to, and acquired by, learners. Rigour is achieved via the ‘holy trinity’ of validity, reliability and generalisability.

In the other corner we have the Interpretive or Qualitative Paradigm in which subjectivity is inherent and should be acknowledged because complete or pure objectivity is impossible and should never be claimed. For those in this corner ‘truth’ is a matter of consensus amongst informed and sophisticated constructors, not of correspondence with an objective reality. Furthermore, because all measurement is fallible, there is great emphasis on multiple measures and observations in order to able to claim authenticity, and for that authenticity to be recognised.

Those who operate within this paradigm understand that there are multiple realities and that knowledge is subjective, contextualised and value-dependent. They aim for understanding in order to enhance learning, they are openly self-questioning and self-critical, and they welcome scrutiny and debate. Importantly, they view students as co-constructors of their learning, and perceive themselves to be partners and participants in learning as well as guides and mentors. (That position, by the way, does not prevent them from also being experts!).

In order to find a way to deal with all of this epistemological complexity in relation to how we approach assessment, I’m suggesting that one way – and of course there are and will be others – is to approach assessment as a form a qualitative research instead of a quasi-scientific investigation. If we choose to follow the interpretive paradigm in relation to assessment then we need adjust our thinking and our language. Essentially we need to do a form of ‘Find and Replace’.

We need to replace :

Validity with Credibility, Coherence, Consistency, Trustworthiness, Authenticity
Certainty with Relativity
Generalised Explanation with Local Understanding
Source Data with Empirical Materials
“Is it true?” with “Does it work?”
Single Point Perspective with Multiple Perspectives
the Triangle with the Crystal

To be continued……

Next instalment coming soon: Assessment at the Edge 3: Triangles and Crystals

Back to Assessment at the Edge 1

Assessment at the Edge 1: Faultlines

I know I’m not alone in feeling – increasingly as the years roll by – that all too often the way we assess is at odds with the way our students (and we ourselves) actually learn and experience learning. While I and everyone else round the assessment board table is doing their very best to be professional, to ensure that procedures and regulations are followed, and taking great care to ensure that students are treated fairly and reliably….a bit of my brain is suffering from a form of cognitive dissonance and saying ‘This is nuts!’

There used to be a one of those car stickers that went something like ‘Do Not Adjust Your Mind…There Is A Fault With Reality’. And that’s how it feels. There seems to be a serious disjunction or faultline between what appears on the hundreds of assessment print outs – actual or virtual – and the actual day-to-day experience of learning and teaching, of creating work, of pursuing ideas, of encouraging and enabling students to really stretch themselves, to try out new things, to fail gloriously, to boldly go.

As teachers we need to – and are required to – ascertain, with as much validity, reliability and fairness as possible, what our students know and understand. For most of us, learning, teaching and assessment is a form of journey along the highways and byways of a particular subject. We, the guided and the guides, explore the landscape of the discipline. Our role as guides, more often than not, is to enable those we guide to understand the meaning and significance of what is seen, what is heard, what is felt, what is experienced.

Occasionally, because as guides we take our work seriously, and there are matters of accountability and responsibility that need to be attended to, we stop and check to see how much those who have entrusted their education to us know and understand, and what they can do.

In order to assess our students we stop acting as guides and essentially become researchers or purposeful explorers. We set out to discover what they know and understand, and what skills they possess. We ask them, demand of them, to demonstrate their knowledge, understanding and skills. We assess them, evaluate them, judge them, measure them against a set of standards or criteria.

If it’s a relatively simple matter of fact or basic competence then it is relatively straightforward to test it. The student either knows who, or what, or when or how…or they don’t. But the landscapes we explore in education are highly complex, intricate, shifting, multi-layered, multi-faceted. Simple straightforward answers and simple straightforward questions are hard to come by. The terrain does not reveal itself easily. Nor should it. In such a landscape meaningful assessment is also highly complex, intricate, shifting, multi-layered and multi-faceted.

If we consider the types of assessment that dot the landscape, we can see a veritable bio-diversity of assessment. But this diversity is also a challenge, and it is worth noting just how many of these types of assessment result in assessment ‘data’ that is qualitative rather than quantative in nature.

But there is may be a problem with this: the more assessment involves qualitative information, the more subjectivity is involved. Now this would be mitigated and we would have improved reliability if we had strict or stricter assessment criteria and also more structured and proscribed content. But, and this is a big ‘but’, if we had those it would obliterate the essence of qualitative assessment in terms of flexibility, personal orientation and authenticity. Which brings us, eventually, to the question of assessment paradigms and to the Clash of the Paradigms.

Next instalment: Assessment at the Edge 2: Clash of the Paradigms

‘See me. Feel me. Touch me.’ (Pt. 1)

Object lessons and reflections on the HEA Arts & Humanities conference 2016

Brighton-HEA3

Early March. Brighton is an alluring place, despite the chill in the air. The sun is shining, the sea is blue, the promenade and beach lie temptingly just across the road from the conference venue, and the esoteric shops, cafés and bars of The Lanes are just a couple of minute’s walk away. So it was a testament to the commitment of the participants and the quality of the many and varied sessions on offer that so many were able to resist the temptation to ‘skip school’.

While, in some sessions and in Jonathan Worth’s fascinating keynote on the second day, there was an inevitable and valuable focus on the digital and the virtual, the most powerful message – for me – was the extraordinary pedagogic power of the physical, tangible object. From Kirsten Hardie’s opening keynote with accompanying green plastic teapot, pineapple ice bucket and toilet brush, to the Lego sessions of Contemplative Pedagogies, by way of Simon Heath’s wonderful drawings (see image below) that captured the essences of the whole event, it was the object that held centre stage. And there were plenty more sessions that focused on making and doing as a pedagogic activity, not just a practical or physical one.

Photo left: Hannah Cobb @ArchaeoCobb

I have written elsewhere (‘On history and all that’ ) on the power of objects to engage the imagination, to generate stories and lines of enquiry, to provoke philosophical, political, ethical debates, and to provide learning experiences that really ‘stick’. I still recall clearly the ‘History of Decoration’ seminars from my art student days when ‘Simi’ (Ms. Simeon the lecturer) would enliven her lectures on, say, Ancient Egypt, by taking a vase or piece of jewellery or some other artefact out of the cardboard box she always brought. She would casually hand the object to someone to examine and then pass around the room with the words ‘Do try to be careful, dear, that’s three and half thousand years’ old’. This would be repeated every session, whether the topic was Ancient Rome (jewellery), Medieval Europe (a crucifix) or Tudor England (a lace ruff). I only realised what we had been passing round when I heard that, on her death , Simi’s large collection of “just something to look at while I’m talking” had been bequeathed to and enthusiastically accepted by the V&A museum.

What also became clear during the conference, is that ‘object lessons’ are not just the preserve of the creative arts community. Every discipline clearly has its associated artefacts which can be used not only to enhance the teaching of an ‘academic’ subject, but to act as foci for the characteristics and qualities of the sort of learning that Kirsten Hardie talked about: learning that engages, amazes, provokes, exhilarates, takes risk, liberates.

One of the things I remember from those, now distant, art history sessions is something I frequently refer to in my work on curriculum design and assessment. In one her first seminars, Simi passed round an Ancient Greek vase that was covered head to foot in decoration. The reason, she said, for filling every possible square inch was ‘horor vacui’ – fear of open space – because it was through open space that the ‘Evil Eye’ enters the world. That might well be one of the reasons (though I would avoid mentioning the ‘Evil Eye’ or the Devil in module specifications and handbooks) why we insist on filling our curricula with content: ‘Idle hands make the devil’s workshop’ and all that. But we also know that deep learning, creativity and innovation require time and space to incubate and develop.

Objects, importantly, enable us to slow down time: to observe, to really look, to touch, to feel, to explore. Simon Piasecki, at the conference, talked about how he gets his performance students to slow right down and focus on the minutiae of what they are doing, and the artist Marina Abramovich – one of whose concerns is the fact that we don’t stop to really look any more – has a number of exercises she uses with those who come to view her work to achieve the same slowing down. When I worked at the Liverpool Institute of Performing Arts (LIPA), one of the first year ‘options’ that I established – open to any student – was a traditional life-drawing class. All the students that participated in that quiet, contemplative two hours on a Wednesday evening, amidst an extraordinarily hectic timetable (‘horor vacui’!), reported that they understood that it wasn’t about being able to draw. It was about having the time and space to slow down and really observe not only the ‘object’ (usually another student) but also themselves….and to ‘take a line for a walk’ in Paul Klee’s famous phrase.

Ken Robinson, in his now famous TEDTalk on creativity and education, jokes about academics generally seeing their bodies as a form of transportation to get them to meetings. He, among others, stresses the importance of mind and body, the intellectual and the emotional, the psychological and the physiological. What came through so strongly at the HEA Arts and Humanities conference was that objects – in all their glorious variety – and our close interactions with them, provide a means to engage powerfully in deep, meaningful learning experiences. Objects both inhabit space and create space. We just need the space, the time and, impotently, the confidence to engage in our own object lessons.

Brighton- HEA2

Photos by Paul Kleiman unless otherwise stated

Conference Twitter hashtag: #HEAArts16

On beauty and elegance in education

In his book ‘Fearful Symmetry: the search for beauty in modern physics’, Anthony Zee describes how Einstein displayed a supreme disinterest in any proposed formula or solution, no matter how accurate it might be, that he considered ugly.

“As soon as an equation seemed to him to be ugly, he really rather lost interest in it and could not understand why somebody else was willing to spend much time on it. He was quite convinced that beauty was a guiding principle in the search for important results in theoretical physics.”

Today (13 Aug 2014), amongst the usual ugly headlines of death, destruction and disease, Maryam Mirzakhani is being celebrated as the first woman to have been awarded the prestigious Fields Medal – the Nobel Prize for mathematicians. Her work – as described by those who have some grasp of her achievements – has a “breathtaking scope, is technically superb and boldly ambitious”. She herself describes mathematics as full of “beauty and elegance”.

Now, I’d hate to think that beauty and elegance is the sole preserve of mathematicians dealing, like Mirzakhani, in esoteric fields such as complex geodesics, transcendental objects, and differential geometry. I’d argue that we all need at least a bit of beauty and elegance in our lives and work, and we certainly can see people striving for it (though many just don’t care) in many areas: whether it’s the presentation of food, the design of buildings and spaces, the arrangement of an exhibition, the movement of a dancer across the stage, the order and rhythm of words on a page.

So, why not strive for some beauty and elegance in education and in the curricula and learning experiences we design for our students? Yes, it’s often messy and a bit (or very) chaotic. But just as the mathematics of chaos have a certain underpinning beauty and elegance, we – as ‘architects of education’ – should strive to construct and compose learning and teaching experiences that flow and connect in ways that have a certain beauty and elegance about them. It’s not easy, but surely worth the effort.

Forget Excellence…we need wonder!

Paul Kleiman

(First published in the Higher Education Academy’s EXCHANGE MAGAZINE, Issue 7, 2008)

Excellence! Everyone is writing, talking, researching, obsessing about it. But what is it?

Some years ago PALATINE, the Subject Centre for Dance, Drama and Music, undertook an enquiry into the use of the full range of marks in assessing the performing arts in higher education. As well as provoking the centre’s biggest and most heated electronic postbag, a number of respondents described the distinct discomfort they experienced when considering the assessment of work at the very top of the range. One memorably wrote: “I feel the increasingly heavy pull of gravity on my pen as I get to 75%.”

The response supported research that found that the extremities of the percentage scale are perceived as insecure territory for the assessors of qualitative subject matter. There is a strong sense, in the arts and humanities, that nothing can be that good or, for that matter, that bad, and the research revealed that most marking in the arts and humanities ranged between c. 35% to 75% which, in the eccentric and esoteric honours grading system we use in the UK, still manages to cover everything from a Fail to a First!

Undoubtedly one of the assessment challenges we have set for ourselves in performing arts disciplines is requiring students to demonstrate achievement in a wide range of practical, scholarly and creative modes. High achievement in one is rarely sustained across the breadth of an assessment régime in our disciplines, and we have to work to ensure that ‘excellent’ achievement is reflected in the aggregated marks at module and degree level. This is a pedagogic challenge which is not shared by other, more traditional arts and humanities subjects.

So what does excellence mean in this context?

Going by the result of the debate on excellence at an academic conference, there is a clear majority who feel that the term has lost credibility and value. When all institutions are either ‘excellent’ or, at the very least, ‘striving for excellence’ then we are witnessing a lot of sound (but hopefully not fury) signifying nothing. Excellence has become ‘de-referentialised’. Turning to the dictionary provides little assistance. In the Concise Oxford ‘to excel’ is defined as to surpass or to be pre-eminent (i.e. to be better than the majority), whereas ‘excellence’ is defined as ‘very good’.

Whatever meaning ‘excellence’ once had has become lost in a blizzard of hyperbole. The fate of excellence follows in the tradition of other terms such as ‘community’ and, more recently, ‘creativity’, whose meaning has become devalued and decontextualised through over- and inappropriate use.

In the arts, the term excellence is rarely if ever used as a descriptor except – and this may be relevant in considering educational achievement – in relation to the application or demonstration of skill or craft. Academics, students and arts practitioners tend to avoid the ‘E’ word. The theatrical cliché has never been: “You were excellent, darling”. ‘Wonderful’ is truly a much better word than ‘excellent’ to describe high artistic achievement. Rather than excellent’s rather hard-edged, triumphalist implication of being better than others, ‘wonderful’, i.e. full of wonder, has a sense of the remarkable, the extraordinary, the truly successful that is the mark of the highest quality work.

Excellence, it must be said, is much favoured by arts politicians and bureaucrats who use it both aspirationally and as a justification for funding. Excellence attracts rewards and prizes. But the use of the term has more to do with product branding (as it has in higher education) than with a real concern with the subtle complexities of quality and value.

Here is a typical example: one of our leading UK arts funding bodies, in its mission statement, states: “We believe the arts to be the foundation of a confident and cultured society. They challenge and inspire us. They bring beauty, excitement and happiness into our lives. They help us to express our identity as individuals, as communities and as a nation”.

Wonderful! But then they go and ruin it by reverting to corporate-speak and saying they are going to “serve the people … by fostering arts of excellence through funding, development, research and advocacy”. An external examiner I knew, having seen what was – by general consensus – a remarkable, successful, extraordinary, inspiring … yes, wonderful piece of final-year practical work was dismayed to find that the two internal examiners, who also thought the work was remarkable, successful etc had agreed a mark of 75%. He asked them to start at 100%, and argue persuasively for marks to be deducted. With the assessment criteria in their hands they struggled to get below 95%. To describe that work as merely ‘excellent’ would have been insufficient. It was beyond excellence.

That is perhaps what we should be striving for and, in doing so, we need to look beyond our obsession with trying to define, achieve, assess and reward excellence.

As the old saying goes: education is, or ought to be, a wonderful thing.