Assessment at the Edge 2: Clash of the Paradigms

(or Rumble in the Epistemological Jungle)

clashoftheparadigms

We operate, on the whole, within education systems that are based on a traditional linear, positivist, computational paradigm that has been the dominant scientific paradigm since Newton et al back in the 17th century. It is a paradigm in which education is perceived as a form of industrial or mechanised process.

It is, essentially, a closed system, which is the sum of its parts (learners, teachers, curriculum, content, delivery, technology, etc.). By controlling these parts, we can regulate the performance of the whole system. Educational systems design is the process of regulating these closed systems. It is a system in which human behaviour and performance are assumed to be predictable within known circumstances, and in which knowledge is assumed to be an external, quantifiable object that can be transmitted to and acquired by learners. This enables patterns of behaviour to be analysed and used to make judgements about how learners are thinking or what they have learned.

It is a system in which a ‘line of determination’ is assumed between cause and effect: for example – teaching predictably causes learning. These assumptions over-simplify the world and tend to reduce human learning, performance and achievement to a repertoire of manipulable behaviours. But learning is far more complex and much less certain than these assumptions infer.

In one corner we have the dominant Positivist or Quantitative Paradigm which is based on the epistemological belief that all true knowledge is ‘scientific’ knowledge. In this paradigm there is a single objective reality ‘out there’ that is orderly, predictable, and can be studied, captured and understood by amassing data and triangulating it (I shall return to the triangle).

The overarching aim is to achieve explanation and control, which is possible because knowledge is objective, measurable, value-free and a quantifiable object that is transmitted by the ‘teacher as expert’ to, and acquired by, learners. Rigour is achieved via the ‘holy trinity’ of validity, reliability and generalisability.

In the other corner we have the Interpretive or Qualitative Paradigm in which subjectivity is inherent and should be acknowledged because complete or pure objectivity is impossible and should never be claimed. For those in this corner ‘truth’ is a matter of consensus amongst informed and sophisticated constructors, not of correspondence with an objective reality. Furthermore, because all measurement is fallible, there is great emphasis on multiple measures and observations in order to able to claim authenticity, and for that authenticity to be recognised.

Those who operate within this paradigm understand that there are multiple realities and that knowledge is subjective, contextualised and value-dependent. They aim for understanding in order to enhance learning, they are openly self-questioning and self-critical, and they welcome scrutiny and debate. Importantly, they view students as co-constructors of their learning, and perceive themselves to be partners and participants in learning as well as guides and mentors. (That position, by the way, does not prevent them from also being experts!).

In order to find a way to deal with all of this epistemological complexity in relation to how we approach assessment, I’m suggesting that one way – and of course there are and will be others – is to approach assessment as a form a qualitative research instead of a quasi-scientific investigation. If we choose to follow the interpretive paradigm in relation to assessment then we need adjust our thinking and our language. Essentially we need to do a form of ‘Find and Replace’.

We need to replace :

Validity with Credibility, Coherence, Consistency, Trustworthiness, Authenticity
Certainty with Relativity
Generalised Explanation with Local Understanding
Source Data with Empirical Materials
“Is it true?” with “Does it work?”
Single Point Perspective with Multiple Perspectives
the Triangle with the Crystal

To be continued……

Next instalment coming soon: Assessment at the Edge 3: Triangles and Crystals

Back to Assessment at the Edge 1

Assessment at the Edge 1: Faultlines

I know I’m not alone in feeling – increasingly as the years roll by – that all too often the way we assess is at odds with the way our students (and we ourselves) actually learn and experience learning. While I and everyone else round the assessment board table is doing their very best to be professional, to ensure that procedures and regulations are followed, and taking great care to ensure that students are treated fairly and reliably….a bit of my brain is suffering from a form of cognitive dissonance and saying ‘This is nuts!’

There used to be a one of those car stickers that went something like ‘Do Not Adjust Your Mind…There Is A Fault With Reality’. And that’s how it feels. There seems to be a serious disjunction or faultline between what appears on the hundreds of assessment print outs – actual or virtual – and the actual day-to-day experience of learning and teaching, of creating work, of pursuing ideas, of encouraging and enabling students to really stretch themselves, to try out new things, to fail gloriously, to boldly go.

As teachers we need to – and are required to – ascertain, with as much validity, reliability and fairness as possible, what our students know and understand. For most of us, learning, teaching and assessment is a form of journey along the highways and byways of a particular subject. We, the guided and the guides, explore the landscape of the discipline. Our role as guides, more often than not, is to enable those we guide to understand the meaning and significance of what is seen, what is heard, what is felt, what is experienced.

Occasionally, because as guides we take our work seriously, and there are matters of accountability and responsibility that need to be attended to, we stop and check to see how much those who have entrusted their education to us know and understand, and what they can do.

In order to assess our students we stop acting as guides and essentially become researchers or purposeful explorers. We set out to discover what they know and understand, and what skills they possess. We ask them, demand of them, to demonstrate their knowledge, understanding and skills. We assess them, evaluate them, judge them, measure them against a set of standards or criteria.

If it’s a relatively simple matter of fact or basic competence then it is relatively straightforward to test it. The student either knows who, or what, or when or how…or they don’t. But the landscapes we explore in education are highly complex, intricate, shifting, multi-layered, multi-faceted. Simple straightforward answers and simple straightforward questions are hard to come by. The terrain does not reveal itself easily. Nor should it. In such a landscape meaningful assessment is also highly complex, intricate, shifting, multi-layered and multi-faceted.

If we consider the types of assessment that dot the landscape, we can see a veritable bio-diversity of assessment. But this diversity is also a challenge, and it is worth noting just how many of these types of assessment result in assessment ‘data’ that is qualitative rather than quantative in nature.

But there is may be a problem with this: the more assessment involves qualitative information, the more subjectivity is involved. Now this would be mitigated and we would have improved reliability if we had strict or stricter assessment criteria and also more structured and proscribed content. But, and this is a big ‘but’, if we had those it would obliterate the essence of qualitative assessment in terms of flexibility, personal orientation and authenticity. Which brings us, eventually, to the question of assessment paradigms and to the Clash of the Paradigms.

Next instalment: Assessment at the Edge 2: Clash of the Paradigms

‘See me. Feel me. Touch me.’ (Pt. 1)

Object lessons and reflections on the HEA Arts & Humanities conference 2016

Brighton-HEA3

Early March. Brighton is an alluring place, despite the chill in the air. The sun is shining, the sea is blue, the promenade and beach lie temptingly just across the road from the conference venue, and the esoteric shops, cafés and bars of The Lanes are just a couple of minute’s walk away. So it was a testament to the commitment of the participants and the quality of the many and varied sessions on offer that so many were able to resist the temptation to ‘skip school’.

While, in some sessions and in Jonathan Worth’s fascinating keynote on the second day, there was an inevitable and valuable focus on the digital and the virtual, the most powerful message – for me – was the extraordinary pedagogic power of the physical, tangible object. From Kirsten Hardie’s opening keynote with accompanying green plastic teapot, pineapple ice bucket and toilet brush, to the Lego sessions of Contemplative Pedagogies, by way of Simon Heath’s wonderful drawings (see image below) that captured the essences of the whole event, it was the object that held centre stage. And there were plenty more sessions that focused on making and doing as a pedagogic activity, not just a practical or physical one.

Photo left: Hannah Cobb @ArchaeoCobb

I have written elsewhere (‘On history and all that’ ) on the power of objects to engage the imagination, to generate stories and lines of enquiry, to provoke philosophical, political, ethical debates, and to provide learning experiences that really ‘stick’. I still recall clearly the ‘History of Decoration’ seminars from my art student days when ‘Simi’ (Ms. Simeon the lecturer) would enliven her lectures on, say, Ancient Egypt, by taking a vase or piece of jewellery or some other artefact out of the cardboard box she always brought. She would casually hand the object to someone to examine and then pass around the room with the words ‘Do try to be careful, dear, that’s three and half thousand years’ old’. This would be repeated every session, whether the topic was Ancient Rome (jewellery), Medieval Europe (a crucifix) or Tudor England (a lace ruff). I only realised what we had been passing round when I heard that, on her death , Simi’s large collection of “just something to look at while I’m talking” had been bequeathed to and enthusiastically accepted by the V&A museum.

What also became clear during the conference, is that ‘object lessons’ are not just the preserve of the creative arts community. Every discipline clearly has its associated artefacts which can be used not only to enhance the teaching of an ‘academic’ subject, but to act as foci for the characteristics and qualities of the sort of learning that Kirsten Hardie talked about: learning that engages, amazes, provokes, exhilarates, takes risk, liberates.

One of the things I remember from those, now distant, art history sessions is something I frequently refer to in my work on curriculum design and assessment. In one her first seminars, Simi passed round an Ancient Greek vase that was covered head to foot in decoration. The reason, she said, for filling every possible square inch was ‘horor vacui’ – fear of open space – because it was through open space that the ‘Evil Eye’ enters the world. That might well be one of the reasons (though I would avoid mentioning the ‘Evil Eye’ or the Devil in module specifications and handbooks) why we insist on filling our curricula with content: ‘Idle hands make the devil’s workshop’ and all that. But we also know that deep learning, creativity and innovation require time and space to incubate and develop.

Objects, importantly, enable us to slow down time: to observe, to really look, to touch, to feel, to explore. Simon Piasecki, at the conference, talked about how he gets his performance students to slow right down and focus on the minutiae of what they are doing, and the artist Marina Abramovich – one of whose concerns is the fact that we don’t stop to really look any more – has a number of exercises she uses with those who come to view her work to achieve the same slowing down. When I worked at the Liverpool Institute of Performing Arts (LIPA), one of the first year ‘options’ that I established – open to any student – was a traditional life-drawing class. All the students that participated in that quiet, contemplative two hours on a Wednesday evening, amidst an extraordinarily hectic timetable (‘horor vacui’!), reported that they understood that it wasn’t about being able to draw. It was about having the time and space to slow down and really observe not only the ‘object’ (usually another student) but also themselves….and to ‘take a line for a walk’ in Paul Klee’s famous phrase.

Ken Robinson, in his now famous TEDTalk on creativity and education, jokes about academics generally seeing their bodies as a form of transportation to get them to meetings. He, among others, stresses the importance of mind and body, the intellectual and the emotional, the psychological and the physiological. What came through so strongly at the HEA Arts and Humanities conference was that objects – in all their glorious variety – and our close interactions with them, provide a means to engage powerfully in deep, meaningful learning experiences. Objects both inhabit space and create space. We just need the space, the time and, impotently, the confidence to engage in our own object lessons.

Brighton- HEA2

Photos by Paul Kleiman unless otherwise stated

Conference Twitter hashtag: #HEAArts16

The Emperor’s Folderol or Tales of the TEF

(with apologies to Hans Christian Andersen, adapted from the translation by Jean Hersholt.)

Many years ago there was an Emperor so exceedingly fond of excellence in everything and everyone that he spent all his time and money on ensuring that everything and everyone was, indeed, truly excellent. He cared for nothing else. He had a test and a metric for everything and everyone, and instead of saying, as one might, about any other ruler, “The King’s in council,” here they always said. “The Emperor’s pursuing excellence.”

One day the Emperor announced that he wanted to create a magnificent and complex instrument – to be called The Emperor’s Folderol – that would tell him quickly and accurately who and what was excellent and, importantly, who and what was not excellent. Those who were found to be excellent were to be amply rewarded, while those who were found to be unable to meet the Emperor’s expectations of excellence were to be punished severely.

The many courtiers surrounding the Emperor could not be bothered to pronounce ‘The Emperor’s Folderol’ in full, and so proceeded to refer to it as the ‘TEF’.

In the great city where the Emperor lived, life was always dynamic. Every day many strangers came to town (despite strict border controls) and among them one day came two swindlers. They let it be known they were expert Folderol constructors, and they said they could create the most magnificent TEF imaginable. Not only were the tests, metrics and materials they used uncommonly fine, but a TEF made from them had a wonderful way of becoming invisible to anyone who was unfit for his or her office, or who was unusually stupid.

“That would be just the Folderol for me,” thought the Emperor. “If I used it I would be able to discover which institutions and people in my empire are unfit for their purpose and posts. And I could separate the wheat from the chaff, the good from the bad, and the elite from the hoi polloi. Yes, I certainly must get a Folderol constructed of those special materials made for me right away.” He paid the two swindlers a large sum of money to start work at once.

They set up a large workshop with two imposing looking machines and pretended to construct the TEF, though the machines in fact produced nothing at all. All the fees and costs which they demanded went into their traveling bags, while they worked the machines far into the night.

“I’d like to know how those constructors are getting on with the TEF,” the Emperor thought, but he felt slightly uncomfortable when he remembered that those who were unfit for their position would not be able to see the product. It couldn’t have been that he doubted himself, yet he thought he’d rather send someone else to see how things were going. The whole town knew about the TEF’s peculiar power, and all were impatient to find out how stupid their neighbours were.

“I’ll send my honest old minister to the TEF workshop.” the Emperor decided. “He’ll be the best one to tell me how it looks, for he’s a sensible man and no one does his duty better.”

So the honest old minister went to the room where the two swindlers sat working away at the machines, producing nothing.

“Heaven help me,” he thought as his eyes flew wide open, “I can’t see anything at all”. But he did not say so.

Both the swindlers begged him to be so kind as to come near to approve the excellent tests, the beautiful metrics. They pointed to the machines, and the poor old minister stared as hard as he dared. He couldn’t see anything, because there was nothing to see. “Heaven have mercy,” he thought. “Can it be that I’m a fool? I’d have never guessed it, and not a soul must know. Am I unfit to be the minister? It would never do to let on that the TEF doesn’t appear to exist at all.”

“Don’t hesitate to tell us what you think of it,” said one of the constructors.

“Oh, it’s beautiful – it’s magnificent.” The old minister peered through his spectacles. “Such a comprehensive system of tests, what excellent metrics of genuine value and utility, such exquisite workmanship! I’ll be sure to tell the Emperor how delighted I am with it.”

“We’re pleased to hear that,” the swindlers said. They proceeded to name all the tests and to explain the intricate metrics. The old minister paid the closest attention, so that he could tell it all to the Emperor. And so he did.

The swindlers at once asked for more money, more fees and more costs, to get on with the construction of the TEF. But it all went into their pockets. Not a single useful component went into the machines, though they worked at the machines as hard as ever.

The Emperor presently sent another trustworthy official to see how the work progressed and how soon it would be ready. The same thing happened to him that had happened to the minister. He looked and he looked, but as there was nothing to see coming out of the machines he couldn’t see anything.

“Isn’t it a beautiful piece of work?” the swindlers asked him, as they displayed and described their imaginary construction.

“I know I’m not stupid,” the man thought, “so it must be that I’m unworthy of my good office. That’s strange. I mustn’t let anyone find it out, though.” So he praised the intricate workings he could not see. He declared he was delighted with the beautiful tests and the exquisite metrics. To the Emperor he said, “It held me spellbound.”

All the town was talking of this splendid TEF, and the Emperor wanted to see it for himself while it was still in the workshop. Attended by a band of chosen men, among whom were his two old trusted officials – the ones who had been sent to see what was going on – he set out to see the two swindlers. He found them working the machines with might and main, but producing nothing that he could see.

“Magnificent,” said the two officials already duped. “Just look, Your Majesty, what tests! What a design! What craftsmanship!” They pointed to the machines, each supposing that the others could what the machines were producing.

“What’s this?” thought the Emperor. “I can’t see anything. This is terrible! Am I a fool? Am I unfit to be the Emperor? What a thing to happen to me of all people! – Oh! It’s exquisite. So elegant.” he said. “It has my highest endorsement.” And he nodded approval at the empty loom. Nothing could make him say that he couldn’t see anything.

His whole retinue stared and stared. One saw no more than another, but they all joined the Emperor in exclaiming, “Oh! It’s wonderful,” and they advised him to place the TEF on the grandest carriage at the front of the great procession he was soon to lead. “Magnificent! Excellent! Unsurpassed!” were bandied from mouth to mouth, and everyone did his best to seem well pleased. The Emperor gave each of the swindlers a cross to wear in his buttonhole, and declared them to be ‘Knights of the Order of the TEF’.

Before the procession the swindlers sat up all night and burned more than six candles, to show how busy they were finishing the TEF. They pretended to move the machine into the centre of the workshop, and they spent hours pretending to polish it. And at last they said, “Now The Emperor’s Folderol is ready for him.”

Then the Emperor himself came with his noblest noblemen, and the swindlers stood either side of their imaginary machine and pointed to it. They said, “Here is the testing instrumentation, and here is the metric matrix, and here is the excellence calculator which is linked dynamically to the financial reward generator. All the components are made of the lightest and finest materials. No expense has been spared. The machine runs absolutely silently, and one would almost think there was nothing there, but that’s the whole point.”

“Exactly,” all the noblemen agreed, though they could see nothing, for there was nothing to see.

“If Your Imperial Majesty will condescend to accept our humble construction,” said the swindlers, “we will now demonstrate how it works, and after the parade we will provide you with detailed instructions on how to operate it.”

The Emperor nodded his assent, and the swindlers pretended to operate the machine. Turning a dial here. Switching a switch there. They invited each of the noblemen to enter the machine and stand between them. After more imaginary turning of dials and switching of switches, and taking great care to check the imaginary results, the TEF contructors declared each nobleman to be truly excellent and they congratulated the Emperor on his choice of courtiers.

And the courtiers said: “What a wonderful machine!”, “How clever your Majesty is to have thought of it!” “Those metrics, so perfect! Those tests, so suitable! It is a magnificent machine!”

Then the minister of public processions announced: “Your Majesty, the procession is waiting outside.”

“Well, if not now, when?” the Emperor said, and gave orders for the TEF machine to be carefully lifted and placed on the lead carriage. “It is a remarkable machine, isn’t it?”

The noblemen stooped low and reached for the floor as if they were picking up an exceedingly heavy object. “No, remember it’s extraordinarily light. Just be very careful” said one of the constructors. Then they pretended to lift and hold it high. They didn’t dare admit they had nothing to hold.

So off went the Emperor in procession, sitting in his throne behind the TEF machine, while several noblemen stood around the machine as it wound through the city. Everyone in the streets and the windows said, “Oh, what a wonderful machine! Look at the exquisite workmanship! What a truly excellent Emperor!” Nobody would confess that he couldn’t see anything, for that would prove him either unfit for his position, or a fool. No machine the Emperor had ever had built before was ever such a complete, excellent success.

“But there’s nothing there!” a little child said.

“Did you ever hear such innocent prattle?” said its mother. And one person whispered to another what the child had said, “There’s nothing there. A child says there’s nothing there.”

“But there’s nothing there!” the whole town cried out at last.

The Emperor felt his blood run cold, for he suspected they were right. But he thought, “This procession has got to go on, there is far too much at stake.” So he waved at the crowds more enthusiastically than ever, as his noblemen stood proudly alongside The Emperor’s Folderol that wasn’t there at all. And the excellent procession continued.

On beauty and elegance in education

In his book ‘Fearful Symmetry: the search for beauty in modern physics’, Anthony Zee describes how Einstein displayed a supreme disinterest in any proposed formula or solution, no matter how accurate it might be, that he considered ugly.

“As soon as an equation seemed to him to be ugly, he really rather lost interest in it and could not understand why somebody else was willing to spend much time on it. He was quite convinced that beauty was a guiding principle in the search for important results in theoretical physics.”

Today (13 Aug 2014), amongst the usual ugly headlines of death, destruction and disease, Maryam Mirzakhani is being celebrated as the first woman to have been awarded the prestigious Fields Medal – the Nobel Prize for mathematicians. Her work – as described by those who have some grasp of her achievements – has a “breathtaking scope, is technically superb and boldly ambitious”. She herself describes mathematics as full of “beauty and elegance”.

Now, I’d hate to think that beauty and elegance is the sole preserve of mathematicians dealing, like Mirzakhani, in esoteric fields such as complex geodesics, transcendental objects, and differential geometry. I’d argue that we all need at least a bit of beauty and elegance in our lives and work, and we certainly can see people striving for it (though many just don’t care) in many areas: whether it’s the presentation of food, the design of buildings and spaces, the arrangement of an exhibition, the movement of a dancer across the stage, the order and rhythm of words on a page.

So, why not strive for some beauty and elegance in education and in the curricula and learning experiences we design for our students? Yes, it’s often messy and a bit (or very) chaotic. But just as the mathematics of chaos have a certain underpinning beauty and elegance, we – as ‘architects of education’ – should strive to construct and compose learning and teaching experiences that flow and connect in ways that have a certain beauty and elegance about them. It’s not easy, but surely worth the effort.