F-AI-L: forget about assessment?

You can’t move these days in Higher Education without reading about how AI has completely compromised most, if not all, the approaches to assessment that require a student to produce an artefact e.g. an essay, a poster, a portfolio, etc.  It would appear that the only way to ensure ‘assessment security’ is to, first, thoroughly body scan and search each student for any devices – external or internal – that might be used to ‘assist’ them to pass the exam and then lock them in a Shielded Room or Faraday Cage under very strict and tight surveillance. Or, as a colleague commented at the recent Assessment in Higher Education Conference (AHE) conference: “March them naked into the exam room armed only with a pencil”.

Not exactly an enlightened or constructive educational experience. 

Having focused on assessment for much of my career, I’m beginning to wonder whether we should give up on assessment altogether. But there is one secure form assessment, one that we have practiced for centuries, that goes back to assessment’s Latin root ‘assidere (to sit together or beside)’: the viva. Simply sitting down with a student and asking them or observing them rigorously and systematically in regard to what they know, what they understand, what they have learned, what they think and what they can do.

There are, essentially, two questions that a university has to answer in regard to their graduates: 1) does the student possess sufficient knowledge, understanding and skills in their particular discipline in order to be awarded a qualification? and 2) does the student possess sufficient knowledge, understanding and skills to stand a good chance of successfully navigating their way through an uncertain and complex world? 

Given that many graduates (an old figure in the UK was c. 48% but I suspect it hasn’t changed much) end up working in a field unrelated to their course of study or discipline, the efficacy of the first question is…..questionable (except in the case of professional qualifications e.g. doctors, engineers etc.). As to the second question there’s an assumption they are ready to go out into the ‘real’ world, but that remains an assumption in many cases. 

So…..what if we scrap assessment as we know it?

One thing we do know about assessment is that the inter-rater reliability of a group of experts observing, discussing and assessing a student’s work tends to be higher than when those same experts assess against a set of criteria or rubrics.   

I’m suggesting we scrap the inordinate amount of time and energy spent on marking and grading assessments that are now insecure and unsafe and replace them with regular ‘assessment tutorials or mentoring’ with at least two members of staff. 

In order to do that properly those tutorials must be built into the teaching/contact hours. Teaching is no longer about ‘delivery’ i. e. a teacher-centred and content-oriented paradigm, and shifts to a student-centred, learning-oriented paradigm in which assessment is about sitting down with a student and discovering the answers to those two  questions. 

About me:

I am an independent higher education consultant and co-founder of the educational consultancy Ciel Associates specialising in organisational and transformational change and enhancing learning, teaching and assessment. I have over thirty years experience in a wide range of higher education providers both as a consultant and, before that, in a number of management, leadership and strategic roles. My work is informed by my belief in the transformational power of education and a commitment to enabling institutions to create meaningful and sustainable change. Contact me at Ciel Associates .

Adventures in HE: time for a Skunk Works?

A few years ago, a UK university established a new initiative with funding from central government. The initiative was one of about 70 such initiatives across the HE sector which were established in the hope that, within the five years of the funding, the universities involved would have taken risks and pioneered innovative learning approaches that could be adopted and/or adapted across the sector. 

The university in question established a project with the aim of developing  more practice-based arts provision within the university, establishing more links between the university and external artists and arts organisations, and to increase the links between the subject areas within the creative  and performing arts programmes within the university. The project rapidly established a reputation within the university for taking risks and for hav­ing a different way of looking at things. It was also known for having a different way of talk­ing about things and even a different way of naming things, for example, modules. A critical factor in the project’s ability to forge an innovative path was that its director had been brought in from outside academia, and was relatively and refreshingly unencumbered by things like history and precedence and the engrooved customs, practices and discourses of the institution.

The director had created a Skunk Works.

Skunk Works are, typically, small and loosely structured groups of people who research and develop a project for the sake of innovation. There are several important and essential features of a Skunk Works: one is that it operates independently of the normal research and development operations of the company or organisation; another is that, in order to achieve unusual results, the people working on the project work in ways that are outside the normal ways of working. 

One of the major projects initiated by the director was the development of a pilot MA in interdisciplinary arts practice. The aims of the course were to explore and extend the boundaries of various arts disciplines, and all aspects of the course were designed to be innovative and radical – the curriculum, the delivery, and the assessment. As with all courses, the proposal had to go through the university’s validation process and there were intense discussions about what might or might not be acceptable, especially given the relatively traditional academic culture of the university. The director stuck to her guns and, much to the surprise and, indeed, chagrin of some colleagues, the university validated a course that included modules called ‘Adventures in Interdisciplinarity’ and ‘Further Adventures in Interdisciplinarity’ in which the course participants– a mixture of recent graduates and mature, part-time students– gathered on a Friday and worked intensively through the weekend on various experimental projects and exploratory assignments with artists, directors, film-makers and composers. 

These ‘Adventures’ were designed to provide unusual, typically hazardous (creatively, intellectually, etc.), experiences or activities.  Calling the modules ‘Adventures’ was a very deliberate move and was highly resonant of Ronald Barnett’s call to educators to ‘hang onto a language of delight, wonder, care, excitement, fun, engagement and love– the language in which a student is caught and even entranced’ (Barnett, 2008).

At a time of particularly rapid change with a host of serious challenges facing higher education, perhaps establishing a Skunk Works and investing in the space to breathe, to invent, to imagine, to innovate, and to create something that is different, exciting, and genuinely transformational might be a very good and beneficial move.

Barnett, R. 2008. A will to learn. London: Open University Press.

Image created by AI

About me:

I am an independent higher education consultant and co-founder of the educational consultancy Ciel Associates specialising in organisational and transformational change and enhancing learning, teaching and assessment. I have over thirty years experience in a wide range of higher education providers both as a consultant and, before that, in a number of management, leadership and strategic roles. My work is informed by my belief in the transformational power of education and a commitment to enabling institutions to create meaningful and sustainable change. Contact me at Ciel Associates .

Assessment: paradigm shift required?

There has been a slew of recent journal articles, blogs, podcasts etc.  on the challenges posed by Generative AI in higher education and, particularly, the threat GenAI poses to assessment integrity and security along with  possible approaches to mitigate the threat. 

The picture that emerges is one in which none (perhaps bar one: the viva?) of the traditional forms of assessment are secure – were they ever? It seems clear that not only does a new assessment paradigm need to emerge but also a new paradigm for learning and teaching. As Gramsci wrote in the 1930s: ‘The old is dying and the new is yet to born: in the interregnum, all kinds of morbid symptoms appear’.

The challenge we face in regard to somehow ensuring or at least maximising assessment integrity and security in the face of GenAI is that, essentially, assessment – in the age of mass higher education – has become an economic and logistical issue, not a pedagogical one. An industrialised process of mass production that relies on the production-line workers (lecturers) doing much of the quality control i.e. assessment and marking, in their spare time in order to meet the production deadlines

What has largely disappeared from the way we assess students is the idea of assessment, inherent in its Latin root ’assidere’ (to sit together or sit beside), as a dialogic process. What hasn’t disappeared, though somewhat diminished, is the dominance of the teacher-centred, curriculum-focused paradigm rather than a student-centred, learning-focused paradigm. 

In a recent and worth reading GenAI and assessment-focused journal article by Guy J. Curtis*, he wrote: ‘Students can reasonably expect to have their learning assessed on what they have been taught.’ He’s right but, just as important, students should also expect to be assessed on, or at least have an intensive discussion about, what they have learned. This may well (and should) extend beyond disciplinary specificity as they head towards an increasingly complex and uncertain future in which, whether we like it or not, AI will play an integral role.

So, while we struggle to find ways to make assessment secure without resorting to unacceptable and detrimental levels of surveillance, we might usefully attempt to answer the question: how might we best assess students in ways that reveal what they actually know, what they have learned and what they can do? The pedagogical answer may well be by sitting beside them and questioning them rigorously and systemically. But that begs another question: is higher education willing and able to shift to a new assessment paradigm? 

* Guy J. Curtis article

Image created by AI

About me:

I am an independent higher education consultant and co-founder of the educational consultancy Ciel Associates specialising in organisational and transformational change and enhancing learning, teaching and assessment. I have over thirty years experience in a wide range of higher education providers both as a consultant and, before that, in a number of management, leadership and strategic roles. My work is informed by my belief in the transformational power of education and a commitment to enabling institutions to create meaningful and sustainable change. Contact me at Ciel Associates .

Notes from the edge: piano lessons

I aim to play the piano most days, if I am near one. I work from home a great deal of the time, and I do a lot of writing. The work often involves some complex problems – large and small – that need to be addressed. When I am stuck, simply fed-up and frustrated or just need a break, I’ll go to the room with the piano and play for 10, 20, maybe 30 minutes. When I sit at the piano, I might choose to run through one or two of the classical pieces I’ve learned to play reasonably well over the years. Or I might choose a jazz or popular standard that I’ve picked up by ear, which involves a bit of improvisation in that sense of working relatively loosely within a recognised framework. I never play the same tune in exactly same way: but then, who does?

Usually I just place my hands on or over the keys, and I wait to see what happens. I have no idea of what is going to happen before it takes place. Something stirs. Something starts. A note or a chord is played. And off I go. Or off ‘it’ goes, because I feel I’m not in conscious control of my fingers. I am, of course, but it doesn’t feel that way.

David Sudnow, in his now classic work Ways of the Hand (1978) which is a remarkable insider’s account of learning to improvise jazz piano that was based mostly on his own introspection, describes having the most vivid impression of his hands making music by themselves. Sometimes, for me, it feels a bit awkward, as I travel down some musical cul-de-sac or find myself in a particular and sometimes too-familiar groove. Other times it just flows, I’m ‘in the zone’, and I know, especially when it really flows, that it clears and refreshes not only my mind but also my spirit.

In relation to ‘flow’ and Czikszentmihalyi’s influential work on that topic, we know that performances that combine flow states with a degree of risk taking might hold the key to achieving optimal levels of musical communication in improvisation. Being in the flow or ‘groove’ sometimes enables experienced improvisers to move beyond or extend their previous cognitive limits.

Sudnow uncovered many principles about learning, and particularly, what we might call embodied learning. His analysis and observations resonate powerfully with the work and research around creativity in learning and teaching, and what might be referred to as learning or teaching at the ‘edge of chaos’.

Photo by Paul Kleiman

”It’s cheating Jim, but not as we know it”: the problematic arts of plagiarism

Recently, while the academic world attempts to negotiate its path through the minefield posed by Generative AI, I was looking at some university policies about academic integrity and stumbled across these two statements:

There are few intellectual offences more serious than plagiarism in academic and professional contexts.

To submit a paper or comparable assignment that is not truly the product of your own mind and skill is to commit plagiarism.  To put it bluntly, plagiarism is the act of stealing the ideas and/or expression of another and representing them as your own.  It is a form of cheating and a kind of scholastic and professional dishonesty which can incur severe penalties.  It is important, therefore, that you understand what constitutes plagiarism, so that you will not unwittingly jeopardize your college career.

Reading those dire warnings about academic misconduct took me back to the now seemingly far off days when we obsessed about plagiarism and you couldn’t move for workshops and seminars on topics such as ‘Designing Out Plagiarism From Assessment‘. But it also reminded me that for anyone with an arts background or a knowledge of art, literature or music history, the idea of plagiarism as a black and white issue is an absurd idea. There’s a lot of grey.

Consider this famous engraving of ‘The Judgment of Paris’ by Raphael, created circa 1510-20, and pay particular attention to the group of three figures on the lower right. Do they remind you of anything?

Let’s now skip over the engravings made soon after by Raimondo (left) and Marco Dente da Ravenna (right)….

….and consider this famous painting from the 19th century.

As Manet’s figures are clearly based on the three figures in those engravings is the painting ‘truly the product of his mind and skill’?

But the chain of ‘borrowing’ continues. Picasso sees the Manet painting and creates this, at least acknowledging his immediate source (or at least the Musee Picasso is acknowledging the source)

Then, in 1981, this album cover appears:

Or consider these album covers and that university statement about there being ‘few intellectual offences more serious than plagiarism in academic and professional contexts.’

It seems clear, particularly in regard to art works, that a conception of the creative process that imagines that new works are original and autonomous may often be at odds with actual acts of creation that in many instances involve copying, directly referencing, adapting and other uses of existing works.

While some artists e.g. Manet and Picasso are lauded for their appropriation of previous art, the popular artist Jack Vettriano, who died recently, was frequently dismissed by the art establishment for being derivative and unoriginal (as well as popular!). The often quoted example is that the two dancing figures in his most famous work ‘The Singing Butler’ (section below) – the most popular art print in the UK – were virtually direct copies of the dancing figures from The Illustrator’s Figure Reference Manual.

In a number of disciplines and fields of study terms such influence, intertextuality, formulaic cultural production, appropriation and borrowing are important parts of the disciplinary discourse. In art and literary criticism, terms such as intertextuality, allusion, quotation, and influence are used, In musicology terms used to discuss relationships between musical texts include borrowing, self-borrowing, transformative imitation, quotation, allusion, homage, modeling, emulation, recomposition, influence, paraphrase, and indebtedness. Brahms, for example, openly admitted the strong influence of Beethoven. His First Symphony is sometimes referred to as ‘Beethoven’s Tenth’ and Brahms famously stated, “You have no idea how it feels to hear his footsteps constantly behind you.” In the context of plagiarism in popular music, the work of the forensic musicologist Joe Bennett is worth reading and listening to. One of the problems with identifying plagiarism is that, if and when it comes to court and as Bennett also makes clear in his work and research, what sounds superficially similar – certainly on first hearing – to a jury of ‘ordinary people’ is actually far more complex. Certain combinations of notes and chords are so ubiquitous across the musical landscape that they have become ‘commonplace’ items and are, in fact, common property.

As examples accumulate it becomes apparent, as Jonathan Lethem wrote in The Ecstasy of Plagiarism (2007) ‘that appropriation, mimicry, quotation, allusion, and sublimated collaboration consist of a kind of sine qua non of the creative act, cutting across all forms and genres in the realm of cultural production.’

The final example, returning to the two images at the start, is the cautionary tale of a book cover from 1974 and a very large painting that was a Turner Prize finalist in 2000. The full story of the plagiarism furore it caused and the law suit that followed can be read here https://artquest.org.uk/artlaw-article/originality/ , but here are the bare bones.

In 1974, Robert Heinlein’s book ‘Double Star’ was published with a cover created by the sci-fi artist Anthony Roberts. In 2000, the artist Glenn Brown’s large painting ‘The Love of Shepherds’ was chosen as a finalist in that year’s Turner Prize. Brown frequently uses the work of other artists in developing his large-scale work and is known for the use of art historical references in his paintings (as did Manet and Picasso). Starting with reproductions from the works of other artists, his biography states that he “transforms the appropriated image by changing its colour, position and size”.

Anthony Roberts was alerted to the similarity with his own painting for the book cover after a visitor to the Turner Prize exhibition noticed the similarity and the story hit the headlines. Roberts sued Brown and eventually the case was settled out of court. The painting is now titled ‘The Love of Shepherds’ (after ‘Doublestar’ by Anthony Roberts).

When a university states that students must “understand what constitutes plagiarism, so that you will not unwittingly jeopardize your college career” perhaps they should clarify precisely what they mean. If they are referrring to plagiarism in a purely academic context i.e. writing essay, dissertations etc. then what is and is not plagiarism is, or should be clear. But beyond that, particularly in the areas of creative practices, we enter the ‘Grey Zone’, and we should be ensuring that understanding plagiarism also means enabkling students successfully and effectively to understand, appreciate and negotiate their way through that grey zone.