Rene’s Assessment

Scientific, by Brittany G

I’m currently reading The Black Swan: The Impact of the Highly Improbable by Nassim Nicholas Taleb. It’s a refreshing and humbling account of our inability to understand randomness. Two of the arguments in the book stuck me as being very relevant to assessment.

The first of these is an urgent reminder of what Karl Popper taught us about the scientific method: A million confirmations of a thesis will not yield us much information, as we might have just missed the million-and-first circumstance in which the thesis does not hold. However a single falsification of the thesis will establish that is is untrue.  And so we can only really learn through falsification, not justification.

A summative assessment is really a test of the hypothesis that our learners are  competent. With this in mind, good scholarly practice dictates that we test this hypothesis by finding possible weaknesses in it, and attempting to expose them. So maybe we should not be looking for learners to demonstrate their competence, but we should be attempting to let them reveal any incompetence they might have.

With this in mind, there is perhaps reason to question certain assessment practices such as negotiated assessments, essays and portfolios. These all allow the learner to demonstrate competence largely on their own terms, without allowing us to test for incompetence on ours. It is the learner who takes the lead here in what to put on display for our prying eyes, which also provides the opportunity to perhaps not display the more questionable areas of their attainment. While from a motivational standpoint this has many benefits, it does seem to contradict the basic principles of the scientific method.

There are other approaches that seem better aligned with this principle. Exams are an obvious choice. But I also think a well designed problem based assessment can work in accordance with scientific principles, as long as the assignment is designed with the right pitfalls and challenges that could trigger falsification. A viva voce also provides an excellent opportunity for falsification, provided that the examiners question the candidate appropriately.

Is this something that should inform our practice? Is an assessment regime without a significant component focussing on falsification actually valid, or are we subjecting ourself to a fallacy?

The second topic links to how our behaviour is very domain specific, and what this might imply for assessment practice. I will save that for another post.

Tweet This Post

I was reading an article this morning which referred to the book ‘Academic Discourse‘. The book investigates the importance of language in learning. I think everyone will recognise that language, and in particular the jargon and linked body of concepts in a discipline, are a key part of learning. To engage effectively with a subject, it is important that one is familiar with important constructs and the way they are expressed and referred to. And so it is only logical that an important part of our teaching, and assessment, focuses on those key constructs.

In Higher Education, essays are often the medium of choice to evaluate learning. The wisdom handed down through the ages dictates that essays are suitable to assess higher order skills and understanding. But is that really the case? Of course the freedom to construct your own answer, or perhaps even choose which questions to answer, gives the student maximum freedom in expressing his or her understanding. But that freedom is also very easy to abuse.

Because we must realise that students aren’t always looking to express what they learned. They might be looking to meet the expectations that will lead to the desired result, usually a grade. And when pursuing this quest, students often find that writing a good essay is a problem that can be solved with some linguistically skills, and doesn’t necessarily require the attainment of any new understanding. And so in this light, perhaps we should investigate the value of very open and unfocused assignments. Because, while in a very different way then for instance multiple choice exams, they too can promote surface learning strategies when not designed with due care.

Furthermore, it also questions the value of computer marked essays. Most of these systems are designed largely around linguistic criteria, and so only exaggerate this problem. This is especially true if we consider the consequences of students understanding how their essays will be marked by such a system.

Tweet This Post

I just finished reading an article in the Times Higher Education in which it is suggested that exam papers should be published in advance to students to cut down on stress. This idea apparently stems from a paper published by the University Mental Health Advisors Network.

Now I hope this is an oversimplification of what the paper actually suggests. When taken literally, the suggestion seems rather awkward. Surely just publishing questions in advance would lead to all sorts of problems. Papers are often designed to test only a subset of the curriculum. This is only a valid approach in combination with a moratorium on the questions during learning (otherwise learning would most likely be limited to these questions).

What we need is to move towards more authentic and negotiated assessment, and away from the eternal exam and essay constructions. That is hardly a new notion however, and not really anything to do with disability in particular.

Tweet This Post

One of the frequent criticisms on e-Assessment is the perceived limits in item types that can be supported by technology. While there are long debates to be had about assessing higher order skills with constrained response item types, I don’t think these debates are going to take away the prime concern: Free text items.

I must say that I have serious doubts about marking free text by computer. I don’t know enough about the principles involved to say this with any sort of authority, but I am aware of the kind of heuristics used in automated essay marking for instance. These heuristics are often grammatical and rhetorical in nature, and have fairly little to do with the subject matter (although it must be said that many human markers have been shown to use similar heuristics). Nevertheless, interesting progress is being made in this area, and eventually I am sure that language processing will be commonplace.

One of the interesting project that I recently became aware of, is the OpenComment project, which is lead by Denise Whitelock at the Open University. The project is looking to use latent semantic analysis to analyse learners responses to open ended questions in history and philosophy. Another interesting fact is that the project is developing this as a question type in Moodle, and so it should be relatively easy for everyone to reap the benefits of this technology within their own learning environments.

Automated marking is by no means the only value of using technology in assessment. The OpenMentor project, again from the Open University, is a great example. OpenMentor compares a mark assigned to a piece of work to the amounts of positive and negative feedback given, and checks this for consistency. In this way it can help in the coaching process of new teachers. Given the importance of feedback, I think it’s wonderful to have explicit standards and training in giving it.

The ABC (Assess By Computer) software has so far escaped my radar. I wasn’t aware of it until queried by the Times Higher Education for the article they were doing. The software has a support role similar to OpenMentor, but this time the support is provided around the marking process. The software can highlight keywords, compare answers to model answers and more. All of this for the sole purpose of making it easier on the human marker, but also improve consistency between human markers. Especially the latter is very welcome I think, as marking open ended questions and assignments can sometimes be somewhat of a dark art.

I only just discovered that bits of the e-mail I sent to the reporter actually appear in the article. If I would have known that I probably would have paid a bit more attention to my grammar :S

Tweet This Post

I have often questioned the prejudice a lot of academics have in favour of essays, and against a lot of other means to assess learners. Perhaps this is a matter of how they were taught and assessed themselves. One the other hand I have often thought that this is a matter of a lack of training. After all, most lecturers do not get that much training in how assessment should be done properly. In addition most lecturers don’t have much time to spend on the assessment either. The result is an assignment that is easy to develop (although a lot harder to mark). Either way, this prejudice is one of the major barriers to the uptake of e-assessment. It is also a serious cause for concern about the validity of our degrees.

So it was with some curiosity and expectations that I started reading It’s not plagiarism, it’s an easy essay on the Learn Online blog, where an interview was posted with a provider of an online essay writing service. I thought it was rather appalling.

As mentioned, I’m no fan of essays. They are certainly overrated, overused and usually very poorly delivered. However I do not think they are useless. Someone’s critical thinking is rather wasted if it isn’t combined with the ability to express that thinking. If the learner has any sort of ambition to climb the corporate (or other) ladder, writing reports and proposals will be something they do regularly. So as long as essay assignments are given some sort of relevant subject and format, I think they are a very valid form of assessment.

The limited value of essays however does not validate the existence of services like this however. I don’t care how the service providers attempt to rationalize this, as is done in this article. It is just morally wrong to provide a service that is obviously designed to let people cheat. The audacity to claim that the objective here is to transform education baffles me. If you really want to change education, I could think of a million other and better ways of doing it then by making money out of helping people cheat. I have no respect for anyone in this line of business whatsoever.

Tweet This Post