Rene’s Assessment

Authentic

I’ve referred to the idea of authentic assessment before. Most recently in my previous post, in which I promised to delve into the subject again.

In his book The Black Swan: The Impact of the Highly Improbable , the author Nassim Nicholas Taleb mentions an experiment done in 1971 by psychologists Danny Kahneman and Amos Tversky. In the experiment the duo challenged statisticians with basic statistical problems that were presented as normal everyday issues, and not as statistical problems. Interestingly a large number of statisticians failed a significant number of these challenges which illustrates how  our understanding, and especially triggering it’s application, is very domain specific.

Sovjet-jeep

A slightly different anecdote on this issue relates to attempts of the US army to train neural networks in the recognition of enemy  vehicles. After a long training period using pictures of both allied and (at the time) Soviet vehicles the neural network seemed to have learned to flawlessly identify friendly and enemy craft. The next stage of the project involved working with actual vehicles, and the project moved outdoors to a desert where, to the surprise of the military, the neural network suddenly failed to identify any of the collected Soviet machinery as belonging to the enemy. After careful analysis it had turned out that the neural network had actually learned to distinguish the latitude by looking at the length of the shadows on the presented photographs. As the training pictures of the Russian craft were taken in Russia, this worked fine. in stage 1, but stopped working when the actual vehicles were presented on lower US latitudes.

What I’m trying to illustrate here is a few reasons to make assessment, but also learning, as authentic as we can. We want to be sure that our learners have learned something they can apply in a real situation. We also want to ensure that when we assess them, we assess the right constructs, so that the achievement on the assessment will correlate to a ‘real-world’ capability, and not just to an entirely academic one, or worse, one with no relation to the subject at  hand, such as the latitude on which pictures are taken.

Tweet This Post

Scientific, by Brittany G

I’m currently reading The Black Swan: The Impact of the Highly Improbable by Nassim Nicholas Taleb. It’s a refreshing and humbling account of our inability to understand randomness. Two of the arguments in the book stuck me as being very relevant to assessment.

The first of these is an urgent reminder of what Karl Popper taught us about the scientific method: A million confirmations of a thesis will not yield us much information, as we might have just missed the million-and-first circumstance in which the thesis does not hold. However a single falsification of the thesis will establish that is is untrue.  And so we can only really learn through falsification, not justification.

A summative assessment is really a test of the hypothesis that our learners are  competent. With this in mind, good scholarly practice dictates that we test this hypothesis by finding possible weaknesses in it, and attempting to expose them. So maybe we should not be looking for learners to demonstrate their competence, but we should be attempting to let them reveal any incompetence they might have.

With this in mind, there is perhaps reason to question certain assessment practices such as negotiated assessments, essays and portfolios. These all allow the learner to demonstrate competence largely on their own terms, without allowing us to test for incompetence on ours. It is the learner who takes the lead here in what to put on display for our prying eyes, which also provides the opportunity to perhaps not display the more questionable areas of their attainment. While from a motivational standpoint this has many benefits, it does seem to contradict the basic principles of the scientific method.

There are other approaches that seem better aligned with this principle. Exams are an obvious choice. But I also think a well designed problem based assessment can work in accordance with scientific principles, as long as the assignment is designed with the right pitfalls and challenges that could trigger falsification. A viva voce also provides an excellent opportunity for falsification, provided that the examiners question the candidate appropriately.

Is this something that should inform our practice? Is an assessment regime without a significant component focussing on falsification actually valid, or are we subjecting ourself to a fallacy?

The second topic links to how our behaviour is very domain specific, and what this might imply for assessment practice. I will save that for another post.

Tweet This Post

Time for some reflection, and answer the big question for December 2008: what did you learn about learning in 2008? And in doing so, perhaps time to come clean about my own learning disabilities.

Geek Inside

I’m a classical geek: A bit of a recluse, overly rational and sometimes perhaps lacking a bit in some social graces. On most days I read more then my girlfriend talks. I will trust you to correctly apply the stereotypes in that sentence. They are characteristics that, for the most part, have actually served me quite well. Our education system has bestowed the most extraordinary amount of positive action on the minority of people like me. People that can, and are willing to, learn at least some things solely by engaging with the written word.

The problem is of course that this is not how most people learn, and it’s not how the most important things are learned. But it’s easy to stick with something that works, and so it has taken me quite a bit of time to really understand that. To understand that we only really learn by (inter)acting. And while the education system is still accepts, and even favours people with similar disabilities to mine to cling to their narrow learning habits, modern social and professional life make it increasingly hard to be successful with that disposition. Because this disposition is suited primarily for information gathering. And information is of course just another commodity, and an increasingly cheap one.

But aside from this personal realisation, this is something that needs to be translated to our practices, and our institutions and systems, which is not trivial. How are we going stimulate learning, if we are not teaching? How do we verify that learning has taken place, if we aren’t sure how create authentic and valid assessments for the competencies we are now aspiring to instil? And aside from solving these challenges on an intellectual level, how will we actually implement them and change the entrenched system of education in which we work. It’s a worthy challenge, one for after the Christmas holidays…

Tweet This Post

My brother pointed out this great TEDtalk to me yesterday by Susan Savage-Rumbaugh, called Apes that write, start fires and play Pac-Man. it’s a great watch, and if you have 20 minutes I would certainly recommend you have a look, as it is quite amazing how much these primates are like us, and how much they can learn. I was also intrigued by what she said about how they learn:

“The most important thing is not to teach them.”

The Bonobos pick up things very naturally and freely  as long as they are in an environment that is fun, with meaningful individuals that they like to interact with. They watch, experiment and learn. And in this Bonobos are probably a lot like us. I think most of us are starting to understand that learning isn’t about knowledge transfer. It is about making connections to concepts, activities and individuals that are meaningful to us in some way. And doing so is not something we can be told to do, it is something we have to want to do, and allowed to do freely, because it has a relevance, an importance to us.

Anyway, I thought it would be a good motto for 2009: The important thing is not to teach them!

Tweet This Post

It’s official, I gave up on Drupal. I’m sure it’s a great tool, but I just don’t really care for the steep learning curve that it demands. So I moved the wordpress blog to the root of the site, and that’s where it will stay for the time being. Apologies for all the messing around. Some of you may have received old posts reactivated in their feeds, and I am still going through all my old posts to make sure the internal links work as we speak. However, from now on the major moves should be over (even if just to allow me some time to check out why my oven chose the week before Christmas to break down…)

Thanks for your patience, and a merry Christmas.

Tweet This Post