Rene’s Assessment

I attended a very interesting seminar this afternoon by Zoltan Dienes about unconscious knowledge. While the seminars  focused more on how to assess unconscious knowledge, it was the underlying models about knowledge and decision making that caught my attention. Zoltan presented 4 basic ways of making decisions:

• Guessing
• Intuition (based on a subconscious understanding)
• Rules (based on a conscious understanding)
• Memory (based on identical previous situation)

Modes of decision making

It struck me that each of these are a compromise between accuracy and generalisability, as depicted in the illustration on the right. In formal education we tend to focus on rules. It all seemed logical at the time, but now I can’t help but wonder why.

I’m quite ok with ignoring guessing and memory, they represent extremes of the spectrum that either have very little accuracy, or very little applicability. But intuition actually seems very valid, especially ina world in which rules and facts are changing at an ever increasing speed.

Intuition is crucial in domains where rules are non-existent, or too complex to support a timely decision. But it is also important to recognise that intuition can be a precursor to a more  conscious understanding of rules. In both cases, experience, wether actual or through games and simulations, are crucial media for learning. Ideally this learning should include at least 2 very different experiences based on the pattern we wish to develop. In the case of the further development and refinement of this intuition towards rules, these games could be followed by a meta game, or game analysis, in which rules are reconstructed.

This model is a replacement of the model that uses facts as a precursor to rules. That model perhaps had some validity in a more static world, requiring a highly specialised workforce. But today I think this has little value, and we are better off developing our instincts, then ourknowledge of facts.

Tweet This Post

Authentic

I’ve referred to the idea of authentic assessment before. Most recently in my previous post, in which I promised to delve into the subject again.

In his book The Black Swan: The Impact of the Highly Improbable , the author Nassim Nicholas Taleb mentions an experiment done in 1971 by psychologists Danny Kahneman and Amos Tversky. In the experiment the duo challenged statisticians with basic statistical problems that were presented as normal everyday issues, and not as statistical problems. Interestingly a large number of statisticians failed a significant number of these challenges which illustrates how  our understanding, and especially triggering it’s application, is very domain specific.

Sovjet-jeep

A slightly different anecdote on this issue relates to attempts of the US army to train neural networks in the recognition of enemy  vehicles. After a long training period using pictures of both allied and (at the time) Soviet vehicles the neural network seemed to have learned to flawlessly identify friendly and enemy craft. The next stage of the project involved working with actual vehicles, and the project moved outdoors to a desert where, to the surprise of the military, the neural network suddenly failed to identify any of the collected Soviet machinery as belonging to the enemy. After careful analysis it had turned out that the neural network had actually learned to distinguish the latitude by looking at the length of the shadows on the presented photographs. As the training pictures of the Russian craft were taken in Russia, this worked fine. in stage 1, but stopped working when the actual vehicles were presented on lower US latitudes.

What I’m trying to illustrate here is a few reasons to make assessment, but also learning, as authentic as we can. We want to be sure that our learners have learned something they can apply in a real situation. We also want to ensure that when we assess them, we assess the right constructs, so that the achievement on the assessment will correlate to a ‘real-world’ capability, and not just to an entirely academic one, or worse, one with no relation to the subject at  hand, such as the latitude on which pictures are taken.

Tweet This Post

Assessment has been in the news an awful lot lately, albeit not very positively. There is of course the whole SAT’s palava, but I will resist the temptation to comment on that. My position on this is outlines in previous posts on this blog, and I can only say that it is good to see that a lot of the momentum around this seems to be finally heading in the right direction. Its a shame we often need some sort of disaster to finally be open to change. A more surprising current issue is that of the Dyslexic student’s exams battle. Which deals with a medical student’s problems with multiple choice tests, something further clarified by the BBC in a follow-up article: Why can’t people with dyslexia do multiple choice?

The comment by the student’s solicitor that “Every professional body or employer who relies for a professional qualification, or as a promotional gateway, on multiple choice questions is heading for a fall.” is of course a bit of a joke. Quite frankly I am rather appalled by what seems like a rather misguided attempt to ‘make a splash’ at the expense of something as crucial as our exams system. While there are many gripes that you could reasonably hold against multiple choice question, I don’t think the link to dyslexia is really that valid. Considerations around presentation, or even using screen readers, can reasonably address most potential issues that might result from a disability. in addition, I think we should not shy away from critical reflection on the degree of special provisions that we put in place to accommodate students, as these provisions could significantly alter the nature of an assessment and then compromise the validity and equitability of the award. There will always be differences between learners in how well they perform in various types of assessment. This is one of the reasons to make sure there is a variety of assessment methods being used.

The more interesting question though, is around authenticity. The student in question is quoted in the article, saying that “In normal day life, you don’t get given multiple choice questions to sit. Your patients aren’t going to ask you ‘here’s an option and four answers. Which one is right?”. And to an extend I think she has a point there. While there will always be situations in which we will have to rely on ‘proxy’s’ to infer attainment, I do agree that currently we rely way too much on proxy’s that are sometimes quite remote from the competencies that we try to measure. In this sense education system is stuck in it’s traditions, in stead of applying the objective and critical reflection that we say we value so much in higher education.

A similar point, and some suggestions for moving forward, are made in the blog post 21st Century Assessment, where this ‘formula’ is proposed for a modern fit-for-purpose assessment system. Especially the elements of collaboration and peer assessment are extremely important and very much underutilised in our current practice. Partly I suspect that this links in with how uncomfortable we still are with the loss of our position as the holder and tranferrer of all knowledge. This role warranted a 1 to many broadcast model of education. Education today however is moving much more towards a many to many model, whereby the role of the teacher is much more one of guidance, coaching and accreditation of a learning process that involves peers, external resources and actors and experiences from previous professional roles. I’m not quite sure we are really ready to fulfil that role yet though.

Tweet This Post