Rene’s Assessment

One of the topics that came up several times over the past days in Reykjavik, is that of the differences in culture around assessment. Different countries have different ways in which they perceive and deal with assessment, and this can have a significant impact on the effect of the assessments, and the success of the educational system as a whole.

One particularly interesting approach was outlined by Jakob Wandall, who’s work in the Danish national tests I have blogged about last year in High stake national assessments and ranking. I tried to capture Jakob’s slide on a picture, but unfortunately that failed rather miserably, so I have tried to recreate his message in the graphic below:

The graph outlines how both the focus of the assessment (on the horizontal axis) and the purpose for which the results are primarily used (on the vertical axis) vary from country to country. I thought the visualisation was very interesting. Comparing this to, for instance, the outcomes of the 2006 PISA, it is interesting to note that neither the approach of the Scandinavian schools (who focus primarily on learner focused formative assessment) nor the Anglo-Saxon approach 9that is much more heavy on the measurements of indicators for performance, tied in to funding) really yields the best results.

The starts of PISA are of course the Finnish, and the unique approach is apparent from this graph. in stead of moving somewhere between the top left and the bottom right of the graph, they sit toward the top right. The Finish system highly values national measurements, evaluating the success of the system by objective measurements. However these measurements are not tied to any control, either through formal channels or more informal ones such as public rankings. In stead the measurements made in the Finnish system have the purpose to inform teaching and learning. An evidence based approach to teaching shall we say.

When I translate this to our own practice, I can’t help but relate this to demands to increase the amounts of formative assessment in our teaching. And while I am sympathetic to these demands, these assessments are similar to those in the top left of the above graph, informing and supporting individual learning processes. And so perhaps in stead of focusing primarily on formative individual assessment, we should focus (also) on assessment and evaluation that informs teaching. Building an infrastructure through which lecturers can stay in touch with the progress, successes and difficulties of all their students, and modify their teaching based on this understanding continuously.

Tweet This Post

At the EU workshop I attended in Ispra, Italy last year (see blogposts Psychometrics versus pedagogy and High stakes national assessments and ranking) we agreed to write some articles on quality aspects of computer based assessments to go towards a report for the European Commission. I’m glad to say that the report has now been published, and can be accessed online via the following link: Towards a research agenda on computer-based assessment

I think there’s many interesting articles and views within the report, and I will certainly be reviewing the interesting perspectives that my colleagues presented at the workshop in this report. Do have a look, I am positive there will be something of interest there for virtually anyone.

Tweet This Post

One of the things I noticed during the workshop I attended last week, is the fundamental differences to the approach of computer based assessment between psychometrists and educators. it all boils down to the use and value of statistics.

I think within education we often don’t evaluate our teaching and assessment practice enough, in particular by means of objective standards. For assessment practice the most well known methods of scientific evaluation are Classical Test Theory and, most importantly, Item Response Theory. I don’t think many educators really bother with these, and some will stare very blankly should I ever bring up these terms in conversation. In stead we rely on our 6th pedagogic sense that rather mysteriously enables us to divine what assessment methods and questions work, and which do not.

The psychometric approach is radically different. Almost religiously sometimes, items are tested and analysed using IRT. the most meticulous detail (question order, item order, delivery medium etc.) is reviewed for it’s influence on responses. These statistical analysis only focus on one thing though, and that is the alignment of the item with the overall test. What the church of IRT seems to sometimes forget however, is to question whether or not the test in itself actually measures what it is assumed to measure. To a degree IRT is a bit of a circle argument, if not used carefully and in conjunction with other arguments.

It seems to me we could both do with a bit of each others zeal. Educators should really try and build in some structured objective evaluation of their assessment practices, and psychometrists should perhaps question the appropriateness of their means more fundamentally.

Tweet This Post

It’s funny how things coincide sometimes. Today a student came to our office, wondering if we could help her develop an assessment in support of her dissertation. For some reason several students have made this request this year, while none used to before. Personally I think it’s a wonderful thing, and testament to how students now view technology as an integral and important part of their lives and careers. It also shows how they are much less subject to pigeonholing technology.

For years the Centre for Interactive Assessment Development has been supporting lecturers by developing rich assessments. The university of Derby has in general adopted a far more innovative approach then most, progressing e-Assessments far beyond the domain of the multiple choice quiz. still the applications sought for innovative assessment practice have been rather limited. Primarily assessments were measurements of learning, mostly summative or formative only in the sense of providing practice and a benchmark for a later summative exercise. Assessments that actually teach, or diagnose are a relative new addition to our portfolio. Assessments for other purposes, such as research or evaluation, have never even been considered as part of the centres value and expertise. This is something I am desperate to change. I’m glad at least students seem to agree with me on that one.

Tweet This Post