Rene’s Assessment

I’ve been at the Questionmark European User Conference 2009 in Manchester the past 2 days. I must say that I have never been a great fan of Questionmark, and the previous time I looked at the product (in 2006) I found it a terribly unstructured and cluttered beast. Then again the big problem with computer based assessment technology, is that there is very little of it that is any good. Some of it is terribly technical, and requires high amounts of software development or psychometric understanding (or both) to be useful. The vast majority of it is very feature poor, in particularly when it comes to item-types. And then there is the enormous graveyard of failed Open Source projects funded by JISC and others that never made it to a stage of maturity, and adoption by anyone beyond the developers.

And so when looking at a replacement for the in-house system that we have developed over the past decade in Derby, choice was limited. After a long and painful stage of denial, we ended up having to admit that, short of a major development effort, Questionmark was probably one of the few viable alternatives. And although some of it’s inherent weaknesses remain, there are also a number of interesting developments that have actually sparked my sincere interest.

The most important of those is the Open Assessment platform. Like other vendors (such as Blackboard) Questionmark seems to have understood the importance of the Open Source movement. Every vendor deals differently with this phenomenon, but Questionmark seem serious about embracing it constructively. They are working on opening up their API’s, but also Open Sourcing community editions of integration software, such as connectors to Moodle, Blackboard and Sharepoint. This I think is a great start in opening up the product, and creating added value through collaboration with, what will hopefully be, a vibrant community of developers.

What I really hope for though, is that this openness will extend to the data, and the application framework itself. There are thousands of specialised use-cases, in particular question types, that are highly desired by higher education but that will never be a viable commercial priority for a vendor like Questionmark. However if a University could extend the Questionmark platform and create these question types, and better yet if several universities would do this and share their efforts, the value of the Questionmark platform would increase dramatically. What we need for this to happen is an application architecture for both the authoring environment and the assessment rendering engine that would support extentions, plug-ins, just like Firefox and so many other modern applications do. That way we could create our own question templates that could be authored and delivered from within Questionmark. (This short of questions being true objects that can expose themselves in authoring or delivery mode, but I will spare you that highly technical argument against the current question definition methodologies).

While deep down I would love for a collaborative fully open source assessment platform to be developed, realistically that is not going to happen. It has been tried unsuccessfully too many times to ignore. This open assessment platform might actually be the next best thing, and I will be looking to maximise this opportunity over the next year in which we are piloting the software.

Tweet This Post

The report  “The Transition to Computer-Based Assessment – New Approaches to Skills Assessment and Implications for Large-scale Testing” has been published. The volume, which is based on a set of workshops that was held in Iceland in September 2008, was edited Julius Björnsson and Fritz Scheuerman. I think it gives a very broad and comprehensive overview of the current state of, and issues around, computer based assessment.

I would especially recommend The article on “CAT as a pedagogic tool” by Jakob Wandall, and “Issues in Computerized Ability Measurement: Getting out of the Jingle and Jangle Jungle” by Oliver Wilhelm.

Tweet This Post

One of the topics that came up several times over the past days in Reykjavik, is that of the differences in culture around assessment. Different countries have different ways in which they perceive and deal with assessment, and this can have a significant impact on the effect of the assessments, and the success of the educational system as a whole.

One particularly interesting approach was outlined by Jakob Wandall, who’s work in the Danish national tests I have blogged about last year in High stake national assessments and ranking. I tried to capture Jakob’s slide on a picture, but unfortunately that failed rather miserably, so I have tried to recreate his message in the graphic below:

The graph outlines how both the focus of the assessment (on the horizontal axis) and the purpose for which the results are primarily used (on the vertical axis) vary from country to country. I thought the visualisation was very interesting. Comparing this to, for instance, the outcomes of the 2006 PISA, it is interesting to note that neither the approach of the Scandinavian schools (who focus primarily on learner focused formative assessment) nor the Anglo-Saxon approach 9that is much more heavy on the measurements of indicators for performance, tied in to funding) really yields the best results.

The starts of PISA are of course the Finnish, and the unique approach is apparent from this graph. in stead of moving somewhere between the top left and the bottom right of the graph, they sit toward the top right. The Finish system highly values national measurements, evaluating the success of the system by objective measurements. However these measurements are not tied to any control, either through formal channels or more informal ones such as public rankings. In stead the measurements made in the Finnish system have the purpose to inform teaching and learning. An evidence based approach to teaching shall we say.

When I translate this to our own practice, I can’t help but relate this to demands to increase the amounts of formative assessment in our teaching. And while I am sympathetic to these demands, these assessments are similar to those in the top left of the above graph, informing and supporting individual learning processes. And so perhaps in stead of focusing primarily on formative individual assessment, we should focus (also) on assessment and evaluation that informs teaching. Building an infrastructure through which lecturers can stay in touch with the progress, successes and difficulties of all their students, and modify their teaching based on this understanding continuously.

Tweet This Post

I just arrived in Reykjavik for a conference on PISA 2006 and the transition to e-assessment. It’s my first time in Iceland, and I must say it was a bit surreal. I’m just reading Green Mars by Kim Stanley Robinson, which contains a lot of descriptions of a newly terraformed Mars: cold, lots of rocks and lots of lichen. Trust me, walking around in Iceland came scarily close to how I had been imagining the novel in my head up till then.

I’m hoping to find some time over the next days to post my thoughts on the conference. There is a very impressive lineup of international experts on assessment scheduled to speak, and I am looking forward to exchanging ideas and opinions with them. Pictures will have to wait I’m afraid, as I forgot to pack the cable that connects my camera to my laptop…grrr…

Tweet This Post

This week I have had the pleasure of attending ALT-C in Leeds. We had an awesome opening by Hans Rosling, but unfortunately I cannot find the recording for that. For those of you who have not heard of Hans, I thoroughly recommend looking at his TEDtalk, and the Gapminder website.

While several papers and presentations were the ussual rehashes and repeats of previous years, there were also some very interesting nuggets. One was from the University of Vienna, who have been looking at the development of an IMS LD design tool for lecturers within the EU funded project Prolix. While I couldn’t easily find much documentation on the tool they developed, you can download the source for GLM (based on Eclipse) from sourceforge. I’ll certainly be having a look at it over the next few weeks.

Another highlight was the talk by George Siemens. If you are interested you can still look at the recordings for the session in Eluminate (you’ll have to download their Java applet for it though). it was a shame I only found out George was staying in the same hotel as I was when I was checking out, and too hung over to try and engage in some conversation.

My third pick would be the SLAM session on the Digital Divide, where we all created small clips on the digital divide in little groups which was a lot of fun. The recordings should all be up on the wiki, although the last time I checked ours still hadn’t made it there However, I did find this picture by Christina Costa of the group I worked with.

Tweet This Post