Rene’s Assessment

I’ve been at the Questionmark European User Conference 2009 in Manchester the past 2 days. I must say that I have never been a great fan of Questionmark, and the previous time I looked at the product (in 2006) I found it a terribly unstructured and cluttered beast. Then again the big problem with computer based assessment technology, is that there is very little of it that is any good. Some of it is terribly technical, and requires high amounts of software development or psychometric understanding (or both) to be useful. The vast majority of it is very feature poor, in particularly when it comes to item-types. And then there is the enormous graveyard of failed Open Source projects funded by JISC and others that never made it to a stage of maturity, and adoption by anyone beyond the developers.

And so when looking at a replacement for the in-house system that we have developed over the past decade in Derby, choice was limited. After a long and painful stage of denial, we ended up having to admit that, short of a major development effort, Questionmark was probably one of the few viable alternatives. And although some of it’s inherent weaknesses remain, there are also a number of interesting developments that have actually sparked my sincere interest.

The most important of those is the Open Assessment platform. Like other vendors (such as Blackboard) Questionmark seems to have understood the importance of the Open Source movement. Every vendor deals differently with this phenomenon, but Questionmark seem serious about embracing it constructively. They are working on opening up their API’s, but also Open Sourcing community editions of integration software, such as connectors to Moodle, Blackboard and Sharepoint. This I think is a great start in opening up the product, and creating added value through collaboration with, what will hopefully be, a vibrant community of developers.

What I really hope for though, is that this openness will extend to the data, and the application framework itself. There are thousands of specialised use-cases, in particular question types, that are highly desired by higher education but that will never be a viable commercial priority for a vendor like Questionmark. However if a University could extend the Questionmark platform and create these question types, and better yet if several universities would do this and share their efforts, the value of the Questionmark platform would increase dramatically. What we need for this to happen is an application architecture for both the authoring environment and the assessment rendering engine that would support extentions, plug-ins, just like Firefox and so many other modern applications do. That way we could create our own question templates that could be authored and delivered from within Questionmark. (This short of questions being true objects that can expose themselves in authoring or delivery mode, but I will spare you that highly technical argument against the current question definition methodologies).

While deep down I would love for a collaborative fully open source assessment platform to be developed, realistically that is not going to happen. It has been tried unsuccessfully too many times to ignore. This open assessment platform might actually be the next best thing, and I will be looking to maximise this opportunity over the next year in which we are piloting the software.

Tweet This Post

Open Source

I’m extremely pleased with the news that the UK government has decided to get serious about backing open source software. While some of this may be inspired by a perceived cost saving, it is also clearly inspired by the increasing understanding of the value of openness and open standards. A very important part of this new policy is that it seems to finally recognise and address the need to look at procedures for tendering and procurement that do not disadvantage open source software unfairly. A more detailed discussion about this, if you are interested, can be found on the OSSWatch blog.

Tweet This Post

I finally found some time to read the Cape Town OER Declaration, and a selection from the deluge of comments that have piled up in my RSS-reader the past weeks. Given the critical tone of most of these, I was expecting something very fundamentally flawed.

The declaration is an initiative of the Shuttleworth Foundation (yes, that’s the same Shuttleworth as the one in Ubuntu). The purpose of the declaration is to accelerate the international effort to promote open resources, technology and teaching practices in education. Unfortunately many advocates of open learning have not really welcomed the declaration with open arms.

A noteworthy example of this can be found in the blog Half an Hour: Criticizing the Cape Town Declaration by Stephen Downes. While I normally find Stephens post very eloquent, I cannot support many of the arguments he makes. It leaves me with the impression that his main point (and that of many others) is that they are a bit miffed of they weren’t consulted. To me the whole ‘let’s decide everything in a big all encompassing committee’ culture is exactly the reason that hardly anything ever gets done, or done properly in education. Open source communities understand that democracies don’t work. A benevolent dictator, or a meritocracy (or both) is what you need. I’m sure Mark Shuttleworth understood exactly that when he limited participation in drafting this initial declaration.

I for one support the initiative. I’m going to sign up for it now, and I would invite you to consider the same.

… Which reminds me, I still need to formally license the stuff on here with a creative commons license…

Tweet This Post

I’ve been looking into criteria for assessment technologies a lot lately. One reason is that we are looking into migrating our current system to a new platform (as the old one, Authorware, is no longer supported). The other reason is that I have been invited by the Joint Research Centre to take part in a workshop on quality criteria for computer based assessments. I will be posting on the outcomes of that workshop next week. For now though, here are some of my thoughts on the topic.

Flexibility
The main strength of our current system is flexibility. This has several aspects, that are all important in their own right:

• Flexibility in design: The layout of the question can be modified as desired, using media and such to create an authentic and relevant presentation
• Flexible interactions: There is no point in systems that have parameterised 5 question types for you, and all you can do is define a title, question text, alternatives and select the right answer. Interactions testing and supporting higher order skills are, or should be, more complex then that.
• Detailed and partial scoring: A discriminating question does not just tell you whether you were completely right, or completely wrong. It can tell you the degree to which you were right, and what elements of your answer had any value. It might also penalise you for serious and fundamental mistakes.
• Detailed feedback: A lot of mistakes learners make are predictable. If we allow assessment systems to capture these mistakes and give targeted feedback, learners can practice their skills while lecturers can focus there time on more in depth problems that require their personal engagement.
• Extensive question generation and randomisation options: For the re-usability of assessments, generating questions using rules and algorithms given a single question almost infinite re usability. On the assessment level, the same is true for assessment generation based on large banks with questions tagged with subject matter and difficulty.

So far, no real news for TRIADS users (although no proprietary system I know of really supports this well).

Questions without assessments
As Dylan Wiliam so eloquently worded at the ALT-C conference (you can find his podcast on the matter on http://www.dylanwiliam.net/), the main value in learning technology lies in “to allow teachers to make real-time instructional decisions, thus increasing student engagement in learning, and the responsiveness of instruction to student needs.” I could not agree more. However, this means that questions should not just exist within the assessment, but instead be embedded within the materials and activities. Questions become widgets that can of course still function within an assessment, but also work on their own without loosing the ability to record and respond to interaction. This, as far as I’m aware, is unchartered territory for assessment systems. Territory that we hope to explore in the next iteration of our assessment engine.

Tweet This Post