Rene’s Assessment

I’ve been at the Questionmark European User Conference 2009 in Manchester the past 2 days. I must say that I have never been a great fan of Questionmark, and the previous time I looked at the product (in 2006) I found it a terribly unstructured and cluttered beast. Then again the big problem with computer based assessment technology, is that there is very little of it that is any good. Some of it is terribly technical, and requires high amounts of software development or psychometric understanding (or both) to be useful. The vast majority of it is very feature poor, in particularly when it comes to item-types. And then there is the enormous graveyard of failed Open Source projects funded by JISC and others that never made it to a stage of maturity, and adoption by anyone beyond the developers.

And so when looking at a replacement for the in-house system that we have developed over the past decade in Derby, choice was limited. After a long and painful stage of denial, we ended up having to admit that, short of a major development effort, Questionmark was probably one of the few viable alternatives. And although some of it’s inherent weaknesses remain, there are also a number of interesting developments that have actually sparked my sincere interest.

The most important of those is the Open Assessment platform. Like other vendors (such as Blackboard) Questionmark seems to have understood the importance of the Open Source movement. Every vendor deals differently with this phenomenon, but Questionmark seem serious about embracing it constructively. They are working on opening up their API’s, but also Open Sourcing community editions of integration software, such as connectors to Moodle, Blackboard and Sharepoint. This I think is a great start in opening up the product, and creating added value through collaboration with, what will hopefully be, a vibrant community of developers.

What I really hope for though, is that this openness will extend to the data, and the application framework itself. There are thousands of specialised use-cases, in particular question types, that are highly desired by higher education but that will never be a viable commercial priority for a vendor like Questionmark. However if a University could extend the Questionmark platform and create these question types, and better yet if several universities would do this and share their efforts, the value of the Questionmark platform would increase dramatically. What we need for this to happen is an application architecture for both the authoring environment and the assessment rendering engine that would support extentions, plug-ins, just like Firefox and so many other modern applications do. That way we could create our own question templates that could be authored and delivered from within Questionmark. (This short of questions being true objects that can expose themselves in authoring or delivery mode, but I will spare you that highly technical argument against the current question definition methodologies).

While deep down I would love for a collaborative fully open source assessment platform to be developed, realistically that is not going to happen. It has been tried unsuccessfully too many times to ignore. This open assessment platform might actually be the next best thing, and I will be looking to maximise this opportunity over the next year in which we are piloting the software.

Tweet This Post

Open Source

I’m extremely pleased with the news that the UK government has decided to get serious about backing open source software. While some of this may be inspired by a perceived cost saving, it is also clearly inspired by the increasing understanding of the value of openness and open standards. A very important part of this new policy is that it seems to finally recognise and address the need to look at procedures for tendering and procurement that do not disadvantage open source software unfairly. A more detailed discussion about this, if you are interested, can be found on the OSSWatch blog.

Tweet This Post

I’ve been looking into criteria for assessment technologies a lot lately. One reason is that we are looking into migrating our current system to a new platform (as the old one, Authorware, is no longer supported). The other reason is that I have been invited by the Joint Research Centre to take part in a workshop on quality criteria for computer based assessments. I will be posting on the outcomes of that workshop next week. For now though, here are some of my thoughts on the topic.

Flexibility
The main strength of our current system is flexibility. This has several aspects, that are all important in their own right:

• Flexibility in design: The layout of the question can be modified as desired, using media and such to create an authentic and relevant presentation
• Flexible interactions: There is no point in systems that have parameterised 5 question types for you, and all you can do is define a title, question text, alternatives and select the right answer. Interactions testing and supporting higher order skills are, or should be, more complex then that.
• Detailed and partial scoring: A discriminating question does not just tell you whether you were completely right, or completely wrong. It can tell you the degree to which you were right, and what elements of your answer had any value. It might also penalise you for serious and fundamental mistakes.
• Detailed feedback: A lot of mistakes learners make are predictable. If we allow assessment systems to capture these mistakes and give targeted feedback, learners can practice their skills while lecturers can focus there time on more in depth problems that require their personal engagement.
• Extensive question generation and randomisation options: For the re-usability of assessments, generating questions using rules and algorithms given a single question almost infinite re usability. On the assessment level, the same is true for assessment generation based on large banks with questions tagged with subject matter and difficulty.

So far, no real news for TRIADS users (although no proprietary system I know of really supports this well).

Questions without assessments
As Dylan Wiliam so eloquently worded at the ALT-C conference (you can find his podcast on the matter on http://www.dylanwiliam.net/), the main value in learning technology lies in “to allow teachers to make real-time instructional decisions, thus increasing student engagement in learning, and the responsiveness of instruction to student needs.” I could not agree more. However, this means that questions should not just exist within the assessment, but instead be embedded within the materials and activities. Questions become widgets that can of course still function within an assessment, but also work on their own without loosing the ability to record and respond to interaction. This, as far as I’m aware, is unchartered territory for assessment systems. Territory that we hope to explore in the next iteration of our assessment engine.

Tweet This Post

It’s not about the technology. We often say it, but we rarely really mean it I think. Let’s be fair now: Technology is kinda fun. I know I get carried away far to easily with new shiny things. But sometimes I forget that not everyone is paid to play with new shiny things. Some of us actually have to teach students on a regular basis. So for those colleagues, here’s some lovely simple tools that I think can give you a lot of bang for just a little investment.

Course Genie
Course Genie is a lovely tool. It allows you to create nice looking materials and quizzes without any skills other then MS Word. It’s a great tool if you want to develop something slightly more interactive then just uploading a module specification or PowerPoint to you VLE, and it integrates nicely with Moodle, WebCT and Blackboard. The quiz options are relatively simple, but actually have some powerful capabilities such as rich feedback based on answers given. The downside is that Course Genie does not save answers, scores, or track progress

QUIA
God knows who thought up the name for this one. In all honesty, QUIA would not be a tool that I’d use myself. it does a lot of things (games, quizes, surveys), but it does none of them very well. Nevertheless if you want a cheap tool that instantly will allow you to create a lot of simple interactive resources this might be the tool for you. The absolute plus to QUIA is that you instantly tap in to the whole QUIA community, which allows you to share and rate all developed materials. QUIA also has a very decent result tracking option, which even allows you to expert a detailed result analysis to Excel.

Electronic Voting Systems
A great way to make teaching, especially in larger groups, more interactive. It’s a great way to find out what students think, understand or want during your lesson, so that you can adapt to their needs in the appropriate manner. We’ve been using Turning Point for a short while now, and so far I am quite impressed. It’s extremely easy to set up a simple poll supported by PowerPoint. it is however also possible to much more complex thing, such as linking in responses to demographics.

Tweet This Post