Rene’s Assessment

I’ve been at the Questionmark European User Conference 2009 in Manchester the past 2 days. I must say that I have never been a great fan of Questionmark, and the previous time I looked at the product (in 2006) I found it a terribly unstructured and cluttered beast. Then again the big problem with computer based assessment technology, is that there is very little of it that is any good. Some of it is terribly technical, and requires high amounts of software development or psychometric understanding (or both) to be useful. The vast majority of it is very feature poor, in particularly when it comes to item-types. And then there is the enormous graveyard of failed Open Source projects funded by JISC and others that never made it to a stage of maturity, and adoption by anyone beyond the developers.

And so when looking at a replacement for the in-house system that we have developed over the past decade in Derby, choice was limited. After a long and painful stage of denial, we ended up having to admit that, short of a major development effort, Questionmark was probably one of the few viable alternatives. And although some of it’s inherent weaknesses remain, there are also a number of interesting developments that have actually sparked my sincere interest.

The most important of those is the Open Assessment platform. Like other vendors (such as Blackboard) Questionmark seems to have understood the importance of the Open Source movement. Every vendor deals differently with this phenomenon, but Questionmark seem serious about embracing it constructively. They are working on opening up their API’s, but also Open Sourcing community editions of integration software, such as connectors to Moodle, Blackboard and Sharepoint. This I think is a great start in opening up the product, and creating added value through collaboration with, what will hopefully be, a vibrant community of developers.

What I really hope for though, is that this openness will extend to the data, and the application framework itself. There are thousands of specialised use-cases, in particular question types, that are highly desired by higher education but that will never be a viable commercial priority for a vendor like Questionmark. However if a University could extend the Questionmark platform and create these question types, and better yet if several universities would do this and share their efforts, the value of the Questionmark platform would increase dramatically. What we need for this to happen is an application architecture for both the authoring environment and the assessment rendering engine that would support extentions, plug-ins, just like Firefox and so many other modern applications do. That way we could create our own question templates that could be authored and delivered from within Questionmark. (This short of questions being true objects that can expose themselves in authoring or delivery mode, but I will spare you that highly technical argument against the current question definition methodologies).

While deep down I would love for a collaborative fully open source assessment platform to be developed, realistically that is not going to happen. It has been tried unsuccessfully too many times to ignore. This open assessment platform might actually be the next best thing, and I will be looking to maximise this opportunity over the next year in which we are piloting the software.

Tweet This Post

In the past academic year we have been looking into using virtual environments for learning. Initial applications during the initial Second Life hype had left a lot of us uninspired. What afterall is the value of a virtual lecture theatre in which we can virtually raise our hands? And unfortunately many of the initial uses of Second Life were of that nature.

Our partnership with the Institutes of Quarrying and Asphalt technology yielded an opportunity to explore a much more interesting use of Second Life.

Through the creation of a Virtual Quarry, a safe and accessible learning environment was created that allows learners to learn by experience in way that would never be possible (practical, responsible) in real life. The quarry currently hosts health and safety exercises, and teaches the correct execution of blasting. In the year to come we are hoping to extend these scenarios both within the quarry, but also in other areas such as forensic sciences, environmental health and geology.

There are more interesting projects being carried out. In the UK, Daden Limited was one of the frontrunners, working with several partners in the creation of simulated scenario’s that supported problem based learning. The psychology department of the University of Derby was involved in one of these projects, and is still very active in this area. I’m not yet sure if this type of learning will find wide application as there are many practical barriers to overcome, cost not being the least of these. Nevertheless these are valuable and interesting experiments, and I’m looking forward to researching their aplication further.

Tweet This Post

The report  “The Transition to Computer-Based Assessment – New Approaches to Skills Assessment and Implications for Large-scale Testing” has been published. The volume, which is based on a set of workshops that was held in Iceland in September 2008, was edited Julius Björnsson and Fritz Scheuerman. I think it gives a very broad and comprehensive overview of the current state of, and issues around, computer based assessment.

I would especially recommend The article on “CAT as a pedagogic tool” by Jakob Wandall, and “Issues in Computerized Ability Measurement: Getting out of the Jingle and Jangle Jungle” by Oliver Wilhelm.

Tweet This Post

One of the frequent criticisms on e-Assessment is the perceived limits in item types that can be supported by technology. While there are long debates to be had about assessing higher order skills with constrained response item types, I don’t think these debates are going to take away the prime concern: Free text items.

I must say that I have serious doubts about marking free text by computer. I don’t know enough about the principles involved to say this with any sort of authority, but I am aware of the kind of heuristics used in automated essay marking for instance. These heuristics are often grammatical and rhetorical in nature, and have fairly little to do with the subject matter (although it must be said that many human markers have been shown to use similar heuristics). Nevertheless, interesting progress is being made in this area, and eventually I am sure that language processing will be commonplace.

One of the interesting project that I recently became aware of, is the OpenComment project, which is lead by Denise Whitelock at the Open University. The project is looking to use latent semantic analysis to analyse learners responses to open ended questions in history and philosophy. Another interesting fact is that the project is developing this as a question type in Moodle, and so it should be relatively easy for everyone to reap the benefits of this technology within their own learning environments.

Automated marking is by no means the only value of using technology in assessment. The OpenMentor project, again from the Open University, is a great example. OpenMentor compares a mark assigned to a piece of work to the amounts of positive and negative feedback given, and checks this for consistency. In this way it can help in the coaching process of new teachers. Given the importance of feedback, I think it’s wonderful to have explicit standards and training in giving it.

The ABC (Assess By Computer) software has so far escaped my radar. I wasn’t aware of it until queried by the Times Higher Education for the article they were doing. The software has a support role similar to OpenMentor, but this time the support is provided around the marking process. The software can highlight keywords, compare answers to model answers and more. All of this for the sole purpose of making it easier on the human marker, but also improve consistency between human markers. Especially the latter is very welcome I think, as marking open ended questions and assignments can sometimes be somewhat of a dark art.

I only just discovered that bits of the e-mail I sent to the reporter actually appear in the article. If I would have known that I probably would have paid a bit more attention to my grammar :S

Tweet This Post

I thought I’d have a go at answering the The Learning Circuits Blog: December Big Question – What did you learn about learning? One of the projects I have worked on this year, is the development of a tool supporting the Accreditation of Prior Learning (APL). It has been truly enlightening for me in many ways.

APL is going to be a core activity of a lot of Universities I reckon. Content no longer seems to be the core business of the sector, as has been shown by initiatives such as Open Learn. Coming to grips with this is a bit like trying to understand Open Source business models I think, it requires a fundamental rethink of what is valuable. For most universities I think that value is going to increasingly lie in guidance and coaching on the one side, and assessment and accreditation on the other.

There seems to be a problem with accreditation of learning that has not taken place within the controlled environment of a course though. Very few universities are serious about APL, and I can’t help but wonder why. Part of it, I am sure, is to do with fees and such, but not all. After some reflection I think we must also admit that APL exposes some critical weaknesses of our assessment processes. In theory our assessments are supposed to discriminate between those learners that have attained certain outcomes, and those who haven’t. If that was all there was to it, then surely learners claiming APL could as simple as doing the regular assessment, but without attending the course.

The reason this isn’t common practice I think, is that most assessments don’t really assess the right outcomes. Most assessments I think are designed to trigger an echo of teaching, and not of learning. And of course our teaching is so good, that if the learner echo’s a confirmation of our teaching, then surely that means the intended learning has taken place. But what if learning has not been a result of our teaching? Suddenly we cannot short circuit the inherent difficulty of assessing competence by resorting to looking for the echo of teaching.

I think it would be interesting to dig into assessment practices used by recruitment agencies. In a way they are asked to make assessments that employers aren’t confident we have made. Furthermore, whatever they assess is always without the luxury safety net of knowing what has probably been learned and by which means.

Tweet This Post