Rene's Assessment

Finally an extensible assessment platform for HE?

René — Tue, 22 Sep 2009 16:35:53 +0000

I’ve been at the Questionmark European User Conference 2009 in Manchester the past 2 days. I must say that I have never been a great fan of Questionmark, and the previous time I looked at the product (in 2006) I found it a terribly unstructured and cluttered beast. Then again the big problem with computer based assessment technology, is that there is very little of it that is any good. Some of it is terribly technical, and requires high amounts of software development or psychometric understanding (or both) to be useful. The vast majority of it is very feature poor, in particularly when it comes to item-types. And then there is the enormous graveyard of failed Open Source projects funded by JISC and others that never made it to a stage of maturity, and adoption by anyone beyond the developers.

And so when looking at a replacement for the in-house system that we have developed over the past decade in Derby, choice was limited. After a long and painful stage of denial, we ended up having to admit that, short of a major development effort, Questionmark was probably one of the few viable alternatives. And although some of it’s inherent weaknesses remain, there are also a number of interesting developments that have actually sparked my sincere interest.

The most important of those is the Open Assessment platform. Like other vendors (such as Blackboard) Questionmark seems to have understood the importance of the Open Source movement. Every vendor deals differently with this phenomenon, but Questionmark seem serious about embracing it constructively. They are working on opening up their API’s, but also Open Sourcing community editions of integration software, such as connectors to Moodle, Blackboard and Sharepoint. This I think is a great start in opening up the product, and creating added value through collaboration with, what will hopefully be, a vibrant community of developers.

What I really hope for though, is that this openness will extend to the data, and the application framework itself. There are thousands of specialised use-cases, in particular question types, that are highly desired by higher education but that will never be a viable commercial priority for a vendor like Questionmark. However if a University could extend the Questionmark platform and create these question types, and better yet if several universities would do this and share their efforts, the value of the Questionmark platform would increase dramatically. What we need for this to happen is an application architecture for both the authoring environment and the assessment rendering engine that would support extentions, plug-ins, just like Firefox and so many other modern applications do. That way we could create our own question templates that could be authored and delivered from within Questionmark. (This short of questions being true objects that can expose themselves in authoring or delivery mode, but I will spare you that highly technical argument against the current question definition methodologies).

While deep down I would love for a collaborative fully open source assessment platform to be developed, realistically that is not going to happen. It has been tried unsuccessfully too many times to ignore. This open assessment platform might actually be the next best thing, and I will be looking to maximise this opportunity over the next year in which we are piloting the software.

Tweet This Post

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Philosophy of success

René — Sun, 02 Aug 2009 09:58:15 +0000

Bless TED for introducing me to so many fascinating people and ideas. This morning I watched the talk by Alain de Botton: “A kinder, gentler philosophy of success”. It was an eye opening perspective on our society. I never really appreciated that when you create the illusion of a meritocracy you are implying that, in the same way that people who deserve to rise to the top, will rise to the top, people who are down at the bottom are also there because they deserve it. They are losers, not unfortunates. And given the enormous influence of randomness and chance in every life, that is an unfair judgement. It is a judgement that creates not only ‘low self-esteem’, but also the fact that suicide rates are higher in modern individualistic societies, then anywhere else in the world.

I think I might have to add some of Alain’s books to my ‘to-read’-pile.

Tweet This Post

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Experiential learning in a virtual environment

René — Wed, 29 Jul 2009 10:58:06 +0000

In the past academic year we have been looking into using virtual environments for learning. Initial applications during the initial Second Life hype had left a lot of us uninspired. What afterall is the value of a virtual lecture theatre in which we can virtually raise our hands? And unfortunately many of the initial uses of Second Life were of that nature.

Our partnership with the Institutes of Quarrying and Asphalt technology yielded an opportunity to explore a much more interesting use of Second Life.

Through the creation of a Virtual Quarry, a safe and accessible learning environment was created that allows learners to learn by experience in way that would never be possible (practical, responsible) in real life. The quarry currently hosts health and safety exercises, and teaches the correct execution of blasting. In the year to come we are hoping to extend these scenarios both within the quarry, but also in other areas such as forensic sciences, environmental health and geology.

There are more interesting projects being carried out. In the UK, Daden Limited was one of the frontrunners, working with several partners in the creation of simulated scenario’s that supported problem based learning. The psychology department of the University of Derby was involved in one of these projects, and is still very active in this area. I’m not yet sure if this type of learning will find wide application as there are many practical barriers to overcome, cost not being the least of these. Nevertheless these are valuable and interesting experiments, and I’m looking forward to researching their aplication further.

Tweet This Post

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

An end to the monopoly on education

René — Sun, 19 Jul 2009 17:12:35 +0000

Today George Osborne, the current shadow chancellor and most likely the UK’s chancellor within 9 months, casually announced that which I have been dreading for a while now: The end of the states responsibility for education. He didn’t quite put it in these words of course, but in popular right wing phrases involving consumer choice and ends to state monopolies. You can find the full transcript of the interview here, but I will quote the relevant section:

“In education, you know we are looking at bringing in new providers – private companies or voluntary groups or charities – that can offer state education paid for by the taxpayer, but offer a choice to parents and break up the state monopoly on the state provision of education.”

It is disgraceful that as a result of bailing out private interests in the banking sector, we will now be taking a step back from one of the most fundamental human needs and rights: Education. it might be that competition will spur the education system into some needed changes that it might otherwise not easily implement. But I think it more likely that a competition with a private education sector that is less restrained by government bureaucracy and funding limitations will not have a fair chance in this arena. And once education has become a for-profit business, will we have any chance of ensuring that every child and adult has access to an equal support for the deployment of their talents, both for their own benefit as for society as a whole?

I doubt it.

Tweet This Post

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

The decline of Higher Education

René — Fri, 15 May 2009 14:50:19 +0000

It is clear, and I think a matter of broad consensus that our succes and prosperity, and that of our society, requires increasing amounts of learning, reflection and adaptation. It is this assertion that makes widening participation, and lifelong learning so important. In particular in these difficult times, the investments we make now in our skills and competences will determine how strong we will arise from the ashes of this recession.

It is therefore rather shameful to see how the higher education sector is being driven to ruin. While government is advocating that we widen articipation, at the same time it cuts back funding for teaching by 65 million pounds in the next academic year. Earlier it had already stopt funding the education of those of us who already have a degree (Equivalent or lower level Qualifications). It is strange isn’t it, that on the one side we get told how important it is that we all keep learning, as the halftime of knowledge is decreasing so rapidly. Yet our funding system now seems to say that once you have learned something, no further investment or maintenance of that learning will ever be required. We spend billions of pounds on rescuing a banking system that few of us actually really want, but we cannot commit to even a fraction of that funding to guarantee us the learning and development we all so desperately want and need. Lifelong learning is on the verge of extinction, the Times Higher Education reports, but very few of us seem to really care. Or have we just become too numb over the past months to realise what is happening?

Our regulations and practices, both in education and professional bodies increasingly breed compliance in stead of creativity. Are we selling off Higher Education to corporations, and to those with the personal wealth to fund their own development? A system that should be about the development of critical individuals, which I believe to be in the long term benefit of society as a whole, is risking slowly transforming into a goverment sponsored corporate training ground. I believe that is the very last thing that we need.

Tweet This Post

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Is feedback really all it’s cracked up to be?

René — Fri, 08 May 2009 12:37:34 +0000

I attended a guest lecture by Jesse Martin on how Psychology can contribute to the scholarship of teaching. It was an interesting talk, but the thing that really caught my attention concerned feedback.

Jesse was illustrating how we often fail to apply well understood psychological principles and research to our practice by looking at the current hype around feedback. According to existing cognitive research, extensive feedback actually breeds compliance, not learning. While targeted feedback at key points in someones development is important, continuous detailed feedback actually inhibits independent critical thought.While a bit of a shock at first, I must admit that this does make sense on an intuitive level. So then what do we do?

Well, there are some usefull suggestions from existing psychological research here as well. One observed phenomenon that we might exploit is the testing effect. This effect describes how neural connections are strengthened by retrieval. practice, such as through formative assessment, can therefor enhance the retention of what has been learned. it has also been shown to improve the retention over time.

So formative assessment is still seen as crucial. But it might be that the activity is much more important then the feedback resulting from it.

Tweet This Post

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

The lecture is dead, long live the lecture!

René — Thu, 26 Mar 2009 22:38:03 +0000

It seems that, after static content, the lecture has now definitely made it into the realm of ‘what you share for free’ in stead of what you ‘sell for money’. We’ve had Teachertube for a while now, but this always struck me as being a tutor-to-tutor resource:

Bu in recent weeks we’ve had a launch of two really great resources that provide world class lectures directly to learners. A few weeks ago we saw the launch of Academic Earth which is aspiring to bring us ‘Thousands of video lectures from the world’s top scholars’ such as this lecture from Paul Bloom:

It seems from the embed-tag that Academic earth uses blip.tv, which is interesting. But what I really like about Academic Earth is that is often provides full transcripts and reading assignments, which makes these videos into a truly accessible and valuable independent learning resource.

And today I stumble on Youtube’s attempt to create an online iTunesU: Youtube EDU, which does not seem to have the high production value of Academic Earth, but will most likely develop very impressively in terms of sheer available content, such as this series on special relativity:

Other then being very grateful for these wonderful resources, I must also say that I am intrigued by the speed which with the traditional University is being dismantled. That information, content, and perhaps even knowledge have been commodotized is not really that surprising, but that, only a few years after that development, are already declaring the lecture dead in terms of marketable value, that does surprise me. Don’t get me wrong, it’s not that I disagree, I am just baffled by the speed of developments. Either way, I’m going to wrap up this post, as I have some lectures to watch.

The lecture is dead: Long live the lecture!

Tweet This Post

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

The Transition to Computer-Based Assessment

René — Mon, 16 Mar 2009 15:25:17 +0000

The report “The Transition to Computer-Based Assessment – New Approaches to Skills Assessment and Implications for Large-scale Testing” has been published. The volume, which is based on a set of workshops that was held in Iceland in September 2008, was edited Julius Björnsson and Fritz Scheuerman. I think it gives a very broad and comprehensive overview of the current state of, and issues around, computer based assessment.

I would especially recommend The article on “CAT as a pedagogic tool” by Jakob Wandall, and “Issues in Computerized Ability Measurement: Getting out of the Jingle and Jangle Jungle” by Oliver Wilhelm.

Tweet This Post

Finally an extensible assessment platform for HE? I’ve been at the Questionmark European User Conference 2009 in...

Related posts brought to you by Yet Another Related Posts Plugin.

Learning to fail

René — Tue, 10 Mar 2009 22:07:35 +0000

I must first of all thank Stephen Downes for his truly excellent post on the Monkeysphere Ideology. It is a profound and insightful analysis of some unhealthy fundamentals in society, of which this banking crisis is perhaps just a single symptom. This post is about how the education sector perhaps is suffering the same symptoms, and so might actually be headed for a similar meltdown.

Fail

It is argued that the financial crisis is one of destructive incentives, and hyperinflation of quality ratings, that lead to a massive over-investment in what turned out to be worthless assets. A truly horrific oversimplification, but it will do for the purposes of this post. Higher Education also rates it’s products, and ironically it is a credit rating system. Attainment in the UK is measured in academic credit on levels 4 (first year undergraduate) to 8 (doctorate). Credit on these levels is highly valued, and often seen as key to successful participation in the modern information society. As a result there is a dramatic effort to widen participation in an attempt to get up to 40% of the population equiped with a level 4 qualification or higher. And in a similar way to how mortgages were handed out without ensuring clients had sufficient capital and earnings for repayment, we are now handing out academic credit without ensuring that when we hand it out, it represents adequate intellectual capital.

There is no need for this, as my case certainly doesn’t revolve around some elitist notion that the majority of the population is fundamentally incapable of learning and functioning on this level. The problem is that we are being given the wrong incentives. We are incentivized to pass, preferably with a first. Our funding, and our ranking in league-tables depends on it. And slowly but surely, this erosion is also eroding the value of Higher Education. And unfortunately, it erodes the value of all degrees, just as the credit crisis is devaluing all our houses, regardless of our credit-worthiness. And with the value of qualifications diminishing, and their cost increasing, inevitably the time will come that this eroded qualification is no longer worth the inflated tuition fee that is being asked for it.

The key for recovery to me is failure. We should be allowed to fail, learn from that failure, and let a phoenix arise from it’s ashes. We seem to have missed the point that success is a value that is relative, relative to failure. We forget that the greatest of successes have often come from strings of failures. Who knows what great innovative company might arise from the ashes of a bankrupt General Motors, but we will probably never know. Just like we will never know how great some of our students might have been after overcoming criticism, setbacks and failure, because in stead we have sent them away with only a marginal and ever decreasing successes. Failure is the foundation of greatness, and we need to learn to embrace it.

Tweet This Post

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

UK Government to promote Open Source

René — Thu, 26 Feb 2009 08:15:06 +0000

Open Source

I’m extremely pleased with the news that the UK government has decided to get serious about backing open source software. While some of this may be inspired by a perceived cost saving, it is also clearly inspired by the increasing understanding of the value of openness and open standards. A very important part of this new policy is that it seems to finally recognise and address the need to look at procedures for tendering and procurement that do not disadvantage open source software unfairly. A more detailed discussion about this, if you are interested, can be found on the OSSWatch blog.

Tweet This Post

Finally an extensible assessment platform for HE? I’ve been at the Questionmark European User Conference 2009 in...

Related posts brought to you by Yet Another Related Posts Plugin.