Online peer assessment is a big deal. Some thoughts on how it can be done well.
A couple of days ago I noticed #beyondthetextbook emerging on Twitter. It turns out that this hashtag related to an gathering sponsored by Discovery Education in Washington D.C.
My (remote, somewhat helicopter-like) contribution, was pretty much summed up by the following:
After reading Audrey Watters’ post about the gathering (as well as those by others), I’d like to expand up on that and highlight some thoughts from others with whom I’m in agreement.
I want us to weigh classroom practices, power, authority, politics, publishing, assessment, expertise, attribution, and the culture(s) of the education system. I would argue that the textbook in its current form — and frankly in almost all of the digital versions we’re also starting to see now — is tightly woven into that very fabric, and once we tug hard enough at the “textbook” thread, things come undone.
The textbook is easy to talk about. It’s a physical thing that people have known as students and, for some, as educators. The trouble is that, just as with any technology, it’s difficult to separate the thing from the practices that surround the thing.
There’s nothing inherently wrong with textbooks – especially if you define them as Bud Hunt does as “A collection of information organized around thoughtful principles intended to provide support to instruction.” I’m not so keen on the word ‘instruction’ (I’d substitute ‘learning’) but like his basis in ‘thoughtful principles’.
Getting assessment right
One of the reasons I’m such a big fan of badges for lifelong learning is that assessment is broken. I don’t mean ‘broken’ in the sense that a bit of a repair job would fix. I mean structurally unsound and falling apart. Liable to collapse at any moment. That kind of broken.
It’s a problem I felt as a classroom teacher. It’s an issue I had to deal with as a senior manager. It’s evident in my sector-wide role in Higher Education. The hoops through which we’re asking people to jump not only don’t mean anything any more, but they don’t necessarily lead anywhere.
To me, that constitutes a crisis of relevance. So when we’ve got textbooks solely focused on providing content in bite-sized chunks in order to allow people to pass summative tests, then we’ve got a problem. A huge problem.
But let’s be clear: the problem is to do with the high-stakes assessment. It’s akin to the current attacks on the efficacy of teachers. The problem isn’t with (most) teachers, it’s with what you’re asking them to do. Likewise, with textbooks, it’s not the collecting of information in one place – it’s what people are expected to do with that information.
Open content and the blank page
I’ve seen many state their belief that the best kind of textbook is the blank page. By that, they mean that textbooks should be co-constructed. I certainly can’t argue with that, but we must always be careful that we don’t substitute one form of top-down structure with another.
Back in 2006 I wrote a couple of posts on my old teaching blog. One covered the idea of teachers as lifeguards, and other focused on the teacher as DJ. In the former I talk about the importance of teachers ‘knowing the waters’ so that they can allow students to explore the waters, growing in confidence (but be there when things go wrong). In the latter I discuss the similarities between teachers and DJs around ‘tempo’ and ‘playlists’.
Both the lifeguard and DJ analogies work with textbooks, I think. The difficulties are always going to be around time and competency. It’s all very well for those new to the profession, willing to burn the candle at both ends to remix the curriculum and create their own textbooks to move #beyondthetextbook. But that’s a recipe for burnout.
As usual, I’ve more questions than answers, but if I have one contribution to the #beyondthetextbook debate it’s that our current use of textbooks is a symptom of the problem, not the problem itself. It’s difficult to debate nuanced things online, and even more so via Twitter.
I think we need a renaissance in blogging – and the kind of blogging where we reference other people’s work. If we’re going to debate problems in education, let’s do so at length, with some nuance, and in a considered way.
Thanks for reading this far. I’d love to read any comments you have below!
At TEDx Warwick last Saturday I was first up, meaning I could sit back, relax and listen properly to the other speakers. Whilst I could write several posts about floating islands, medical implants and sustainability, I want to focus on just one of them.
John Kay, author of Obliquity: why our goals are best achieved indirectly, gave what I considered to be a fascinating talk. His book has been on my Amazon wishlist almost since it came out and I’ve now got several more reasons to read it.
Before I go any further, the stimulus for writing this post comes from the TES article ‘Wales asks schools to teach to the Pisa test‘. As I mentioned in my recent Purpos/ed Ignite presentation, education is under the control of governments, and PISA is pretty much the only tool they have to compare educational outcomes. Unfortunately, it focuses on a very narrow aspect of education and is accused of using discredited statistical techniques.
John Kay’s key point was that we often achieve the ends we desire indirectly. He summed this up perfectly by his example of ICI, the chemical company. I haven’t got the exact quotations he used, but in the 1970s ICI’s mission statement focused firmly on innovation and customers. ‘Shareholder value’ was merely a by-product of the core function of the company. By the 1990s this had reversed, with ‘shareholder value’ being the number one priority.
Of course, ICI no longer exists having lost its innovative edge. Likewise, if we focus on narrow and questionable measures of education to make comparisons with other countries, we miss the point of learning. PISA is the educational equivalent of ‘shareholder value’. Focusing on the by-product rather than the core mission is worrying.
Perhaps it’s time to take education out of the hands of politicians?
If you’re interested in debating the purpose(s) of education, you might want to join the debate at purposed.org.uk. Look especially at our call for more #500words contributions!
You know what? If I could, god-like, step outside of time and decide just when to make a large-scale change to western education systems, I think I’d choose right now. Why? People are ready for change. The current system isn’t working and we haven’t got the money to prop it up any more.
And you know what else? If I could choose an organisation to come up with an alternative system, I’d entrust a non-governmental international organisation that had demonstrated its non-profit and open credentials to drive things forward. Kind of like Mozilla, then. But you’d also need big-hitters and a backer credible with educators. Step forward HASTAC, the US Department of Education and NASA.
It’s now or never. As a father of two children under five, I passionately want something different in place for my children going forward. I think this is the only chance for changing educational assessment radically before my children become adults.
If you’re involved with education at any level you know how much assessment drives learning. Whether we’re talking about intrinsic or extrinsic motivation relating to badges, we can all agree of the importance of getting something out of learning experiences. Something that shows what you know. That’s why I think badges are perfect for MOOCs, for example.
But it’s no good sitting on the sidelines. I’m off to the Mozilla Festival today to meet more people involved in the #openbadges project and, hopefully, Mark Surman (CEO of Mozilla) to discuss his proposal for a web literate planet. What can YOU do? Well, there’s $2 million of (international) competition money available for a range of things relating to badges:
- I encourage educators to get involved in the content competition, identifying the types of content may suit badges. Closing date: 14 November 2011
- I encourage researchers (faculty members and students alike) to get involved in the research competition to establish a sound theoretical basis for badges. Closing date: 28 November 2011
- I encourage designers and coders to get involved in the design competition to implement the technologies required for a badge infrastructure. Closing date: 12 January 2012
As I’ve explained recently, the world doesn’t change in and of itself: it takes human agency to do so. Are you going to step up and help move things forward?
Image CC BY-NC-SA keoshi
Seven weeks ago I proposed a ‘semester of learning’ about Mozilla’s Open Badges. This was originally going to be hosted on an installation of BuddyPress, but eventually resided at P2PU.org in a group called Open Badges and Assessment. It attracted a diverse mix of people, most of whom I’d never encountered before (I love it when that happens!) Many of use are continuing the conversation at a new HASTAC group.
In a similar way to a MOOC (Massive Online Open Course) the semester of learning was an informal affair where participants (of which there were 84 altogether) could be as active as they want. Again, as with MOOCs, many were content to listen in upon what others were talking about. Others played a more active role. I’ve archived the study group, but it will remain available indefinitely on P2PU.org for your perusal.
Things tailed off slightly towards the end, for two reasons. The first was that I was in the last couple of weeks of my thesis, so was spending all of my spare time on that. Secondly, the conversation moved from being in a niche area to being much more mainstream (via Twitter, etc.) with the launch of the DML Competition
As a taster of what went on in the semester of learning here’s some comments from the beginning and towards the end.
There are key questions around ensuring quality for these badges to take hold. If they are to become something valuable on a CV for example then a prospective employer needs to be able to ascertain the level & rigour involved in the aquisition of the badge. (Dan Stucke)
I’m really impressed by the scope of this Mozilla project. I must admit, I signed up merely because I am interested in looking at ways for developing badges in a high school context, so to see this scale up in such a monumental way is pretty inspiring.
The potential for a new standard in qualifications that learners continue to build upon is very interesting. For example, my own degree and teaching qualifications are relatively old compared to everything I have learned since, and even though there is no formal recognition of my increased learning over the years, save a few references from previous employers, I’d say the undocumented skills I have now make me a far more qualified person than I appear to be on paper. I think the case studies from the open badge system framework draft make this point quite well. (Jackson Bates)
My main worry about the badges appraoch is that it will only be a kind of add-on to the normal educational modle. What I’m mainly interested in doing is entering into direct confrontation with the university as it currently exists. I want to fight with the university, offer an alternative to it, and fundamentally challange the values at work in the university. I’m worried that a badge just isn’t going to cut it, that it won’t be taken seriously enough or that it will only be taken seriously as an add-on to a “real” university education. (Thomas Gokey)
Every time a new educational fad erupts it seems to be polarizing, which seems to hold true in the conversations surrounding the dml announcement. Instead of talking about whether we agree or disagree with the movement a better topic would be, what can these badges do for education, specifically assessment?
I am excited to see what comes of the research grants for the badges. Will we start giving badges instead of end of course assessments/exams? Would that be a good thing? How would it work?
Yes of course it would be messy, but what if students had to obtain specific badges to pass into the next grade or to receive a high school diploma? Would it motivate students to complete their coursework or would it only further increase dropout rates? At any rate it is obvious that we would have to get the buy in of students to pull this off effectively…. (AndiStrack)
In the twitter about the grants, people expressed concern that there would be a proliferation of badges of dubious value. Nobody can stop that from happening and it would not be desirable. Our organization plans to categorize and rank badges by difficulty. We think our website that lists the badges will get substantial traffic just as our lists of open textbooks have done. (Jacky Hood)
I found P2PU.org a fantastically easy way to setup a study group and would certainly do so again. I think that the semester of learning helped point people towards certain resources that they may not otherwise have seen and, perhaps more importantly, engage with other people they may not have come across. It was great to see, given some of the superficiality and shallow reading evident from those reacting in various backchannels during the announcement, that those who were part of the group were committed to going away to think and read.
What did we learn? Well, I think I can speak on behalf of us when I say that talking of ‘badges for lifelong learning’ sounds simple but actually contains a lot of nuance and hidden complexity around assessment. I’m very much looking forward to continuing the conversation both on Twitter (using the hashtags #openbadges and #dmlbadges) and within the new HASTAC group. 🙂
In My Belbin Results – Part 1 I outlined what the Belbin process is and listed the nine different characteristics that the process identifies for those who are part of a team. At the end of the blog post I asked people, whether they knew me solely online or also offline, to ascertain which three of the nine characteristics were most like me. Go and read that post (and especially the comments) before proceeding. 🙂
It was interesting that those who know me solely online seem to view me differently from those who know me offline as well. That showed up in my ‘official’ Belbin report as well – there was one external assessor who I’ve only ever talked to on Skype and over the phone.
So what were my results? In order:
Plant – Creative, imaginative, unorthodox. Solves difficult problems. Ignores incidentals. Too pre-occupied with own thoughts to communicate effectively.
Resource Investigator – Extrovert, enthusiastic, communicative. Explores opportunities. Develops contacts. Over-optimistic. Can lose interest once initial enthusiasm has passed.
Shaper – Challenging, dynamic, thrives on pressure. Has the drive and courage to overcome obstacles. Prone to provocation. Liable to offend others.
The above were agreed upon by all five of my observers, apart from one who came up with ‘Specialist’ instead of ‘Shaper’. As for me, I had down Plant and Shaper, but had ‘Monitor Evaluator’ down as number one. Perhaps I’m not so ‘serious minded’ after all… 😉
The pigeon-holing is interesting but some of the report was intriguing. There were six people who assessed me, if you include the self-assessment; here are the top words and phrases people selected to describe me (in order of most frequent):
- encouraging of others
- technically skilful
- professionally dedicated
The word ‘aggressive’ also came out a couple of times, but right next to it was ‘calm & confident’ so I saw them as cancelling each other out. Talking to one of the people who assessed me, they explained the former as positive and being akin to ‘tenacious’. 🙂
There’s various other bits of feedback you get, including a ‘strengths’ and ‘possible weaknesses’ report, along with (hilariously) a ‘counselling report’. Here’s some choice excerpts:
Has innovative tendencies and needs to work in a mentally challenging environment. Requires work where he can use his outgoing nature… Needs to work in an environment which offers scope for personal expression.
Could have problems adapting to a supportive and subordinate role when necessary.
Needs to work within a loose framework. Will function best when given the freedom to roam.
Yours is essentially a pioneering profile. You are one of the few people equally read to develop new ideas on your own or in conjunction with others. Your best line of work is one in which you are required to explore possibilities and to take advantage of new opportunities. You have some features of the visionary. But take care you do not become isolated from others and resistant to the contributions they can make to the development of what is new.
For you above all others, it is best to establish the moment of exit. Do not outstay your welcome.
Your operating style is that of one who always seeks to be at the cutting edge of change. So remember that this is a hazardous spot to occupy. You will need to respect others of more traditional habits if you are to win respect yourself.
Does that seem a fair assessment? 😀
At the JISC infoNet quarterly planning meeting on Tuesday we got our Belbin feedback. For those who don’t know what that is (which would have included me until recently), go and read the Wikipedia article.
I’m not a huge fan of being pigeon-holed, but I found the results interesting nevertheless. I’ve only got a paper version of the results at the moment and, given it’s copyrighted material, I’m just going to share edited highlights. 🙂
There are nine defined roles with the Belbin process, the characteristics of which an individual is judged to exemplify to a greater or lesser extent. These are:
- Plant – Creative, imaginative, unorthodox. Solves difficult problems. Ignores incidentals. Too pre-occupied with own thoughts to communicate effectively.
- Resource Investigator – Extrovert, enthusiastic, communicative. Explores opportunities. Develops contacts. Over-optimistic. Can lose interest once initial enthusiasm has passed.
- Co-ordinator – Mature, confident. Clarifies goals. Brings other people together to promote team discussions. Can be seen as manipulative. Offloads personal work.
- Shaper – Challenging, dynamic, thrives on pressure. Has the drive and courage to overcome obstacles. Prone to provocation. Liable to offend others.
- Monitor Evaluator – Serious minded, strategic and discerning. Sees all options. Judges accurately. Can lack drive and ability to inspire others.
- Teamworker – Co-operative, mild, perceptive and diplomatic. Listens, builds, averts friction. Indecisive in crunch situations.
- Implementer – Disciplined, reliable, conservative in habits. A capacity for taking practical steps and actions. Somewhat inflexible. Slow to respond to new possibilities.
- Completer Finisher – Painstaking, conscientious, anxious. Searches out errors and omissions. Delivers on time. Inclined to worry unduly. Reluctant to let others into own job.
- Specialist – Single-minded, self-starting, dedicated. Provides knowledge and skills in rare supply. Contributes on only a limited front. Dwells on specialised personal interest.
For those who know me (either wholly through my work online or in person) I’d be interested in you participating in a little experiment:
If YOU had to choose three of these roles to describe me, which would you choose? Why?
(for a ‘Brucey bonus’ list some keywords you’d use to describe me)
I’ll share the keywords and roles my colleagues think fit me best in a forthcoming post. 😀
Over at P2PU.org I’m co-ordinating a ‘semester of learning’ on Mozilla’s upcoming Open Badges framework. This past week we’ve been looking at ‘learner stories’, scenarios for using badges to credentialise learning. Here’s my (fictional) attempt to explain how badges could work in the contexts with which I’m familiar.
Sarah: recognition for extra-curricular learning
Sarah is a 15 year-old pupil in an average English secondary school. As a pupil on the ‘gifted and talented’ list, she is working towards 14 GCSEs, including the English Baccalaureate subjects. Her passion, however, is music. She is a guitarist in a newly-formed band, something her parents and schoolteachers deem a distraction from her studies.
Recently, Sarah’s desire to spend more time writing and recording music has come into conflict with her studies. She has started to dabble in music production, but knows that to get a qualification in this area will probably have to wait.
How badges help:
Hearing of a new project using badges to credentialise learning in the music production arena, Sarah talks to her tutor and parents and the next parent/teacher evening about her passion for music. Because there is a way to credentialise it, her parents and tutor are happy with her dropping one of her GCSEs to free up time to pursue music production.
Within a few months, Sarah has her Music Production 101 badge. Realising she has module exams coming up, she decides to focus solely on her schoolwork for a month, then goes back to work on a badge that has been custom-made for her by a local producer: Producing kick-ass guitar music.
Sarah’s other band members begin to take on her ideas much more as she can talk knowledgeably about how something will sound as they are writing. Her teachers and parents notice she is happier in and with school, and notice an increase in Sarah’s confidence.
Badges provide a way to show parents and teachers the value of particular topic/subject/interest. In a world of high-stakes testing, badges allow for the credentialisation of passions, interests, and curiosities.
CC BY-NC rachel sian
What: An informal, collaborative group learning more about Mozilla’s Open Badge architecture.
Why: To consider the ways Open Badges could be used to credentialise educational outcomes.
When: Saturday 13th August – Friday 30th September 2011
I proposed a ‘semester of learning’ on Open Badges and assessment earlier this week. The idea seemed to gain some enthusiasm and traction, so I’ve gone ahead and set up a study group at P2PU.org. There’s no real need for a commitment other than joining the group and lurkers are as welcome as frequent contributors.
There’ll be a live Q&A and discussion session via IRC at 20.00 BST (GMT+1) tonight, Saturday 13th April, and every Saturday night. The link to get involved with that is in the sidebar at P2PU.org.
The first week’s task is really easy: read up on Open Badges so that in Week 2 we can negotiate how we’re going to move forward discussing and debating how they could be used within education.
Update: This will now take place at P2PU.org
At the Thinking Digital Conference 2011 Nicole Yershon, Director of Innovative Solutions at Ogilvy mentioned an idea that immediately struck a chord with me. The idea? Having a semester of learning. This, of course, is a term loosely borrowed from universities but the way Nicole described and applied it (before I drifted off into a reverie of what I could spend a semester learning about) was much more self-directed and informa.
I’d like to propose a collaborative semester of learning.
After being initially sceptical, I’m now super-excited about the revolutionary potential of Mozilla’s Open Badges project and I want to investigate it further. I want to go beyond the type of research I would do for a single blog post and go a lot more in depth.
I have a feeling some may wish to join me in this.
Mozilla’s project in a nutshell:
- Today’s learning happens everywhere, not just in the classroom. But it’s often difficult to get credit for it.
- Mozilla and Peer 2 Peer University are working to solve this problem by developing an Open Badges infrastructure.
- Our system will make it easy for education providers, web sites and other organizations to issue badges that give public recognition and validation for specific skills and achievements.
- And provide an easy way for learners to manage and display those badges across the web — on their personal web site or resume, social networking profiles, job sites or just about anywhere.
- The result: Open Badges will help learners everywhere unlock career and educational opportunities, and regonize skills that traditional resumes and transcripts often leave out.
I’m proposing the following as a basic minimum for this semester of learning:
- It lasts about six weeks, from Saturday 13th August to the end of September.
- Participants pool their findings and have asynchronous discussions (location TBC)
- Synchronous discussions as and when required.
The whole thing would be very light-touch, completely interest-based and informal. We’ll experiment with approaches for badge-giving within communities both educational and otherwise and, well, generally just see how it goes.
So… you in? (let me know in the comments)