Tag: Research (page 1 of 9)

Evaluation: the absolute basics

Whilst I’ve done some work in the past around evaluation I’ve needed to brush up on it since joining the Mozilla Foundation. This post reflects some hours spent in Durham University Education Library earlier this week.

Office of Mayhem Evaluation

Introduction

Evaluation is a contested term. Even people involved in the field can’t agree what they mean:

“No single-sentence definition will suffice to fully capture the practice of evaluation.”

(Patton, 1982:4, quoted in Clarke, 1999)

However, the following general guidance is useful:

“The most important purpose of evaluation is not to prove but to improve.”

(Stufflebeam and Shinkfield, 1985:2, quoted in Clarke, 1999)

And among the definitions I’ve come across, the one I like best is this:

“Evaluation is the systematic investigation of the merit or worth of an object (program) for the purpose of reducing uncertainty in decision making.”

(Mertens, 1998:219)

The ‘evaluand’

Like every field, Evaluation methods has its jargon terms. For example, an evaluand is the subject being evaluated. The following are useful questions asked by Mertens (1998:231) in relation to the evaluand:

  1. Is there a written description of what is to be evaluated?
  2. What is the status of the evaluand? Relatively stable and mature? New? Developing? How long as the program been around?
  3. In what context will (or does) the evaluand function?
  4. Who is the evaluand designed to serve?
  5. How does it (the evaluand) work? Or how is it supposed to work?
  6. What is it supposed to do?
  7. What resources are being put into the evaluand (e.g. financial, time, staff, materials, etc.)?
  8. What are the processes that make up the evaluand?
  9. What outputs are expected? Or occur?
  10. Why do you want to evaluate it?
  11. Whose description of the evaluand is available to you at the start of the evaluation?
  12. Whose description of the evaluand is needed to get a full understanding of the program to be evaluated?

Dimensions to evaluation

There are many dimensions to evaluation, the most commonly known being summative vs. formative evaluation. Whilst formative evaluation has as its audience those within the program being evaluated, the audience for summative evaluation is those outside the program (such as policy makers, funders, or the general public).

Other dimensions over and above formative vs. summative should be considered (according to Malla Reddy, 2000:3) including:

  • Inside vs. Outsider
  • Experimental vs. Illuminative
  • Democratic vs. Bureaucratic
  • Product vs. Process
  • Quantitative vs. Qualitative

The last bullet point of this list has many books and articles dedicated to each element. Very basically, quantitative evaluation focuses on ‘hard’ numbers, whereas qualitative evaluation focuses on ‘soft’ experience.

Planning an evaluation

Mertens (1998:230) suggests the following steps when planning an evaluation study:

Focusing the evaluation

  • Description of what is to be evaluated
  • The purpose of the evaluation
  • The stakeholders in the evaluation
  • Constraints affecting the evaluation
  • The evaluation questions
  • Selection of an evaluation model

Planning the evaluation

  • Data collection specification, analysis, interpretation, and use strategies
  • Management of the evaluation
  • Meta-evaluation plans

Implementing the evaluation

  • Completing the scope of the work specified in the plan

O’Sullivan (2004:7) gives a brief overview of the various evaluation models or approaches from which to choose:

  1. Objectives – focuses on objectives to determine degree of attainment
  2. Management – focuses on information to assist program decision makers
  3. Consumer – looks at programs and products to determine relative worth
  4. Expertise – establishes peer and professional judgements of quality
  5. Adversary – examines programs from pro and con perspectives
  6. Participant – addresses stakeholders’ needs for information

To be honest, all of this seems a little over-the-top for some of the things I’ll be evaluating. That’s why I found Colin Robson’s book Small-scale Evaluation useful. Robson (2000:46) suggests using the following questions in an evaluation:

  1. What is needed?
  2. Does what is provided meet client needs?
  3. What happens when it is in operation?
  4. Does it attain its goals or objectives?
  5. What are its outcomes?
  6. How do costs and benefits compare?
  7. Does it meet required standards?
  8. Should it continue?
  9. How can it be improved?

Structuring an evaluation report

Robson also suggests how to structure an evaluation report (2000:122):

  • Heading – make it short and clear
  • Table of contents – simple list of headings and page numbers (without subheadings)
  • Executive summary – key findings and conclusions/recommendations
  • Background  – one-page setting of the scene as to why the evaluation was carried out, what questions you are seeking answers to, and why the findings are likely to be of interest
  • Approach taken – when, where and how the study was carried out (detail goes in appendices)
  • Findings – the largest section giving answers to the evaluation questions with the main message going at the beginning and using subheadings where necessary
  • Conclusions/recommendations – draws together main themes of the report and their implications
  • Appendices – include information needed by the audience to understand material in the main report (references)

He also suggests including the names/contact details of the evaluators.

Conclusion

This brief overview of evaluation should enable me to be more confident when evaluating things for my day-to-day role. Hopefully it’s also given you enough of a starting point to carry out your own evaluations.

Image CC BY-NC-SA xiaming

References

  • Clarke, A. (1999) Evaluation Research: an introduction to principles, methods and practice, London: Sage
  • Malla Reddy, K. (ed.) (2000) Evaluation in Distance Education, Hyderabad: Booklinks Corporation
  • Mertens, D. (1998) Research Methods in Education and Psychology: integrating diversity with quantitative and qualitative approaches, London: SAGE
  • O’Sullivan, R.G. (2004) Practicing Evalution: a collaborative approach, London: Sage
  • Robson, C. (2000) Small-scale Evaluation, London: Sage

 

Sullen.

“Well, you do come across as a bit sullen.”

“Really?”

“And you do whinge a lot on your blog.”

The Sullen Cow

I suppose I do.

Not everything is broken. Not everything has a dark, fetid underside. Not everything is a commoditised, market-influenced mess.

What I try to communicate (here and elsewhere) is the importance of recognising how inter-connected and inter-dependent we are. I think it’s far too easy to focus on the former without recognising (and acting upon) the latter.

For example, I’m delighted that TeamGB has met with so much success at the 2012 Olympic Games. But it’s far from merely a collection of individual performances. There’s infrastructure involved. There’s interdependence. There’s solidarity.

Many of our athletes rely not only on British infrastructure, but that of other countries – for example British Somali gold medallist Mo Farah, who trains in the US alongside American athlete Galen Rupp. There’s a complicated, tangled, inter-dependent web of human relationships that allow elite athletes to succeed.

Let’s take another example. Breakthroughs in science and technology rarely take place solely because of an individual moment of genius. Rather, they occur because (in Newton’s phrase) individuals and teams “stand on the shoulders of giants”. Researchers, academics, technologists – people who work primarily with their brains rather then their bodies – also rely upon a network of human relationships.

So forgive me if I complain about journal articles being behind paywalls. Forgive me when I point out that access to the best facilities is reserved for those with privilege, status, and money. Forgive me if I come across as sullen. Somewhat paradoxically, it’s because I care so much about others.

Image CC BY-NC theunquietlibrarian

My new work blog and other RSS goodies.

Doug's Work Blog

Spurred by several things including our most recent JISC infoNet planning meeting and Will Richardson’s decision to quit long-form blogging at Weblogg-ed and move to Tumblr, I’ve set up a work blog at http://dajbelshaw.tumblr.com

The theme hopefully reflects how I want to use it – as a visual snapshot of my research. Rest assured that, unlike Will, I’ll still be blogging here as well. I love writing. My work blog is more for clipping and quickly commenting on stuff relating to Open Educational Resources, Mobile Learning and Digital Literacies (my 3 main research areas).

As a reminder, you can also find other posts by me at:

I’ve collated the RSS feeds for my research and original writing in two separate über-feeds (which you can also subscribe to via email if you click through):

Doug's Writing Feed Doug’s Writing Feed (http://feeds.feedburner.com/DougsWritingFeed)

Doug's Research Feed Doug’s Research Feed (http://feeds.feedburner.com/DougsResearchFeed)

I’ve updated the sidebar at dougbelshaw.com/blog to make these quick and easy to find. For those people wanting to do something similar, RSSmix seemed to be lot easier and hassle-free than fiddling with Yahoo! Pipes…

From my research: New Literacies around the world

In case you’ve not subscribed to the RSS feed yet, I’m updating my new blog literaci.es regularly with the outputs from my ongoing Ed.D. work:

‘Digital literacy’ in Norway?

The history and status of digital literacy in Norway is complex. The term is presumed by English-speaking researchers and educators to mean, in a straightforward way, the same in Norwegian as it does in English. However, given the difficulty in translating words such as ‘literacy’ into Norwegian, and words such as ‘kompetanse’ from Norwegian, ‘media literacy’ is a term preferred increasingly to ‘digital literacy’.

 

New literacies (or the lack of them) in Singapore

In this standards-based, heavily-pressured educational culture – a society where, anecdotally, painkillers are stocked alongside exam-preparation books (Bracey, 2008) – it is unsurprising to find the dominant ‘new literacy’ to be Media Literacy. In addition, much of the available research literature into new literacies comes from, or through the lens of, Singapore’s National Institute of Education.

 

Digital Media Literacy in Australia

The seeming Australia-wide agreement on Digital Media Literacy as the accepted form of New Literacies is explained in part by Gibson (2008). He gives an overview of the ‘literacy wars’ in Australia, quoting Ilyana Snyder on how the press and professional journals keep alive the debates between conservatives and progressives (Snyder, 2008). The battleground over different forms and manifestations of traditional (print) literacy allows, suggests Gibson, Digital Media Literacy to show “some promise of a revival of educational optimism” (Gibson, 2008, p.74).

 

The USA: a New Literacies desert?

Due to the standards-based, testing culture in US schools, NYC’s approach is understandable. They have adopted the publication of an authoritative body who, in turn, have reacted to an environment created by US educational policy in the wake of NCLB. Such an environment stresses the importance of being ‘information literate’ and focuses on the traditional basics but, perhaps, at the expense of a cohesive programme for New Literacies.

 

5 free, web-based tools to help you be a kick-ass researcher.

UNIVAC

I do a lot of research. Not only is my day job Researcher/Analyst at JISC infoNet but when I go home I’m researching and writing as part of my doctoral thesis. Quantity and quality are different measures, but I’d hope that I’m at least half-decent at something I spend a fair amount of my life doing.

Being a researcher before the internet must have been a very difficult occupation. Much less access to information but, I suppose, on the other hand, it must have been a much more ’embodied’ existence than spending hours mediated by several different kinds of screens. Without a focus it’s very easy to become confused very quickly and be like a dog chasing after shiny cars.

My focus at the moment, as shown by dougbelshaw.com/research is upon:

  • Open Educational Resources
  • Mobile Learning
  • Digital Literacy

I use several tools to stay up-to-date in these areas and to discover new resources. Here’s five of the best:

Twitter + Storify

Storify

This goes without saying: Twitter is my social dashboard and an absolute treasure trove of useful information. The important thing is that it’s a network (of networks) of people who have expertise, influence and opinion.

Recently I’ve started using Storify to, for want of a better phrase, ‘curate tweets’ about stuff I’m researching. Here’s an example for iPad mindmapping apps. Asking a question, getting replies, curating them and re-sharing helps everybody.

LinkedIn Signal

LinkedIn Signal

This feels like, in a phrase Ewan McIntosh used five years ago, giving away some kryptonite as LinkedIn Signal is truly amazing for researching specific terms. It’s based on your LinkedIn connections, which I’m careful to keep based on people I’ve met. It shows your relation to that person but also the most discussed links about that search term.

Try it. You’ll love it.

Amplify

Amplify

Amplify is for ‘clipping’ content from websites and adding your comments to it. You can find my most recent clippings in the sidebar of this blog. The power of Amplify, however, is twofold: (i) the people you follow who often post things you wouldn’t come across, and (ii) the search functionality.

Futurelab’s EducationEye

EducationEye

The ever-innovative Futurelab have recently announced EventEye, a paid-for version of EducationEye for (unsurprisingly!) events. EducationEye is a service that pulls in posts from blogs (including this one) and arranges them in a visually pleasing and useful way.

Again, there’s a search function available but it’s also handy for serendipitous dipping in and out of in order to keep up with the zeitgeist.

Quora

Quora

I use Quora about once per week. It’s a social question-and-answer site where people can vote answers up and down and summarise answers once there’s plenty of responses. It can work very well and there’s an extremely diverse mix of people on there. It’s certainly worth ‘tracking’ questions to see what kinds of responses they get and from whom.

Conclusion

So there we are! Five recommendations of tools that help me be a better researcher. What have I missed?

Image CC BY-NC-SA Stuck in Customs

My favourite quotations from ‘Teaching with the Tools Kids Really Use’

Teaching with the Tools Kids Really UseIn my current role at JISC infoNet I’m working on a Mobile Learning infoKit to be released later this year. One of the books I’ve been reading in my research for that resource is Teaching with the Tools Kids Really Use: Learning with Web and Mobile Technologies. Whilst it has some relevance to Further and Higher Education I think it’s more directly applicable to schools.

As I did with 10 things I learned from ‘Why Don’t Students Like School?’, the following are some choice quotations from the book:

Failure to adapt

In the long run, [the] flattening of the world can be advantageous for everyone. But to realize these benefits, people from all walks of life, including (and perhaps especially) educators, need to let go of doing business and usual and begin adapting to the changing world. Emerging nations have been quick to pick up the gauntlet – perhaps because they had little to lose and everything to gain. Developed nations have been more resistant to making the changes needed to thrive in this new global society – perhaps because they fear they have everything to lose. But not taking action is a recipe for failure for these nations. (p.1)

Need for more use in order to develop effective models

[W]e find ourselves in the equivalent of the frontier. Until we are able to openly explore effective uses of these technologies as tools for teaching and learning, we are not going to be able to cite good models. (p.3)

Effective education and technology

Effective education is the foundation of successful societies. But in recent years, at least in developed countries, the survival of the existing institution seems to have trumped the importance of providing relevant, timely instruction. This trend can be changed, but the time to take action is now. One way to move education forward is to embrace emerging technologies that make it possible to implement programs where students master core academic content, hone applied 21st-century skills, and learn how to find success in an increasingly digital world. (p.3)

The futility of banning mobile phones

How does [routine confiscation of mobile phones] waste time? Because many students are turning over either old, disconnected phones or replica phones, which they have purchased online for about two dollars. Students cheerfully relinquish and retrieve these devices each period while retaining possession of their real phones… It is far better to find positive ways cell phones can be used as tools for teaching and learning by identifying and enforcing realising parameters within which students may have cell phones in their possession than to fight what is ultimately a losing – and unnecessary – battle. (p.15)

Mobile phones and etiquette

Students misuse cell phones in exactly [the same ways as the rest of society]. But how are they to learn better behavior without appropriate adult models who take the time to teach digital etiquette? Granted, parents need to take responsibility for teaching good manners to their children, but so do teachers and other school personnel who often spend more waking hours with students than do their parents! (p.18-19)

1:1 requires pedagogical underpinning

Experts generally agree that purchasing and installing equipment to reach a 1:1 ratio of students to computing devices is not enough to make a difference in academic achievement. For this investment to pay off, teachers need to rethink their approach to instruction by trying out student-centred strategies that focus on collaboration, communication, and problem solving. In short, although online research and word processing have their place, these activities are starting – not ending – points. (p.41-2)

Objections should not be deal-breakers

Unfortunately… objections are often used as deal breakers. Although it’s important that these concerns be put on the table, the driving purpose should be to enable educators to have open discussions about potential unintended consequences. Once everyone’s concerns are out in the open, it’s possible to consider solutions or strategies for working around problems. (p.113)

Exciting times for educators

This is an exciting time to be an educator. The possibilities for reaching and engaging students are growing daily. As new tools for communication and collaboration continue to be developed and made readily available to people around the world, educators continually need to adapt their approach to instruction to ensure that classroom activities remain relevant. Fortunately, these changes are doable. All that’s required is the will to move forward. (p.121)

My research is your research.

dougbelshaw.com/research

In the spirit of ‘there’s nothing new under the sun’ and ‘you are what you share’, I present:

http://dougbelshaw.com/research

I’ve seeded it with OER, mobile learning and digital literacies. There’s also a found, an electic repository of interesting stuff I come across.

This can be considered part of my ongoing project to streamline and share each part of my workflow. See Web apps and workflows for more by way of explanation. You’ll notice I’ve added a ‘Research’ link in the top navigation for ease-of-access.

HOWTO: Set up Google Scholar to do the heavy lifting for you.

I have unlimited love for Google Scholar, I really do. It’s the one tool that I really wouldn’t be without for academic purposes these days; I really wish it had been around when I was doing my BA in Philosophy and MA in Modern History. Still, I’m not grumbling – it’s around for my Ed.D. research! 🙂

There’s two really powerful things you can do with Google Scholar. The first, which I’ve mentioned to many people many times before, is click on the ‘Cited by…’ link underneath search results. This helps you find seminal papers fast.

The second is the subject of this post – integrating your access to electronic journals with Google Scholar. I’m fortunate in having two methods now – through Durham University because of my Ed.D. research and now through Northumbria University, hosts of JISC infoNet (for whom I now work).

(click on images to enlarge!)

Step 1

Go to Google Scholar and click on ‘Advanced Settings’

Google Scholar howto 01

Step 2

Enter the name of your university/institution in the ‘Library Links’ box and click the button ‘Find Library’.

Google Scholar howto 02

Step 3

When Google Scholar comes up with some suggestions, click the ones that are appropriate. Then click the ‘Save Preferences’ button.

Google Scholar howto 03

Step 4

Search using Google Scholar as usual. Links to PDFs, etc. will appear to the right. Click on them and then login using your university/institution password. You will be directed straight to the PDF without having to login to various repositories.

Google Scholar howto 04

Step 5

Use Google Chrome and the Docs PDF/Powerpoint Viewer extension for quick and easy access! 😀

Google Scholar howto 05

The end of the beginning.

I suppose it’s a bit of a random day to start (April Fool’s Day, the last day before a public holiday…) but I begin a new job today that I’ve very excited about. I’m delighted to announce that I’ve signed a two-year contract (I sound like a professional footballer!) with JISC infoNet as Researcher/Analyst:

JISC infoNet aims to be the UK’s leading advisory service for managers in the post-compulsory education sector promoting the effective strategic planning, implementation and management of information and learning technology.

The team are a great bunch who I’ve already been in to meet since my successful interview a couple of months ago. I’m looking forward to extending my knowledge and experience in education up to FE and HE level!

JISC infoNet is one of eight sub-sections of JISC Advance, which is funded by the UK taxpayer through the Research Councils. I’ll be researching (duh!), putting together infoKits and helping facilitate workshops in colleges of further education and universities around the country. I’m based at, although not actually part of (despite the new @northumbria.ac.uk email address) Northumbria University.

I’m happy to answer any questions you’ve got about the move by email – use this contact form. I’ll reproduce the most commonly-asked questions over at Doug’s FAQ. 🙂

Research supporting collaborative, enquiry-based learning.

Model of Learning - Tools for TeachingOne of the great things of studying in the Education Library at Durham University (instead of at home, in my study) is the books I randomly stumble across. For example, I pulled Models of Learning – Tools for Teaching off the shelf today and it fell open at Chapter 7, entitled ‘Learning through cooperative disciplined inquiry.’

This is perfect for me. One of my Performance Management targets for this year – the one focused on my own classroom practices – is about piloting enquiry-based learning with one of my Year 7 History classes. In addition, I’ll (hopefully) be presenting with Nick Dennis at the SHP Conference in July 2010 on this very topic – including the way technology can help! :-p

It’s always good to have some scholarly research to back up one’s actions, so if you’re planning to do something similar here’s some quotations to help you!

The most stunning thing about teaching people to help kids learn cooperatively is that people don’t know how to do it as a consequence of their own schools and life in this society. And, if anything is genetically driven, it’s a social instinct. If it weren’t for each other, we wouldn’t even know who we are. (Herbert Thelen to Bruce Joyce, circa 1964) p.95

The chapter is based on case studies across the age range, but also contains this nugget on p.98-9:

The assumptions that underlie the development of cooperative learning communities are straightforward:

  1. The synergy generated in cooperative settings generates more motivation than do individualistic, competitive environments. Integrative social groups are, in effect, more than the sume of their parts. The feelings of connectedness produce positive energy.
  2. The members of cooperative groups learn from one another. Each learner has more helping hands than in a structure that generates isolation.
  3. Interacting with one another produces cognitive as well as social complexity, creating more intellectual activity that increases learning when contrasted with solitary study.
  4. Cooperation increases positive feelings towards one another, reduces alienation and loneliness, builds relationships, and provides affirmative views of other people.
  5. Cooperation increases self-esteem not only through increased learning but through the feeling of being respected and cared for by others in the environment.
  6. Students can respond to experience in tasks requiring cooperation by increasing their capacity to work together productively. In other words, the more children are given the opportunity to work together, the better they get at it, with benefit to their general social skills.
  7. Students, including primary school children, can learn from training to increase their ability to work together.

The authors go on to summarise the evidence about improved learning through collaboration on p.99:

Classrooms where students work in pairs and larger groups… are characterized by greater mastery of material than the common individual-study/recitation pattern. Also, the shared responsibility and interaction produce more positive feelings toward tasks… In other words, the results generally affirm the assumptions that underlie the use of cooperative learning methods.

It’s not hard to get started with cooperative learning (p.100):

[A]n endearing feature is that it is so very easy to organize students into pairs and triads. And it gets effects immediately. The combination of social support and the increase in cognitive complexity caused by the social interaction have mild but rapid effects on the learning of content and skills.

The authors dismiss claims from some teachers that ‘gifted students prefer to work alone’ as the evidence does not back this up (Joyce 1991; Slavin 1991). They believe it may rest on a misunderstanding of the relationship between individual and cooperative study; partnership still requires individual effort. There’s no need to be concerned about students’ ability to work together (p.101):

In fact, partnership s over simple tasks are not very demanding of social skills. Most students are quite capable of cooperating when they are clear about what has been asked of them.

The Teacher's ToolkitI’ll not go into them here, but the authors mention a number of ways in which teachers can foster ‘positive interdependence’. They also suggest the ‘division of labour’ into specializations. Instead of learning only a part of what every is supposed to be learning, they have found, ‘jigsaw’ activities and the like lead to more learning across the spectrum. Many of the activities they suggest are, in fact, featured alongside others in one of my favourite education-related books, The Teacher’s Toolkit.

The teacher’s role in cooperative learning moves from that of instructor to ‘counsellor, consultant and friendly critic.’ (p.107) The authors note that this ‘is a very difficult and sensitive’ role ‘because the essence of inquiry is student activity’. Teachers need to:

  • facilitate the group process
  • intervene in the group to channel its energy into potentially educative activities, and
  • supervise these educative activities so that personal meaning comes from the experience

The upshot of this is that ‘intervention by the teacher should be minimal unless the group bogs down seriously’ (p.107).

The authors suggest a 6-phase process for cooperative learning:

Phase 1 – Students encounter puzzling situation (planned or unplanned).

Phase 2 – Students explore reactions to the situation.

Phase 3 – Students formulate study task and organize for study (problem definition, role, assignments, etc.)

Phase 4 – Independent and group study.

Phase 5 – Students analyse progress and process.

Phase 6 – Recycle activity.

In conclusion, the authors note how universally cooperative group investigation can be used (p.111-2):

Group investigation is a highly versatile and comprehensive model of learning and teaching: it blends the goals of academic inquiry, social integration and social process learning. It can be used in all subject areas, and with all age levels, when the teacher desires to emphasize the formulation and problem-solving aspects of knowledge rather than the intake of preorganized, predetermined information.

css.php