Month: February 2023

Voodoo categorisation and dynamic ontologies in the world of OER

Posted on February 15, 2023February 15, 2023 by Doug Belshaw

Note: this appeared on the now-defunct MoodleNet blog in April 2019. It’s something to which I keep wanting to refer but always struggle to re-find — so I’m republishing it here. (Thanks, Creative Commons licensing!) The MoodleNet project to which it refers no longer exists in its original ambitious form, and some of the information it contains may be out of date. However, I’m still interested in the broad-brushstroke ideas.

Introduction

In a previous post on this blog, I described how we’re planning for search to work in MoodleNet. In this post, I want to dig into tagging and categorisation – which, it turns out, is an unexpectedly philosophical subject. Fundamentally, it comes down to whether you think that subjects such as ‘History’ and ‘Biology’ are real things that exist out there in the world, or whether you think that these are just labels that humans use to make sense of our experiences.

What follows is an attempt to explain why Open Educational Resources (OER) repositories are often under-used, how some forms of categorisation are essentially an attempt at witchcraft, and why assuming user intent can be problematic. Let’s start, however, with everyone’s favourite video streaming service.

Netflix

If you asked me what films and documentaries I like, I’d be able to use broad brushstrokes to paint you a picture. I know what I like and what I don’t like. Despite this, I’ve never intentionally sat down to watch ‘Critically-acclaimed Cerebral Independent Movies’ (Netflix code: 89), nor ‘Understated Social & Cultural Documentaries’ (Netflix code: 2428) nor even ‘Witty Independent Movies based on Books’ (Netflix code: 4913). These overlapping categories belong to a classification system developed by Netflix that now stretches into the tens of thousands of categories.

Netflix is popular because the content it provides is constantly updating, but mainly because it gets to know you over time. So instead of presenting the user with a list of 27,000 categories and asking them to choose, Netflix starts from a basis of the user picking three movies they like, and then making recommendations based on what they actually watch.

There aren’t a lot of actions that users can perform in the main Netflix interface: it’s essentially ‘browse’, ‘add to list’ and ‘play’. In addition, users don’t get to categorise what they watch. That categorisation is instead performed through a combination of Netflix’s algorithm and their employees, which work to create a personalised recommendation ‘layer’ on top of all of the content available.

In other words, Netflix’s categorisation is done to the user rather than by the user. Netflix may have thousands of categories and update them regularly, but the only way users can influence these is passively through consuming content, rather than actively – for example through tagging. More formally, we might say that Netflix is in complete control of the ontology of its ecosystem.

Voodoo categorisation

In a talk given back in 2005, media theorist Clay Shirky railed against what he called ‘voodoo categorisation’. This, he explained, is an attempt to create a model that perfectly describes the world. The ‘voodoo’ comes when you then try to act on that model and expect things to change that world.

Shirky explains that, when organisations try to force ‘voodoo categorisation’ (or any form of top-down ontology) onto large user bases, two significant problems occur:

Signal loss – this happens when organisations assume that two things are the same (e.g. ‘Bolshevik revolution’ and ‘Russian revolution’) and therefore should be grouped together. After all, they don’t want users to miss out on potentially-relevant content. However, by grouping them together, they are over-estimating the signal loss in the expansion (i.e. by treating them as different) and under-estimating the signal loss in the collapse (i.e. by treating them as the same).
Unstable categories – organisations assume that the categories within their ontology will persist over time. However, if we expand our timescale, every category is unstable. For example, ‘country’ might be seen as a useful category, but it’s been almost thirty years since we’ve recognised East Germany or Yugoslavia.

The ontologies we use to understand the world are coloured by our language, politics, and assumptions. For example, if we are creating a category of every country, do we include Palestine? What about Taiwan? These aren’t neutral choices and there is not necessarily a ‘correct’ answer now and for all time. As Shirky points out, it follows that someone tagging an item ‘to_read’ is no better in any objective way than conforming to a pre-defined categorisation scheme.

This is all well and good theoretically, but let’s bring things back down to earth and talk very practically about MoodleNet. How are we going to ensure that users can find things relevant to what they are teaching? Let’s have a look at OER repositories and the type of categories they use to organise content.

OER repositories

The Open Education Consortium points prospective users of OER to the website of the Community College Consortium for Open Educational Resources. They have a list of useful repositories, from which I’ve chosen three popular examples, highlighting their categories:

[the original post had a JavaScript-powered gallery that is now lost]

These repositories act a lot like libraries. There are a small number of pre-determined subject areas into which resources can be placed. In many ways, it’s as if there’s limited ‘shelf space’. What would Netflix do in this situation? After all, if they can come up with 27,000 categories for films and TV, how many more would there be for educational resources?

Ultimately, there are at least three problems with OER repositories organised by pre-determined subject areas:

Users have to fit in with an imposed ontology
Users have to know what they are looking for in advance
Users aren’t provided with any context in which the resource may be used

We are trying to rectify these problems in MoodleNet, by tying together individual motivation with group value. Teachers look for resources which have been explicitly categorised as relevant to the curriculum they are teaching. Given the chance, great teachers also look for ideas in a wide range of places, some of which may be seen as coming from other disciplines. MoodleNet then allows them to provide the context in which they would use the resource when sharing their findings with the community.

Dynamic ontologies in MoodleNet

Our research has shown that, as you would expect, educators exhibit differences in the way they approach finding educational resources. While there are those that go straight to the appropriate category and browse from there, equally there are many who prefer a ‘search first and filter later’ approach. We want to accommodate the needs of both.

The solution we are proposing to use with MoodleNet includes both taxonomy and folksonomy. That is to say, it involves both top-down categorisation and bottom-up tagging. Instead of coming up with a bespoke taxonomy we are thinking of using UNESCO’s International Standard Classification of Education (ISCED) fields of education and training which provides a three-level hierarchy, complete with relevant codes:

Table showing UNESCO ISCED fields of education and training — Example of some of the UNESCO ISCED fields of education and training

MoodleNet will require three types of taxonomic data to be added to communities, collections, and user profiles:

Subject area (ISCED broad, narrow, or detailed)
Grade level (broadly defined – e.g. ‘primary’ or ‘undergraduate’)
Language(s)

In addition, users may choose to add folksonomic data (i.e. free-text tagging) to further contextualise communities, collections, and profiles. That would mean a collection of resources might look something like this:

Mockup of taxonomic and folksonomic tagging system in a MoodleNet collection

The way Clay Shirky explains this approach is that “the semantics are in the users, not in the system”. In other words, the system doesn’t have to understand that the Bolshevik Revolution is a subset of 20th century Russian history. It just needs to point out that people who often tag things with ‘Lenin’ also tag things with ‘Bolshevik’. It’s up to the teacher to make the professional judgement as to the value of a resource.

We envisage that this combination of taxonomic and folksonomic tagging will lead to a dynamic ontology in MoodleNet, powered by its users. It should allow a range of uses, by different types of educators, who have varying beliefs about the world.

Conclusion

What we’re describing here is not an easy problem to solve. The MoodleNet team does not profess to have fixed issues that have beset those organising educational for the past few decades. What we do recognise, however, is the power of the web and the value of context. As a result, MoodleNet should be useful to teachers who are looking to find resources directly relevant to the curriculum they are teaching. It should also be useful to those teachers looking to cast the net more widely

In closing, we are trying to keep MoodleNet as flexible as possible. Just as Moodle Core can be used in a wide variety of situations and pedagogical purposes, so we envisage MoodleNet to be used for equally diverse purposes.

Voodoo dolls image CC BY Siaron James

Weeknote 06/2023

Posted on February 10, 2023February 10, 2023 by Doug Belshaw

Sitrep: I’m starting to write on Friday at 17:00, sitting in the car at the side of an artificial football pitch, and not believing the temperature gauge when it says it’s 10°C outside. These things don’t factor in wind chill, as I have learned from bitter experience. Ten Storey Love Song by The Stone Roses has just come on the radio, so it looks like the weekend is starting off a positive note.

I published three posts on three different blogs this week:

There’s another one that I’ve got in draft form on Google Docs which you can take a peek at if you like; I don’t particularly care if you see how the sausage is made. Thanks to John Bevan, ChatGPT, and several people in my network for grammar corrections, content, and questions, respectively!

Last week I neglected to link to January’s issue of Thought Shrapnel so I’ll do so now. So far in February I’ve published the following over there:

On the work front, this seems to be the time of year where we wait for clients to catch up with the work we’ve done for them. I’ve spent a good bit of time in Whimsical, a tool which does a great job of allowing you to quickly visualise things which are either unspoken or buried in mountains of text. I like it.

Without wanting to be enigmatic, I’m going to continue talking about the type of work I’ve been doing this week rather than the particular projects. So, for example, I’ve been:

Re-writing a Code of Conduct which lacked specificity around the spreading of misinformation and steps that would be taken to protect the community if the CoC was contravened.
Coming up with a new workflow to help a client zero-in on what an ambiguously-defined new thing would look like in practice. We used people/product/process, a favourite categorisation of mine, for the second step.
Collaborating with Bryan, who is now working with us through Visual Thinkery. We wanted his help in visualising some things related to setting up new co-ops and what cooperation looks like in practice.
Discussing workers.coop with John Atherton about how WAO can help as it comes into existence.
Creating a visual overview of the equivalent of a ‘sales funnel’ for attracting, training, and retaining people for a particular internal course which we’ve developed and iterated over the last couple of years.
Advising people who booked a slot for the CoTech Digital Candle service.
Running an advisory session for staff of an organisation we’re helping prepare for their offsite next week. We’re helping them with tips around presenting and facilitation.

I mentioned in my last weeknote that I was taking our daughter to the Newcastle United vs West Ham match on Saturday. It turned out that I then went to the Sunderland vs Fulham FA Cup 4th round replay game on Wednesday night. She was supposed to come, but then worked out that she’d be tired for her Science test the next morning. My dad still came though and, although Sunderland lost, we still had a good time.

It’s the usual four sportball games this weekend for our offspring. I do enjoy watching them, but doing so means that I do precious little else. I may manage to get to wash the cars on Saturday morning and hang out with my gaming buddies on Sunday evening. It’s a good job that I only work around 25 hours per week and my life is otherwise deliberately low-stress 😅

Photo of a copy of the book (The Names by Don DeLillo) that I was reading and enjoying until earlier this week. Unfortunately, I had to stop reading immediately as it was impossible for me to continue. Can you see why?

I’m pretty sure ‘status update’ meetings aren’t work

Posted on February 7, 2023February 7, 2023 by Doug Belshaw

Status update meetings are ones where no decisions are made and no forward planning takes place. As such, they can be considered superfluous to well-run organisations and effective collaborations. There are better ways to manage people and projects.

I find that status update meetings are a bad habit that organisations get into for one of several reasons. It could be that they don’t know better. With these kinds of organisations, working with WAO and organisations like us can be a revelation. In fact, that’s been the case many times, especially with smaller charities.

Another reason for the status update meeting can be a lack of standardised toolset. In these kinds of organisations, everyone uses their own ‘to-do’ list, from pen and paper through to some complicated digital workflow. The status update meeting therefore acts as an inefficient kind of ‘API’ (or translation) between these siloed systems.

A third reason that status update meetings exist is that people are employed to work fixed hours. This is the most pernicious. It might not even be a conscious thought, but if you’ve got hours to fill, there’s nothing as low-bar as a status update meeting to while away the time.

The easiest way to get out of the habit of status updates is to know what the alternative is, to decide on a standardised toolset, and to turn those meetings into co-working sessions.

At WAO we used the simplest tools possible to get the job done. Over-complicated toolsets and workflows are the enemy of collaboration and, in fact, can be thought of as a form of procrastination.

Essentially, all you need is a place to put three lists: To Do, Doing, and Done. If you’re physically co-located this could even be on a wall. WAO uses Trello as we find it everyone just ‘gets’ it. You can add extra lists as necessary (we use ‘Epics’, ‘Feedback’, and ‘Zombie Garden’).

What this means is that status update-related conversations happen on the Trello board. The meetings that used to happen to keep everyone up-to-date can now either be eliminated or turned into co-working meetings.

At WAO, we have at least one co-working meeting per client every week. In these meetings we check in, bringing our full selves to work, prioritise what needs doing, and then either work on those things together, or divide and conquer. As we’re fully-remote, the latter looks like muting audio and video for a set period of time (usually 15, 30, or 45 minutes) and working on a task. We can can unmute and ask questions if necessary.

The above can sound like it might drain the fun out of work. I can assure you it’s the opposite. Status update meetings drain energy out of people and projects. Co-working and representing progress visually is invigorating.

Try it! You might be surprised.