DIRTy ToEs - Differentiation by Assessment

My main target as a teacher for this coming year builds upon the work I have done previously on using Taxonomies of Errors (ToEs) which you can read about in an earlier post on this blogsite. Although they are fantastic tools for sharing with students the common mistakes made by the whole class (to compliment the individual feedback given through my marking), these taxonomies do not of themselves cause students to make improvements to their practice. 

So, my target is to use DIRTy activities (Dedicated Improvement & Reflection Time for the uninitiated) in conjunction with my ToEs in order to ensure that the hours I spend on assessing an evaluating students' work are actually worth it. This post is my reflection on the first of my DIRTy ToEs lessons, with my Y12 Sociology class. 

The essay I had marked was one that I had asked them to do at home in timed conditions about the strengths and limitations of official statistics as a method for sociological research. 

In this follow-up lesson I wanted them to take the findings from an interview they had conducted with a family member on mortality rates and learn from their previous 'methods question' errors (individually and as a class) and act upon this learning. Having given them a whole period to feedback in groups and as a whole class about their positive and negative experiences of the interview process, comparing these experiences to the recognised strengths and limitations of the methodology, I then have them back their marked essays. 

When I mark essays at A-Level I scrawl my immediate thoughts on the essays as I read them, but then type up my overall impressions of strengths and areas for improvement on Fronter. I do this so that my comments are un-loseable and to enable the students to look at the comments I have made over time to see if any comments recur again and again. As this was their first formal feedback I printed these off and attached them to the essays. Here's what one of them looks like for Student A. 

After giving the students time to consider my comments and marking I then spoke with them about the Taxonomy of Errors for the whole class. As usual I categorised these into low level errors (ones that stop students achieving a pass), mid-level errors (ones that get in the way of a C) and high-level errors (ones that might prevent the awarding of the highest grades). This is what they looked like this time around.

At the lower level the issue of 'lifting' came about because I asked them to complete their essays at home and, being new AS students, they were more than reliant on the textbook. It's a problem that needs nipping in the bud, and the taxonomy approach helps greatly with this. 

At the mid-level, where most students found themselves, there is a greater array of errors. At this stage in the year, however, I am always keen to address the issue of wasted words (hence the point about repetitive conclusions) and the lack of examples to support points.

At the high level I was keen to stress the need to develop their understanding of 'validity' as a key theoretical construct for the subject, as it lags behind their understanding of 'reliability' and 'representativeness'. The other key point I wanted to focus on was students not accepting a supposed methodological strength or weakness without evaluating it or challenging it. 

The next step was to set them DIRTy activities that asked them to 'put right' an error from the class taxonomy that also featured on their individual feedback sheet. For those students who over-relied on the textbook the task was to rewrite a section in their own words. The students at mid or high levels of errors were given a little more choice to reflect the greater variety of errors that were made. 

The tasks they were given were:

Student A (whose feedback was shown earlier) chose to add a section on 'validity' from the higher DIRTy ToEs tasks. Her improvements looked like this:

All well and good. Not perfect but an improvement on what had gone before. 

Where the lesson really worked was in the main task that followed the DIRTy ToEs activity. 

In this activity, students were asked to go beyond the mistakes that they had already made and aim to skip the mistakes that those who had achieved more highly than them had made. If you want to strive for rapid progress, then this is the way to go!!!  Because Student A was already at the top of the taxonomy (do I need an even higher category for the highest level mistakes that haven't yet been made?) she had another stab at the validity issue and provided something much more coherent than even the DIRTy ToEs task. 

But Student A wasn't the only one to really improve the quality of her work in this lesson. Student B had made the mistake of copying chunks from the textbook, as this shows. 

 

She was required, by the DIRTy ToEs activity to take this section of her initial essay and rewrite it into her own words. This is what she produced. 

Much better!  And note the further addition of a sentence, showing evidence of evaluation and improvement. For her new essay Student B decided to focus on the following targets from the Mid-Level ToE. 

The section below shows how she managed to address the second of these targets. Notice the use of the qualifier 'can' used more than once in this paragraph. 

A third student, Student C, was halfway between A and B in her original essay, and her errors were at the mid-level of the ToE. 

As you can see from my marking in the picture above, one of the things that she needed to focus on in order to move closer to a C grade was to add specific examples of official statistic datasets to illustrate her general points about the method.  She responded wonderfully well to this in the DIRTy ToEs activity. 

As with Student B, C elected to retain the addition of examples from the mid-level ToE, as well as adding targets from the high-level ToE. 

She never managed to explore the term validity as she hoped, preferring instead to consolidate on her use of examples. 

This isn't a problem. When I feed back to her next time I will remind her of the unmet target and before then I shall be addressing the whole class's understanding of validity as a key issue that needs to be addressed. Remember that this post-DIRTy ToEs activity was about aiming to avoid mistakes that others above them had made: very much a stretch activity. The fact that she has addressed a significant issue for her and then consolidated it in a second piece of writing is good enough for me. 

Conclusion: Next steps on DIRTy ToEs

The addition of DIRTy activities to my already well-established Taxonomy of Errors technique has helped my teaching a great deal. 

Without the DIRT, my ToEs would simply be lists of what did and didn't work with a hopeful assumption that students follow up on these in their own time.  Without the ToEs, my DIRTy activities would be undifferentiated and decontextualised from the needs of the students. 

I shall be continuing to get my ToEs DIRTy in the coming months because the learning that was done this lesson, and the consequent improvement made by students, was exceptional. It took almost all of a triple period to get the students to that point, though, and I don't want to have to invest that amount of time in the future, so keeping it regular is going to be a key challenge for me. But I am also aware that repetition is a double-edged sword and, because of this, I want to look to vary the DIRTy activities as much as possible and even look to see how I can modify the ToEs too. 

I'll let you know how I get on. 

Using a 'Taxonomy of Errors' to Enhance Student Responses

I am writing this post to outline and to describe a technique I learnt some years ago from a long-forgotten colleague that has had a very positive impact on my teaching and the learning of my students:  the Taxonomy of Errors.  As well as describing the technique I want to show how I have developed it in recent weeks, how I have used it proactively with my students and what impact it has had on their written work.  Hopefully it may be useful to one or two of you.

In essence, the Taxonomy of Errors is a response to that perennial problem faced by teachers of dealing with a class that are all making the same mistakes.  It is a method for trying to ensure that students learn from each others' mistakes as well as from their own.  It is a method for trying to ensure that when they next attempt the same task, they will improve markedly rather than incrementally because they have addressed a range of foibles in their work.

In practice, the Taxonomy of Errors is little more than a summary of all the feedback that you have given to the students as individuals with a focus on the comments you have written time after time after time.  Sometimes I have even been known to rank them in order of frequency!!!

This is an example of a Taxonomy of Errors (ToE) for my top set English class.  I took them over two weeks ago and immediately got them to complete a full practice exam for me so I could see what they could do in the heat of battle.  This is the ToE for the important question on the Writing Section, which requires them to analyse language across two non-fiction texts and make comparisons.  As you can probably guess this response was not perhaps their finest hour, mainly because they had forgotten exam techniques in ensuring that they performed well in their controlled assessments.  This strategy is part of my portfolio of tools for getting them prepared for their exams later this month.  You will notice that the vocabulary is negative.  Although I sometimes write ToEs positively (there is an example later) the intention is to be bluntly honest about what went wrong, and the fact that the list is based on the whole class makes it easier to be so (although I am bluntly honest individually too!).

Here is another example of a Taxonomy of Errors, this time in response to a collection of timed essays from my AS Sociology students.  In this ToE I made an attempt to separate out the basic errors (at the top) and the more complex error (at the bottom) to ensure that they understood the things that they really ought not have done, regardless of their ability, and the genuine areas for further learning.  I always find that at this time of the academic year students facing imminent external examination make all kinds of foolish errors and they need to be scuttled in order to allow them to get at the more important stuff that will genuinely allow them to achieve higher grades (we always call them the Sheep Grades because they are B, A, A* - Geddit?). I always present this feedback at the very start of the lesson after I have marked the work and am increasingly linking the feedback directly to the activities of the next lesson so that they improve upon their work immediately.

Here's my latest feedback to my Y11 English students, and in it I have focused even more on the creation of a genuine Taxonomy of Errors (that may have been how it was intended and I have just found my way to it the long way around!!): from the basics that are genuinely beneath this groups of students, to the intermediate and advanced.  This feedback was from an essay on An Inspector Calls where the students were (to a greater or lesser extent) all guilty of simply trying to rewrite their controlled assessment work on Arthur and Sheila rather than respond about the Inspector as they had been asked.  I was therefore able to show them how the error at the intermediate level was preventing them from accessing the higher grades, and thereby making the feedback on higher level errors virtually redundant (most students achieved only between 17 and 21 marks out of 30 because of their intermediate errors).

Here is a very typical response from one of the students that led to this Taxonomy of Errors, and was even one of the better ones because he made sure to refer to the Inspector on four occasions.  In the end, though, his best observations were reserved for the character of Sheila and he would be reliant upon the leniency of the examiner at best and, at worst, reliant upon another question coming up that allowed him to crowbar in his understanding of the character of Sheila.  In response to the ToE then, I devised a sequence of activities that was focused on using 15 minutes of their 45 to plan effectively to answer the actual question, not the one that they wish had been set.  This involved brainstorming what they knew already about the Inspector, selecting apposite quotes (ensuring that at least one of these was from the stage directions to allow the response to include reference to stagecraft - from the Advanced section of the ToE), exploring the language of the quotes for dominant and subversive interpretations and then evaluating the quotes in light of the essay question.

And here is the response that the same student generated during this double lesson in response to the activities I had set; activities which had been informed by the Taxonomy of Errors (at this stage of the year my lessons are almost entirely planned in response to their emergent needs). I'm not claiming that it is a startlingly better response, but it does tackle the intermediate error of not focusing on the question to ensure he has a chance at the top marks.  Further to this it addresses the advanced error of a lack of reference to stagecraft and deepens his use of language analysis from a straightforward discussion of the word 'horrible' to a more convincing analysis of the words 'taking charge masterfully' in a better quotation that, again, had more resonance with the actual question he had been asked to answer.

So there you have it.  In my classroom the Taxonomy of Errors is used for three purposes.  In its most simple guise it tells the students what mistakes have been made, by others as well as themselves, so that they can get a sense of their achievements alongside those of their peers.  At a more sophisticated level the Taxonomy of Errors allows me to rank the impact of different errors on their marks and/or grades by showing them how basic or intermediate errors can undermine work that in other ways might have the potential to achieve highly.  But the Taxonomy of Errors is at its most effective when it informs my planning so that students are taught (or re-taught) the knowledge or skills that had been demonstrated so poorly in their submitted work.  The Taxonomy of Errors is at its most potent when it is used in this way and results in the students being asked to edit or rewrite the error-strewn original in a conscious attempt to improve it.  The Taxonomy of Errors is at its most rewarding when it helps make marking have a genuine impact on learning.



SOLO from the assessment cradle to the assessment grave

On Monday this week the results of my Dissertation (and therefore final Masters level) were released by Middlesex University. When I finally was able to logon to the unihub portal - the first time I had tried in my three years on the school-based programme - they only gave me my mark and didn't say what level of award it was equivalent to. I had a hazy recollection that it was in a handbook I had been given and so I looked it up and there, to my utter surprise, was SOLO taxonomy staring back at me!!!

To say I was stunned is an understatement. Although SOLO wasn't a part of my dissertation's conclusions about what good assessment should look like, I found my way to it immediately after thanks to @biomadhatter, and the way SOLO dovetailed with the conclusions of my research was serendipitous in the extreme: it was as if my findings were preparing me for the revelatory nature of SOLO in action. But what shocked me was the fact that the same levels of understanding I had been teaching Y7 English and Y12 Sociology students to apply to their work were being used at Masters level to distinguish between a distinction (extended abstract), a merit (relational) and a pass (multistructural).

Just the previous week I had launched SOLO with my Y12s in a lesson where I got them to highlight their A2 mark scheme for AQA Sociology according to SOLO taxonomy (see my post 'Keeping up with the Soloists'). The results were crystal clear; that extended abstract was the top band, relational the top half of the middle band, multistructural the bottom half of the middle band and so on. This was a powerful lever for me in getting students to appreciate the value of SOLO.

And so, being a fan of serendipity, I decided to use the Middlesex university mark scheme for Masters level dissertations to mark my AS Sociologists' essays, rather than the A-Level mark scheme to see if it would work as well. The results were amazing.

The first of the essays I marked was a real mixed bag, which using a traditional mark scheme would have been really difficult to evaluate. Because I would have been looking at a number of the different mark bands I would have had to cobble together a best fit statement that would have offered little to the student. Instead I am able to point out where she is prestructural, but also where she is multistructural and, at her best, relational. My concluding comments are therefore far more effective at giving her feedback, feed up and feed forward: a proper evaluative response for her to draw strength from and use to identify improvements.

In this second student's response there was far less variability of understanding, and so I was able to focus my feed forward on moving up from multistructural in certain places, to secure a fully relational response as a springboard to aiming at completely extended abstract thinking (I'm hopeful of her achieving an A grade next year, and so this mark scheme fits very well with her targets for the coming year).

This final example is a close-up on the feedback, feed up and feed forward I provided for a third student who has the potential to achieve a grade A or even A* next year. Although I can see things that I might do differently (it was my last lesson with them and I had to mark these essays as they worked interdependently) the thing I am most pleased about with these evaluations is that I found writing the positives and negatives equally easy. When using the A-Level mark scheme I always find that I operate on a deficit model with students, pointing out all the faults whilst finding it hard to look back and see the achievements.

The difference that I have noted from this first foray into using SOLO taxonomy for written evaluation (notice I am not using the word assessment as it seems too shallow for the feedback, feed up and feed forward I have given) is that I am able to evaluate each paragraph and even sentence independently of the whole. This precision of evaluation has allowed me to really accurately pinpoint which concepts they have not grasped, or where they have not applied theory to sociological contexts, or where they have failed to evaluate sociological theories effectively. In doing so, it has helped me to promote the importance of consistency across the essay to them and they were quick to identify sections where they needed to go away and revise concepts or rewrite essays. If I had had more time with them I would have asked them to rewrite the section they felt most able to improve, perhaps after having spent time getting ideas from their peers based upon my evaluation.

I told the students afterwards that I had marked their essays using a Masters level mark scheme. What amazed them most was that SOLO taxonomy could be used to evaluate their work as well as mine, as well as GCSE coursework and as well as KS3 portfolios. It was clear that they realised that thinking about levels of understanding (as exemplified by the SOLO concepts of Extended Abstract, Relational, etc) was something that could connect them to their studies beyond A-Levels, and the fact that they were realising this in our last lesson before their UCAS preparation week was not lost on them, or me.

And what surprised me most? The fact that not one of them asked what grade they had received. And the fact that none of them asked what grade I had received. SOLO was enough for them. And that's enough for me.