Diversifying Assessment 2: Setting standards

Section 2: Setting standards

Reproduced with permission from Brown, S. Rust, C. and Gibbs, G. Strategies for Diversifying Assessment in Higher Education Oxford: Oxford Centre for Staff Development (1994)

If assessment is to achieve any of the functions listed in the previous section, it is vital that it has validity. Whether it is intended as feedback to students on their progress or as part of the grading process for a final qualification, common standards must be applied. However, this is easier said than done; assessment is by no means a pure science. Nevertheless, marking - of essays and exams in particular - is something virtually all academics have to do and most rapidly gain a good deal of experience in it. The marking exercise below draws on this experience.

2.1 An essay marking exercise

Assume that the two essays reproduced below were written by students on an introductory course in technology and its social implications. Mark each of them out of 10 (with 4 as a pass mark). In the spaces indicated at the end of each essay make comments on their strengths and weaknesses, and give advice to the students on how to improve their essays .

This exercise was previously published in Gibbs, G. Module 3 Assessment, Certificate in Higher Education by Open Learning, OCSD (1989).

Commentary

Many hundreds of teachers from all subject disciplines have marked these two essays. There are broad similarities in their comments and their marks, but also important differences. While teachers are often doing something quite similar when they are assessing students' work, they can also disagree profoundly.

Essay 1 is usually given a mark in the range 3-7 with an average of about 5. It is generally considered to have no introduction, no main body, and no conclusion. The order in which the content of the essay is presented could be changed and it would make little difference, because there is no developing argument. The essay consists of facts in a list with no analysis. It is usually thought not to answer the question. The student who wrote it is often commended for having done some relevant background research but criticised for having made little appropriate use of this research.

In contrast Essay 2 is usually given a mark in the range 5-10, with an average of 7. Tutors comment that it goes beyond the scope of the question and considers the nature of the question itself, and the difficulty of reaching simple conclusions. It has a clear argument, draws a conclusion of its own, has a beginning, a middle and an end, and raises and considers a number of issues along the way. It is often considered to be well written.

In short, the author of the first essay is regarded as a poor student and the author of the second a comparatively good student. Whatever our subject expertise, we seem capable of making such general assessments of students and their work.

However there are complications. The general unanimity in the tutors' assessment of the two essays masks some sharply divergent views. The range of marks given for each essay is wider than the difference between the essays. Some people give Essay 1 7/10 almost a first. And some give Essay 2 a bare pass. Some even give higher marks for Essay 1 than Essay 2. This variation extends to the comments teachers write. Some think that Essay 1 does all you could reasonably ask: it provides a good range of facts upon which a decision can be made. Essay 2 is then described as being all over the place, providing little evidence to back up its irrelevant assertions. To some extent, those who feel this way tend to come from the 'hard' sciences. It would appear that sometimes, but not always, different sets of values and expectations operate in different subject areas.

These similarities and differences illustrate some important points about assessment:

• there is broad agreement as to what constitutes quality: about what we should be looking for when we assess students in higher education;
• there are wide differences between teachers in their standards and judgements: often the difference between teachers' appraisal is greater than the difference between students' performance;
• there are identifiably different values operating, to some extent related to subject disciplines;
• these similarities and differences have little to do with the knowledge the authors of the essays displayed: these general characteristics are exhibited independently of the specific content of the essay.

2.2 Making the criteria explicit

You were able to mark the essays in the previous exercise because you had certain criteria in your mind. The reason why teachers in higher education vary in the marks they award to the two essays is that they are using different criteria or, if they are using the same criteria, they are giving different weightings to them in terms of importance. This exercise underlines the importance of working to explicit criteria.

There are good reasons for making criteria explicit.

To be fair to the student.
When you set a task or assignment, make clear the criteria on which it will be assessed, so the student can tailor her or his response to your requirements.

To avoid 'academic drift'.
This is the term used to describe student responses where the criteria are unclear; they assume the key criterion to be an emphasis on content.

To encourage staff to adopt common standards in marking.
If a number of different staff members are involved in marking a piece of work, clearly stated criteria should help to bring closer the standards to be applied. It also gives a clearer basis for discussion of any disagreements either between staff or between students and staff.

Assessment criteria can be introduced to the students in a number of ways. Some of these are suggested below.

Simply specify them

Opposite is an example of a marking protocol where the assessment criteria have been standardised for a whole degree programme within a modular structure. This grid is reproduced in the front of each module's student handbook. While not every criterion would necessarily be relevant to every piece of work in every module, where the criterion is relevant the standards for the different grades are clearly defined. They can then be applied in exactly the same way in each module. The student's overall grade for the piece of work is then based on the column in which the statements are concentrated.

Marking criteria protocol for Health Care Studies degree in a modular programme

Specify the criteria, and get the students to use them in a 'marking' exercise

However clearly criteria are stated, students may not immediately understand them, or appreciate which are the most important, especially at the beginning of a course. A marking exercise can help to overcome this.

Get students to mark work.
Take one or two examples of work, possibly from a previous year, and copy them for this year's cohort to mark individually using your criteria. They can then discuss each other's marking in groups and negotiate an appropriate grade. The experience of applying the criteria and discussing the outcomes is likely to reveal any lack of understanding and help students to see what is required of a 'good answer'.

Set a marking assignment.
Give students the task of marking three pieces of work from a previous year's students. Grades for this can be based on how closely their marks for the three pieces correspond to those given by the tutor. This may sound harsh but the tutor on one engineering course where this has been done argues that as he will be marking much of their work for the next three years, it is important that students understand what he is looking for!

If you do not currently involve your students in any kind of marking exercise, consider how and where it might be possible to introduce it on your course.

Run an exercise for students to generate criteria

• For inexperienced students, this could be as simple as asking them to identify the characteristics of 'the perfect essay' individually or in pairs.
• Consider using the 'First class answer' overleaf.

This is an exercise you might use to help your students see the standards, values and criteria that are being applied in the assessment of their work. Try it for yourself.

(This exercise was previously published in Habeshaw, T., Gibbs, G., and Habeshaw, S., (1987), 53 Interesting Ways of Helping your Students to Study. Bristol TES. )

Once your students have generated their own criteria, you then have to decide whether you use them, or ignore them and use your own. A compromise may be to have certain criteria which are compulsory and non-negotiable, but to allow the students to add their own to these.

For example, in assessing an essay your three non-negotiable criteria might be:

• length of between two and three thousand words;
• use of at least three referenced texts;
• attention to appropriate syntax, punctuation and spelling.

Feedback sheets

Making the criteria explicit not only makes clear to students what is expected of them but also offers you a way of giving detailed feedback in a relatively undemanding way, e.g. by turning the listed criteria into a feedback sheet, as in the example [below].

Essay assessment profile
Name	Mark
GOOD POINTS	SUGGESTIONS
STRUCTURE/ARGUMENT	STRUCTURE/ARGUMENT
Introduction Thesis/focus well defined	Introduction Clearer focus needed  Explain your approach
Argument Well-stated Well supported by data/secondary sources Historically sound Good synthesis skills Critical use of material Good understanding of question	Argument Explain the direction of the argument Analyse more; describe less Deeper analysis needed Identify problems with interpretation Rethink organisation Stay in touch with question Re-work/extend conclusion
Style presentation Content Well-chosen, relevant Avoids trivial content Differences between historians noted Ideological positions explained Good selection of books used	Style presentation Develop neater presentation Improve your handwriting Did you proof-read? (Careless errors) Get sentence structure right Avoid bland quotes. Paraphrase! Ensure meaning is clear. Be critical!
Technical Sources acknowledged References/bibliography correctly set out	Subject specific skills Content Be ruthless with irrelevant material Is all the detail strictly necessary? Incorporate a range of interpretations Use more sources - good ones! Demonstrate use of sources
	Technical Check your referencing format Use references Check layout of bibliography

The strategies outlined in this section are all designed to make explicit to students the criteria by which their work will be assessed. These strategies stem from the conviction that students will then be better placed to deliver work likely to fulfil these criteria. Detailed feedback on the extent to which students have been successful in meeting the criteria then becomes the point of departure for the next 16 phase of the student's learning.