The Art of Assessing

A toolkit of techniques

Assessment can take many forms, and it can be argued that the greater the diversity in the methods of assessment, the fairer assessment is to students. The art of assessing therefore needs to embrace several different kinds of activity. In this article, I would like to encourage colleagues to broaden the range of assessment processes, and I have tried to provide practical suggestions about how to maximise the benefits of each of a number of methods I have addressed. In each case, I have also listed a few of the advantages of using the method, as well as some of the corresponding drawbacks.

Whether we think of ourselves as lecturers, or teachers, or facilitators of learning, the most important thing we do for our students is to assess their work. It is in the final analysis the assessment we do that determines their diplomas, degrees, and future careers. Over the last decade, many of us have seen our assessment workload grow dramatically, as we work with increasing numbers of student, who are ever more diverse. Consequently, the time we have available to devote to assessing each student has fallen. It is therefore more important than ever to cultivate 'the art of assessing'.

God-given gift?

Just as it seems to be assumed that anyone appointed to a teaching post in higher education can automatically teach, it is also implicit that they should be able to assess students' work. Many teachers in higher education wield their red pens for the first time without ever having had any real training in how to assess. Many are embarrassed at the notion of even asking for any guidance, yet are quite intimidated at the responsibility attached to assessing.

A very private act

Teaching is a public affair, and we get all sorts of feedback regarding how well or how badly we teach - even without deliberately seeking feedback. The expressions on students' faces, the attendance at our classes, and the level of students' performance all help us to adjust our teaching techniques. Assessment, however, tends to be a private and intimate affair, and there is seldom anyone looking over our shoulders as we go about designing and implementing assessment. Given the importance of assessment, it is probably the aspect of our profession that should be scrutinised most carefully. Even with the best of intentions, external examiners and moderators can only contribute a limited amount to the processes of assessment, and the primary responsibility for assessment continues to rest with teachers.

Marks, grades and feedback

One of the most useful benefits of assessment can be feedback gained by students on their performance regarding skills they are intended to develop, and their understanding of theories and concepts. It is an important part of the learning process for students to be able to learn from their mistakes as well as their triumphs. With larger class-sizes and increasing workloads, the time staff can devote to giving students detailed feedback on their work has been substantially eroded. It is therefore worth considering whether alternative forms of assessment (student peer-assessment in particular) can help increase the amounts of feedback which students can derive from assessed work.

Why assess?

There are many reasons for assessing students performance. Not all are good reasons. Some of the reasons are:

• we live in a society where people are appointed and employed on the basis of their qualifications.
• students themselves need feedback to help them to find out how their learning is going.
• we need feedback on how well students' learning is going, so that we can adjust and develop our teaching.
• assessment is often the major driving force which gets students down to serious studying.

Ten Forms of Assessment

1. Traditional exams

Traditional 'unseen' exams still make up the lion's share of assessment in higher education. Despite growing concern about the validity and fairness of this type of assessment, for all sorts of reasons it will continue to play a large part in the overall assessment picture. Despite many concerns about exams, I have tried in the following discussion to suggest a number of ways that the use of exams can be improved. I have given more 'tips' for setting exam questions than for setting the other nine types of assessment explored in this article, as in general, good practice in writing exam questions overlaps with, or extends across, each of the other types.

Advantages

• Relatively economical. Exams can be more cost-effective than many of the alternatives (though this depends on economies of scale when large numbers of students are examined, and also on how much needs time and money needs to be spent to ensure appropriate moderation of students' performance).
• Equality of opportunity. Exams are demonstrably fair in that students have all the same tasks to do in the same way and within the same timescale. (However, not all things are equal - ask any hay-fever sufferer, or candidate with menstrual problems).
• We know whose work it is. It is easier to be sure that the work being assessed was done by the candidates, and not by other people.

• Students get little or no feedback about the detail of their performance, which is therefore wasted as far as feedback is concerned.
• Badly set exams encourage surface learning, with students consciously clearing their minds of one subject as they prepare for exams in the next subject.
• Technique is too important. Exams tend to measure how good students are at answering exam questions, rather than how well they have learned.

• Don't do it on your own! Make sure you get feedback on each of your questions from colleagues. They can spot whether your question is at the right level more easily than the author. Having someone else look at one's draft exam questions is extremely useful. It is better still when all questions are discussed and moderated by teams of staff. Where possible, set questions with your colleagues. This allows the team to pick the best questions from a range of possibilities, rather than use every idea each member has.
• Ask colleagues: 'what would you say this question really means?'. If they tell you anything you hadn't thought of, you may need to adjust your wording a little.
• Keep your sentences short. You're less likely to write something that can be interpreted in more than one way if you write plain English in short sentences. This also helps reduce any discrimination to students whose second language is English.
• Work out what you're really testing. Is it decision-making, strategic planning, problem solving, data processing (and so on), or is it just too much dependent on memory? Most exam questions measure a number of things at the same time. Be up-front about all the things each question is likely to measure.
• Don't measure the same things again and again. For example, it is all too easy in essay-type exam questions to repeatedly measure students' skills at writing good introductions, firm conclusions, and well-structured arguments. Valuable as such skills are, we need to be measuring other important things too.
• Include data or information in questions to reduce the emphasis on memory. In some subjects, case-study information is a good way of doing this.
• Make the question layout easy to follow. A question with bullet points or separate parts can be much easier for (tense) candidates to interpret correctly than one which is just several lines of continuous prose.
• Don't overdo the standards. When you're close to a subject, it's easily possible that your questions get gradually harder year by year.
• Write out an answer to your own question. This will be handy when you come to mark answers, but also you'll sometimes find that it takes you an hour to answer a question for which candidates have only half-an-hour.
• Decide what the assessment criteria will be. Check that these criteria relate clearly to the syllabus objectives or the intended learning outcomes. Make it your business to ensure that students themselves are clear about these objectives or intended outcomes, and emphasise the links between these and assessment.
• Work out a tight marking scheme. Imagine that you are going to delegate the marking to a new colleague. Write it all down. You will find such schemes an invaluable aid to share with future classes of students, as well as colleagues actually co-marking with us, helping them to see how assessment works.
• Use the question to show how marks are to be allocated. For example, put numbers in brackets to show how many marks are attached to various parts of the question (or alternatively, give suggested timings such as 'spend about ten minutes on Part 2').
• Try your questions out. Use coursework and student assignments to give components of your future exam questions pilot runs, and use or adapt the ones that work best for exams.

2. Open-book exams

In many ways these are similar to traditional exams, but with the major difference that students are allowed to take in with them sources of reference material. Sometimes, in addition the 'timed' element is relaxed or abandoned, allowing students to answer questions with the aid of their chosen materials, and at their own pace.

Advantages

These have many of the advantages of traditional exams, with the addition of:

• Less stress on memories! The emphasis is taken away from students being required to remember facts, figures, and other such information.
• Measuring retrieval skills. It is possible to set questions which measure how well students can use and apply information, and how well they can find their way round the contents of books and even databases.
• Slower writers helped? If coupled with a relaxation in the 'timed' dimension (e.g. a nominal '2-hour' paper where students are allowed to spend up to three hours if they wish) some of the pressure is taken away from those students who happen to be slower at writing down their answers (and also students who happen to think more slowly).

Disadvantages

• Not enough books! It is hard to ensure that all students are equally equipped regarding the books they bring into the exam with them. Limited stocks of library books (and the impossibility of students purchasing their own copies of expensive books) means that some students may be disadvantaged.
• Need bigger desks? Students necessarily require more desk-space for open-book exams if they are to be able to use several sources of reference as they compose their answers to exam questions. This means fewer students can be accommodated in a given exam room than with traditional unseen exams, and therefore open book exams are rather less cost-effective in terms of accommodation and invigilation.

Tips on setting open-book exam questions

All of the suggestions regarding traditional exam questions still apply. In addition.....

• Decide whether to prescribe the books students may employ. This is one way round the problem of availability of books. It may even be possible to arrange supplies of the required books to be available in the exam room.
• Set questions which require students to do things with the information available to them, rather than merely summarising it and giving it back.
• Focus the assessment criteria on what students have done with the information, and not just on them having located 'the correct information'.
• Expect shorter answers. Students doing open book exams will be spending quite a lot of their time searching for, and making sense of, information and data. They will therefore write less per hour than students who are answering traditional exam questions 'out of their heads'.

3. Structured exams

These include multiple-choice exams, and several other types of formats where students are not required to write 'full' answers, but are involved in making true/false decisions, or identifying reasons to support assertions, or fill in blanks or complete statements, and so on. It is of course possible to design 'mixed' exams, combining free-response traditional questions with structured ones. In the following discussion, I will concentrate on the benefits and drawbacks of multiple choice questions, which also apply at least in part to other types of structured exam questions.

Advantages

• Greater syllabus coverage: it is possible, in a limited time, to test students' understanding of a much greater cross-section of a syllabus than could be done in the same time by getting students to write in detail about a few parts of the syllabus.
• Multiple choice exams test how fast students think, rather than how fast they write.
• Saving staff time and energy. With optical mark readers, it is possible to mark multiple choice exams very cost-effectively, and avoid the tedium and subjectivity which affect the marking of traditional exams.
• Testing higher-level skills? Multiple choice exams can move the emphasis away from memory, and towards the ability to interpret information and make good decisions.

• The guess factor. Students can often gain marks simply by lucky guesses rather than correct decisions.
• Designing structured questions takes time and skill. It is harder to design good multiple-choice questions than it is to write traditional open-ended questions.
• Black and white or shades of grey? While it is straightforward enough to reward students with marks for 'correct' choices (with zero marks for choosing distractors), it is more difficult to handle subjects where there is a 'best' option, and a 'next-best' one, and so on.
• The danger of impersonators? The fact that exams composed entirely of multiple-choice questions do not require students to give any evidence of their handwriting increases the risk of substitution of candidates.

• Try out questions with colleagues and with large groups of students. Make sure that people are, on the whole, selecting correct options for the right reasons - and not because in one way or another the question gives away which is the correct option.
• Make sure that distractors are plausible. If no-one is selecting a given distractor, it is serving no useful purpose. Distractors need to represent anticipated errors in students' knowledge or understanding.
• Try to avoid overlap between questions. If one question helps students successfully to answer further questions, the possibility increases of students picking the right options for the wrong reasons.
• Pilot questions in formative-tests before using them in summative exams. Ideally, multiple-choice questions that appear in formal exams should be tried-and-tested ones. It is worth consulting the literature on multiple-choice question design and finding out how to assess the discrimination index and facility value of each question from statistical analysis of the performance of substantial groups of students.
• Write feedback responses to each option. Where possible, it is useful to be able to explain to students selecting the correct (or best) option exactly why their selection is right. It is even more useful to be able to explain to students selecting the wrong (or less-good) options exactly what may be wrong with their understanding. When multiple choice questions are computer-marked, it is a simple further step to get the computer to print out feedback responses to each student. This practice can equally be applied to formative multiple-choice tests, and to formal multiple-choice exams. Furthermore, the availability of feedback responses to each decision students make lends itself to extending the use of such questions in computer-based learning packages, and even computer-managed exams.
• Ensure that students are well-practised at handling multiple-choice questions. Answering such questions well is a skill in its own right, just as is writing open answers well. We need to ensure that students are sufficiently practised, so that multiple-choice exams measure their understanding and not just their technique.
• Look at a range of published multiple-choice questions. For example, several Open University courses have multiple-choice assignment questions, as well as multiple-choice exams. You may be surprised how sophisticated such questions can be, and may gain many ideas that you can build into your own question-design.
• Gradually build up a large bank of questions. This is best done by collaborating with colleagues, and pooling questions that are found to be working well. It then becomes possible to compose a multiple-choice exam by selecting from the bank of questions. If the bank becomes large enough, it can even be good practice to publish the whole collection, and allow students to practise with it.

4. Essays

In some subjects, assessment is dominated by essay-writing. Traditional (and open-book) exams often require students to write essays. Assessed coursework often takes the form of essays.

Advantages

• Essays allow for student individuality and expression. They are a medium in which the 'best' students can distinguish themselves.
• Essays can reflect the depth of student learning. Writing freely about a topic is a process which demonstrates understanding and grasp of the material involved.
• Students are used to writing essays. In many disciplines, essays represent the form of assessment which students are most familiar with. However, mature students often admit that the medium of assessment which worries them most is essays.

• Essay-writing is very much an art in itself. Students from some backgrounds are disadvantaged regarding essay-writing skills as they have simply never been coached in how to write essays well. For example, a strong beginning, a coherent and logical middle, and a firm and decisive conclusion combine to make up the hallmarks of a good essay. The danger becomes that the presence of these hallmarks is measured time and time again, and students who happen to have perfected the art of delivering these hallmarks are repeatedly rewarded irrespective of any other strengths and weaknesses they may have.
• Essays take time to write (whether as coursework or in exams). This means that assessment based on essay-writing necessarily is restricted regarding the amount of the syllabus that is covered directly. There may remain large untested tracts of syllabus.
• 'Write down the number we first thought of!' Essays are demonstrably the form of assessment where the dangers of subjective marking are greatest. Essay-marking exercises at workshops on assessment show marked differences between the mark or grade that different assessors award the same essay - even when equipped with clear sets of assessment criteria.

Most of the suggestions given earlier in this article about writing traditional exam questions continue to apply - whether essays are to be used as assessed coursework or as exam questions. Some further suggestions are given below.

• Help students to see exactly how essays are marked. Alert students to the credit they gain from good structure and style. One of the best ways of doing this is to involve classes of students in looking at examples of past (good, bad and indifferent) essays, and applying assessment criteria. This can be followed by involving students in peer-assessment of each others' essays.
• Help students develop the skills required to assemble the 'content' for essays. One of the best - and most time-effective - ways of doing this is to set class or coursework tasks which require students to prepare essay-plans rather than fully-finished masterpieces. A concept-map or diagram can show a great deal about the eventual 'worth' of students essays, and can avoid distraction from the elements of style and structure. Students can put together maybe half-a-dozen essay plans in the time it would take them to complete one essay, and perhaps making the plans involves far more thinking and learning.

5. Reviews

Anyone who reviews books for journals or magazines will confirm that there's no better way of making oneself look deeply into a book than to be charged with the task of writing a review of it! Getting students to write reviews is therefore a logical way of causing them to interact in depth with the information they review.

Advantages

• Reviewing is an active process. Reviewing material gives students a task to do which focuses their thinking, and helps them avoid reading passively.
• Reviews are useful for revision. When students have reviewed material, the reviews are useful learning tools in their own right, and may spare students from having to wade through the material on subsequent revision.
• Reviewing involves important mental processes. When students are required to review material from different sources critically, they are necessarily engaged in higher-level skills of comparing, contrasting and evaluating - far beyond passive reading.

• Reviews are necessarily quite individual. For reviews to lend themselves to assessment, it is important that the task should be delineated quite firmly. This may go against the open-ended approach to reviewing which we may wish students to develop.
• There aren't enough books! With large numbers of students and limited library resources, students may find it difficult or impossible to get adequate access to the materials we want them to review.
• Reviewing individually can be lonely. Reviewing a range of resources is often best done as a group task rather than an individual one, maximising the benefits that students derive from discussion and debate. It then becomes more difficult to assess individual contributions to such reviews.

• Get students to assess existing reviews. For example, issue students with a selection of existing reviews, and ask them to identify features of the best reviews, and faults of the worst ones.
• Decide about credit to be awarded to 'search' tasks. It is useful to get students both to locate all relevant major resources addressing a field, and to prioritise (for example) the most-important or most-relevant half-dozen sources.
• Set a tight word-limit for the review. The art of writing a good, short review is more demanding than writing long reviews. When students' reviews are of equal length, it becomes easier to distinguish the relative quality of their work.
• Think about combining collaborative and individual work. For example, suggest that groups of students do a search collaboratively, and identify the most relevant sources together. Then suggest they write individual reviews of different sources. Finally, consider asking them to share their reviews, then write individual comments comparing and contrasting the sources.

6. Reports

Assessed reports make up at least part of the coursework component of many courses. Report-writing is one of the hardest things when it comes to providing students with general advice, as (for example) the nature of a report in Sociology differs substantially from that for Mechanical Engineering.

Advantages

• Report-writing is a skill relevant to many jobs. In many careers and professions, the ability to put together a convincing and precise report is useful. Report-writing can therefore provide a medium where specific skills relevant to professional activity can be addressed.
• Reports can be the end-product of useful learning activities. For example, the task of writing reports can involve students in research, practical work, analysis of data, prioritising, and many other useful processes. Sometimes these processes are hard or impossible to assess directly, and reports provide secondary evidence that these processes have been involved successfully (or not).
• Report-writing can allow students to display their talents. The fact that students can have more control when they write reports than when they answer exam questions or set coursework allows students to display their individual strengths.

• Collaboration can be difficult to detect. For example with laboratory work, there may be a black market in old reports!
• Report-writing can take a lot of student time. When reports are assessed and count towards final grades, there is the danger that students spend too much time writing reports at the expense of getting to grips with their subject matter in a way which will ensure that they succeed in other forms of assessment such as exams.
• Report-marking can take a lot of staff time. With increased numbers of students, it becomes increasingly difficult to find the time to mark piles of reports and maintain the quality and quantity of feedback given to students about their work.

• Make explicit the assessment criteria for reports. Help students to see the balance between the marks associated with the structure of their reports, and those given to the content and the level of critical thinking and analysis.
• Ask students for full reports less often. For example, if during a course students tackle eight pieces of work involving report writing, ask students to write full reports for only two of these, and ask for summary or 'short-form' or 'memorandum' reports for the remaining assignments. These shorter reports can be structured in note-form or bullet-points, and can still show much of the evidence of the thinking and analysis that students have done.
• Allow students to assess past reports. This can be an effective way of alerting students to what makes a good report, and faults to avoid.
• Accommodate collaboration. One way round the problems of collaboration is to develop approaches where students are required to prepare reports in groups - often closer to real-life than preparing them individually.
• Involve students in assessing each others' reports. When marks for reports 'count' significantly, it may be desirable to moderate student peer-assessment in one way or another, but probably the greatest benefit to derive is that students get a good deal more feedback about their work than hard-pressed staff are able to provide. It is far quicker to moderate student peer-assessment than to mark all the reports from scratch.

7. Practical work

Many areas of study involve practical work, but it is often much more difficult to assess such work in its own right; assessing reports of practical work may only involve measuring the quality of the end-product of the practical work, and not the work itself.

Advantages

• Practical work may be really important. For example, none of us wants to be treated by a surgeon who has the best theoretical knowledge available, but whose practical work is slipshod!
• Employers may need to know how good students' practical work is (and not just how good their reports are.) It is therefore useful to reserve part of our overall assessment for practical skills in such cases.
• Practical work is learning-by-doing. Increasing the significance of practical work by attaching assessment to it helps students approach such work more earnestly and critically.

• It is often difficult to assess practical work in its own right. It is usually much easier to assess the end-point of practical work, rather than the processes and skills involved in their own right.
• It can be difficult to agree on assessment criteria for practical skills. There may be several ways of performing a task well, requiring a range of alternative assessment criteria.
• Students may be inhibited when someone is observing their performance. When doing laboratory work, for example, it can be very distracting to be watched!

• Reserve some marks for the processes. Help students to see that practical work is not just reaching a defined end point, but is about the processes and skills involved in doing so successfully.
• Get students to self-assess how well they undertook tasks. This at least helps students to reflect on their performance in practical work, and since the overall contribution to their final grades of the process side of their work will probably be small, the risk of any error due to over-confidence or under-confidence is well worth the benefits accruing from reflection.
• Ask students to include in their reports "ways I would do the experiment better next time". This encourages students to become more self-aware of how well (or otherwise) they are approaching practical tasks.
• Design the right end products. Sometimes it is possible to design final outcomes which can only be reached when the practical work itself is of high quality. For example, in chemistry, the skills demonstrated in the preparation and refinement of a compound can often be reflected in the purity and amount of the final product.

8. Portfolios

Advantages

• Portfolios tell much more about students. They can contain evidence reflecting a wide range of skills and attributes.
• Portfolios can reflect development. Most other forms of assessment are more like 'snapshots' of particular levels of development.
• Portfolios can relect attitudes and values as well as skills and knowledge. This makes them particularly useful to employers, looking for the 'right kind' of applicants for jobs.

• Portfolios take a lot of looking at! It can take a long time to assess a set of portfolios. The same difficulty extends beyond assessment; even though portfolios may contain material of considerable interest and value to prospective employers, it it still much easier to draw up interview shortlists on the basis of paper qualifications and grades.
• Portfolios are much harder to mark objectively. Because of the individual nature of portfolios, it is harder to decide on a set of assessment criteria which will be equally valid across a diverse set of portfolios.
• The ownership of the evidence can sometimes be in doubt. It may be necessary to couple the assessment of portfolios with some kind or oral assessment or interview to authenticate the origin of the contents of portfolios, particularly when much of the evidence is genuinely based on the outcomes of collaborative work.

• Specify or negotiate intended learning outcomes clearly. Ensure that students have a shared understanding of the level expected of their work.
• Propose a general format for the portfolio. This helps students demonstrate their achievement of the learning outcomes in ways which are more easily assembled.
• Specify or negotiate the nature of the evidence which students should collect. This makes it easier to assess portfolios fairly, as well as more straightforward for students.
• Specify or negotiate the range and extent of the evidence expected from students. This helps students plan the balance of their work effectively, and helps them avoid spending too much time on one part of their portfolio while missing out important details on other parts.
• Don't underestimate the time it takes to assess portfolios. Also don't underestimate their weight and volume if you have a set of them to carry around with you!
• Prepare a pro-forma to help you assess portfolios. It is helpful to be able to tick off the achievement of each learning outcome, and make decisions about the quality of the evidence as you work through a portfolio.
• Use post-its to identify parts of the portfolio you may want to return to. This can save a lot of looking backwards and forwards through a portfolio in search of something you know you've seen in it somewhere!
• Consider using post-its to draft your feedback comments. You can then compose elements of your feedback as you work through the portfolio, instead of having to try to carry it all forward in your mind till you've completed looking at the portfolio.
• Provide interim assessment opportunities. Give candidates the opportunity to receive advice on whether the evidence they are assembling is appropriate.

9. Presentations

Advantages

• There is no doubt whose performance is being assessed. When students give individual presentations, the credit they earn can be duly given to them with confidence.
• Students take presentations quite seriously. The fact that they are preparing for a public performance usually ensures that their research and preparation are addressed well, and therefore they are likely to engage in deep learning about the topic concerned.
• Presentations can also be done as collaborative work. When it is less important to award students individual credit for presentations, the benefits of students working together as teams, preparing and giving presentations, can be realised.

• With large classes, a round of presentations takes a long time. This can be countered by splitting the large class into groups of (say) 20 students, and facilitating peer-assessment of the presentations within each group on the basis of a set of assessment criteria agreed and weighted by the whole class.
• Some students find giving presentations very traumatic! However, it can be argued that the same is true of most forms of assessment, not least traditional exams.
• The evidence is transient. Should an appeal be made, unless the presentations have all been recorded, there may be limited evidence available to reconsider the merit of a particular presentation.
• Presentations can not be anonymous. It can prove difficult to eliminate subjective bias.

• Be clear about the purposes of student presentations. For example the main purpose could be to develop students' skills at giving presentations, or it could be to cause them to do research and reading and improve their subject knowledge. Usually, several such factors may be involved together.
• Get the students to establish a set of assessment criteria for their own presentations. You may be pleasantly surprised how good their criteria are. When students have a sense of ownership of the criteria, they tend to work much harder to achieve them.
• Ensure that the assessment criteria span presentation processes and the content of the presentations sensibly. It can be worth reserving some marks for students' abilities to handle questions after their presentations.
• Consider using student peer-assessment of their presentations. Make up grids using the criteria which have been agreed, allocating each a weighting, and get all of the group to fill in the grids for each presentation. The average peer-assessment mark is likely to be at least as good an estimate of the relative worth of each presentation as would be the view of a single tutor doing the assessment.
• Consider giving students some prior practice at assessing presentations. It is useful, for example, to give students a dry run at applying the assessment criteria they have devised, on one or two presentations on video. The discussion which this produces usually helps to clarify or improve the assessment criteria.

10. Vivas

Advantages

• Vivas are useful checks on the ownership of evidence. It is relatively easy to use a viva to ensure that students are familiar with things that other forms of assessment seem to indicate they have learned well.
• Vivas seem useful when searching for particular things. For example, vivas have long been used to help make decisions about borderline cases in degree classifications.
• Candidates may be examined fairly. With a well-constructed agenda for a viva, a series of candidates may be asked the same questions, and their responses compared and evaluated.

• Some candidates never show themselves well in vivas. Cultural and individual differences can result in some candidates underperforming when asked questions by experts and figures of authority.
• The agenda may 'leak'. When the same series of questions is being posed to a succession of students, it is quite difficult to ensure that candidates who have already been examined aren't able to commune with friends whose turn is still to come.
• The actual agenda covered by a viva is usually narrow. Vivas are seldom good as measures of how well students have learned and understood large parts of the syllabus.

• Prepare the agenda in advance, and with colleagues. It is dangerously easy (and unfair to students) for the agenda to develop during a series of interviews with different students.
• Prepare and use a checklist of pro-forma to keep records. Memory is not sufficient.
• Ensure there are no surprises. Share the agenda with each candidate, and clarify the processes to be used.
• Work with one or more colleague. Divide the agenda of questions, so that there can be an observer at each point as well as a questioner.
• Try to put students at ease at the beginning. For example, ask something very straightforward, or simply chat for a moment or two about something quite irrelevant to the viva.
• De-brief each viva at once. Even if only for a minute or two, it pays to discuss briefly impressions and findings after each interview, and to make brief notes of conclusions.

Conclusions

Moreover, we need to ensure that learning is not simply assessment-driven. It can be argued that presently we have far too much assessment, but that neither the quality nor the diversity of this assessment is right. Students are highly intelligent people; if we confront them with a game where learning is linked to a rigid and monotonous diet of assessment, they will learn according to the rules of that game. To improve their learning, we need to improve our game.

Further reading

Brown, S and Knight, P (1994) Assessing Learners in Higher Education Kogan Page, London.
Brown, S, Gibbs, G and Rust, C (1994) Diversifying Assessment Oxford Centre for Staff Development, Oxford.
Coulson, A (1994) Objective Testing Series 11, Red Guides for Staff, No.4, University of Northumbria at Newcastle.
Race, P and Brown, S (1993) 500 Tips for Tutors Kogan Page, London.
Race, P (1994) Never Mind the Teaching - Feel the Learning SEDA Paper 80, SEDA Publications, Birmingham, UK.

Acknowledgements

Phil Race is grateful to Carole Baume (Oxford Brookes University), Sally Brown (University of Northumbria at Newcastle) and Ivan Moore (University of Ulster) for very useful comments on earlier drafts of this paper.

This article first appeared in The New Academic Vol.4 Issue 3, and is reproduced with permission. The second part can be found in Vol.5 Issue 1. For subscription details, contact Jill Brookes, Administrator, The Staff and Educational Development Association, Gala House, Raglan Road, Edgbaston, Birmingham B5 7RA. Tel. (0121) 440-5021 Fax: (0121) 440-5022.