Changing Assessment to Improve Learning

Summary of the final interactive keynote session led by Professor Phil Race at the 1st Northumbria Assessment Conference, UK, 1996.

Introduction

This paper is a summary of the main points given by the presenter, and - more important - a transcript of the contributions made on post-it notes in response to questions posed to the audience. The breadth and diversity of experience among delegates is reflected in the wide range of constructive suggestions they came up with regarding (1) changing exams, (2) changing continuous assessment formats and (3) their own individual action plans resulting from their thinking and experience during the conference.

The intended outcomes of the session were presented as follows:

• to explore how assessment provides a driving force for learning.
• to work out ways of making better links between assessment and learning.
• to collect and collate action points planned by Conference delegates.

Summary of the presentation

First, the analogue of 'assessment being the engine that drives learning' was explored with the audience, some of the symptoms of engine-failure being pictured as follows.

The engine is labouring....

• greater load (for example, larger class sizes)
• steeper hills (shorter time-spans due to semesterisation, modularisation, or both)
• scary speeds (the need to cover syllabus content more rapidly)
• more traffic-lights (short-term planning, policy changes, funding uncertainties)
• tired drivers (increased workload, decreased morale
• more oil needed (funding!)
• better maps needed (continuity and direction need improving)

The drivers are struggling....

• less training (less time and energy - and funding - for educational development)
• sudden bends (short-term crisis management in universities)
• rapid changeovers (new practices being introduced without adequate preparation)
• greater competition (between institutions, between staff, and between students)
• fewer prizes (less rewards for best practice in teaching or assessment)
• more unsigned crossroads (greater uncertainty of purpose or rationale)
• fewer breaks (changed contracts, increased administrative pressure)

Factors leading to successful learning

Delegates were reminded of four factors crucial to ensure effective learning by students:

• wanting to learn (motivation, commitment)
• learning-by-doing (practice, learning from mistakes, trial and error
• learning through feedback (other people's comments)
• making sense of ('digesting' what is being learned.

Concerns about the learning-assessment gap

The presenter than went through ten concerns about traditional unseen written exams, drawing out how such assessment was often at odds with successful learning. Delegates were invited to write on post-its ways that they believed exams could be changed to improve the learning payoff associated with them, or alternative forms of exam which could lead to better measurement of students' competences, knowledge and higher skills. Then, a similar exploration was given of the conflicts often evident between continuous assessment and successful learning, with delegates once again invited to write suggestions for how such assessment could be made more conducive to learning. In the final task, delegates were asked to write down their action plans resulting from the conference, in terms of 'something they could actually do, without having to get permission or backing.' The remainder of this paper is based mainly on their replies, with the task briefings, and the concerns presented to the audience, listed for each of the three tasks. Although there is quite a lot of repetition in the recommendations made by delegates, it is useful to present this without editing, so that readers may gain a sense of how widespread or how localised is each of the ideas involved.

The 'concerns' presented to the audience are given below.

Ten concerns about unseen written exams :

1. They don't do much to increase the want to learn.
2. They are not ideal occasions for learning by doing.
3. The amount of feedback that students receive is not optimal.
4. They don't do much to help students make sense of what they have learned.
5. We mark them in a rush.
6. We're often tired and bored when we mark them.
7. We're not good at marking them objectively.
8. They tend to favour candidates who happen to be skilled at doing exams.
9. They force students into surface learning, and into rapidly clearing their minds of previous knowledge when preparing for the next exam.
10. There are many important qualities which are not tested by traditional exams.

Some quotations from delegates post-its:

• "they only test what a student has learned, i.e. which can be written down. They say nothing about potential or ability, which is often the purpose for which they are used (A-level and in University). They are therefore not fit for purpose."
• "they do not allow sufficient opportunities to discuss personal experiences, particularly on vocational courses."
• "time constraints often affect the quality of the response."
• "they can lead to valuable teaching/learning time having to be used for revision sessions."

• (many replies) give students feedback.
• (many replies) use more 'seen' papers.
• (many replies) help students develop their exam technique.
• make explicit the nature and demands of exams from the very beginning of the course.
• involve students in marking.
• allow students marks for plans of their answers.
• distribute a marking grid to students, so that they will know how the marks will be distributed.
• set questions based on realistic problems.
• write, discuss and improve criteria and marking schemes, relating them to each individual question.
• in a traditional exam system, allow students the choice to speak their learning as well as, or instead of, writing it.
• use written exams to test the ability to write a coherent, sustained, reasoned argument.
• train teachers to write more-effective questions.
• use some questions that are peer-assessed.
• use exams as formative assessment too - give feedback on performance as preparation for further exams.
• make exams short enough to enable marking with ample feedback.
• design questions which examine students' ability to think/argue/analyse, rather than recall.
• provide greater choice of questions, no time limit.
• count marking time as part of lecturers' workload.
• design take-away exams.
• make exams less nerve-racking.
• have sweets/chewing gum available in the exam room.
• hold exams earlier in the semester, so the papers can be returned and discussed.
• take more care designing exam timetables.
• concede that they are so widely used!! Ask what needs are they meeting?
• use an exam as one form of assessment within a range, rather than the only means of assessment.
• remove time constraint.
• discuss subject information to be covered in the exam in advance.
• give students exam technique preparation.
• allow students to see their answers (essays) then discuss the quality of these with the class in detail.
• use (some) seen questions.
• use case study type questions which require analysis and application, not just memory.
• use (some) multiple-choice questions, but still require students to construct their own written answers to others.
• remove time pressure, but think ahead about logistical problems this will cause.
• ensure that the students know in advance that they will have to write about, and the criteria which will be used to assess their answers.
• narrow down what is being assessed by the exam and communicate this to the candidates.
• make exams more realistic and practical, even if this means using simulations and IT.
• tell students what one (or more) question will be about (not exactly).
• give feedback to failed students.
• design unseen papers to require a synoptic synthesis of what students have learned.
• use part-seen or whole-seen papers, so students know the areas being assessed.
• use previously-seen case-studies with unseen questions.
• use problem-focused open book exams, to test application.
• design questions that test more than just recall.
• provide an open and effective marking scheme, and return marked papers (allowing students to check their work against the marking scheme).
• have exams structured around a case-study approach which 'covers' breadth of content and requires application of knowledge.
• when exams are required due to pressure from a professional body, provide proper feedback to students.
• evaluate change in the person, not just knowledge and skills.
• allow students to bring in concept diagrams for each topic, relating to case-studies handed out prior to the exam. This helps students structure their knowledge, rather than depending on memory recall).
• use questions which are broken down into small parts, with objective marking, clear criteria, and model answers.
• disaggregate exams so that each part follows a period of learning, and is integrated with it, and provides helpful and prompt feedback on the learning.
• provide advice from examiner to students, on one sheet of A4 along with question paper, suggesting (e.g.) 'you might like to consider...'
• set in-tray type problems, that involve manipulation and processing as well as recall.
• get rid of as many exams as possible.
• include questions from all parts of the syllabus.
• get the students to write the examination paper.
• have groupwork solutions to exam questions, e.g. 3 students work out answer together, to deepen grasp.
• move away from 3-hour, 3-essay exams.

Ten concerns about continuous assessment

1. If students are under too much pressure, the want to learn is damaged.
2. The range of learning-by-doing may be too narrow.
3. Feedback may be eclipsed by marks or grades.
4. Students may not have the opportunity to make sense of the feedback they receive.
5. It may be hard to detect unwanted collaboration.
6. Too much time may be involved in marking.
7. Students may not be aware of the criteria used to assess their work.
8. Students may get the balance wrong between continuous assessment and exams.
9. Learning may become driven by assessment, and students may only do those things that are assessed.
10. Too little use may be made of the learning that can be achieved when students assess their own - and each others' - work.

• "some argue that standards fall because of continuous assessment."
• "life is continuously assessed: prepare students for it."
• "there is a need to unscramble received beliefs about the purposes of assessment, and the demands of assessment (e.g. meaning of the term 'argument')."
• "make assessment so valuable that it is not a problem that it drives learning."

• make sure there is not too much!
• coordinate deadlines.
• avoid rush of assessments towards the end of a module.
• use a varied mixture of formats.
• use more peer-marking.
• use more self-assessment.
• involve learners in the design of continuous assessment, so that they understand the purposes of it and develop ownership of it.

• devise assignments which give opportunities for active learning.
• give 'real action' tasks, not 'copy out the text' essays.
• make it more formative, by providing early or interim feedback with opportunity for redemption.
• clarify standards by providing examples from past years of excellent-middle-poor.
• suggest to students how many hours/resources etc. they should expect to spend to achieve an A, a B or a C grade, along with the criteria relating to each grade.
• involve students in deciding the best way of assessing, to achieve what they hope/should/need-to learn through the assessment, as well as demonstrating their learning.
• use staged assessment, where students get feedback on early drafts. This also makes marking easier.
• get students to write their own reflective commentary after they've got an assignment back with feedback.
• ensure that criteria for assessment are available (written down) and discussed, to ensure clarity of purpose.
• give out an overall plan of assessment for the entire module/programme, with indications about time management.
• allow students to select their best assessments, e.g. which 4 out of 5 to count.
• allow students to learn about assessment criteria by being assessors, integrating assessment into the learning.
• allow variety and choice to minimise the risk of undetected collaboration.
• scale down the size of assessment tasks so they are manageable in a short period (e.g. 48 hours), but have more of them and provide specific feedback, so building a picture of student development from smaller, more-frequent tasks.
• make assessment central to the module, i.e. work on it during class time and students' own time and for all weeks.
• use integrated assessment across modules, so that students develop more meaningful understanding of modular subjects.
• maintain assessment profiles, to show how much students have developed.
• use a wide variety of strategies to build up a picture of each student - not one essay, etc.
• provide the feedback before the grades.
• show examples of good (and poor) work to students.
• encourage collaboration and sharing of resources using timetabled, student-led workshops.
• encourage networking with agencies in the 'real world.'
• ensure that assessment strategy for the whole course is continually reviewed and revised (not just unit by unit).
• use real case-studies or projects as the basis for continuous assessment.
• get the students to give you feedback on your feedback.
• use self- and peer-assessment screens before tutor-marking, where the students have criteria which are most relevant to the exercise concerned.
• provide some choice in the presentation method, through negotiation, either with the group, or through learning contracts for individuals.
• clearly relate assessment to outcomes.
• stagger submission/completion times.
• provide more examples of ways in which evidence (for a portfolio) could be collected, for a selection of unit learning outcomes. Raising the 'choice profile' of evidence will be encouraging and increase the want to learn.
• avoid overload conflicts with other subjects/units/modules.
• encourage students to value feedback, as opposed to grade.
• diets of continuous assessment must be controlled and scheduled.
• involve students in negotiation of tasks, and aspects of their assessment, such as criteria.
• whole course/year approaches need to be taken to assessment planning and management.
• make continuous assessment more responsive to student needs.
• use peer- and self-review and assessment.
• use group and team work.
• get the balance right: set appropriate tasks giving time for digestion.
• use performance-based assessment (i.e. in class) so that students cease to view assessment as separate from learning.
• measure evidence which is realistic and sufficient.
• ensure that feedback enables learning, and motivates.
• make assessment a two-way process.
• ensure that assessment highlights what needs to be learned as well as what has been done already.

As the final task for the audience, delegates were asked to crystallise their own individual action plans as a result of the conference, according to the following briefing:

Their responses are transcribed below.

Delegates Intentions, and how these will improve learning

• tackle issue of self-assessment, having reflected on this during the conference; applied to appraising performance in a Fine Art modular programme of professional studies, work placement.
  - makes students fully aware of the opportunity of professional studies for them, and a fairer way of assessing their performance using negotiated - and therefore understood - criteria.
• In teachers courses make them formulate their own learning objectives in their own courses. They often lack words to reflect on their teaching.
  - a reflecting teacher will have a better ground to make changes to benefit student learning.
• To involve students in the assessment process;
  - it will create a commitment on the part of the students; ownership of their assessment; motivation.
• Give more flexibility to students about what goes into a portfolio.
  - make portfolio compilation more about learning than about being assessed.
• I am a staff educator. Group projects are causing concern for staff and students. I would like to: prepare a document which outlines some questions that tutors should answer before they decide to use Group Project and some examples of types of criteria to use for participation.
• Follow up idea of personal formulary influence; follow up 'mastery program'; follow up on self-assessment references, read the papers and my notes.
• Get students to set criteria for at least one essay test.
  - hopefully students will feel more in control of their situation and their learning.
• Separate comments/feedback from grades; use self-assessment.
• Discuss with person responsible for 'writing skills training' (important part of year 1) to consider possibilities of peer-assessment as learning-by-critical-analysis of own and others' work.
  - learning by critical analysis of own and others' work.
• Develop record of achievement within the present student portfolio, and think about using it as assessment, formatively and summatively.
  - hopefully will help to integrate theory to practice and show that transferable skills are important and being assessed.
• Include two sets of peer assessment (rather than one) one part-way through group project as well as at the end.
  - feedback on their contribution from peers; and help them to remember who did early work (maybe the ideas person) as well as work towards the end. This will improve fairness of the marks.
• Produce a grid reconciling individual contribution to a group project. The grid will have to be completed twice in a semester.
  - will, I hope, make the criteria of marking individual contributions more explicit.
• Pass on to a colleague various ideas culled from this conference, to feed into staff development activities.
  - (the effect on learners will) inevitable be indirect and long-term rather than short term.
• Carry out peer review of essay prior to assessment. Negotiation of mark? Cut back on my input, moving from content towards process.
• Use the first week of the module to discuss with students why they are on the module and what they hope to achieve (learning and assessment), and how they are going to get there.
  - they will better understand the module objectives and assessment criteria. It will improve their motivation and their sense of security.
• Involve the students in the assessment process of continuous assessment, eg as self- and/or peer-assessment.
  - they will understand better what they need to learn, and be able to devise ways of achieving this.
• More formative assessment, with better feedback. Use self- and peer-assessment. Improve dialogue.
• Numeracy skills: don't APL (or APEL) the good students, but get them to help develop the skills of weaker students. [Comment from Phil: I suggest continue to give students the credit for things they are already good at, but maybe on condition that they prove this by helping weaker students as suggested].
• Do a peer review process for presentations which at present are not assessed at all.
  - involve students in assessing performance by others, to highlight the complexity of presentations as a vehicle of communication. Reduce anxiety level, leading to greater learning.
• Give feedback at a very early stage, as part of the learning cycle.
• Introduce portfolios. Spend a full session to talk to students about assessment criteria.
  - portfolio will improve opportunity to digest feedback, and allow students to see for themselves how they've developed, hopefully giving them more motivation.
• Incorporate peer-assessment of each others' work.
  - supports weaker students; improves opportunity for better students to explain their work, helps students to see more than one approach to the question.
• Intention to use self-assessment for MBA assignment (absolute heresy!)
  - gives some feedback before examination (timetable does not permit this otherwise).
• Start to draw together a database of innovative assessment methods used within my institution.
  - improve student learning through dissemination of good ideas and practices.
• Revisit my presentation on portfolios for personal development, and write a section on reflective learning, including many concepts discussed in presentations at the conference.
  - this will enhance a new approach to workplace personal development, through self-managed approaches through portfolios; it will encourage learning through doing the job, and encourage professionalism and excellence in those who want to take part.
• Produce a booklet for students will all the assessment for the semester in, and criteria; negotiate with tutors reluctant to provide criteria; discuss the course assessment strategy with course team and student representatives.
• Find out more about software which allows students to browse 'answer gardens.'
• Look and try to identify where the responsibilities lie for organising and managing various aspects of assessment (accountability).
• In one subject, two things: reduce the number of assessment pieces (from three); redesign the assessment tasks towards better integration with the objectives of the unit, and of the teaching and learning process.
• Review the ASSHE Projects which involve using IT, and identify areas not included in these projects.
  - encourage more people to use IT to support assessment; this might improve the range of assessment methods used by academic staff in SHEFC initiatives.
• I have attended this conference to become informed of the current issues and debates surrounding assessment, so I can impart this knowledge through the educational technologists within out Department.
• Ensure marking criteria have been discussed in full, and if possible negotiated with students.
  - students will hopefully feel part of their assessment process, and will therefore want to participate.
• Discuss assessment criteria with students, extensively and openly; Incorporate ideas on innovative assessment in course for new lecturers.
  - will improve student learning by relating assessment more directly to their experience.
• Begin peer-assessment in year 1 of course (year 3 at present) when it will be accepted as 'the way to do things' and develop expertise in doing this before it begins to 'count' to degree classes (year 3).
• Unseen exam (year 3) 'Managing Organisations in a Changing Environment' ask students to focus all their answers on one theme, eg on how 'managing' contributes to the topic.
  - relates learning and doing to the key theme of the module.
• Involve students in specifying criteria for marking a specific assignment, and make peer-assessment a component of this.
  - reflection on the purpose of the assignment; knowledge of what others have done; better understanding of the process of assessment.
• Provide oral feedback to students, to expand/explain written comments; give more detailed formative feedback - real evidence as to how to overcome problems identified by assessment (eg chapters to read, activities to try).
  - this improves learning by allowing students to clearly identify needs and how to address them.
• Put assessment jargon into students' own words, and allow them to talk about it (not just to us, but to each other); let people know (staff and students) what students say and think (switch the lights onto the target).
  - let students see what the assessment target is, so they can learn in appropriate ways; improve communication, diminish the gap between 'us' and 'them.'
• Inform colleagues of benefits of innovative assessment (both to students and to academic staff); update my own learning log.
  - help students to approach their learning with less bias towards an end-of-semester grade.
• Provide students with written feedback comments on assignments, before showing them their marks.
  - students should take on board feedback to a greater extent.
• Course management: audit assessment to identify different types used in the course, together with alternative strategies. Share with all lecturers and encourage moves towards a greater degree of variety and relevance.
  - morals: a change is as good as a rest: variety is the spice of life.
• Diversify assessment (not sure exactly how yet - oral presentation? Individual/group?) away from two essays per semester.
  - relieve the stress early in the programme; provide more-useful feedback to students.
• Create a better assessment structure and criteria to encourage deep learning from essay work.
• Use self-assessment with my own class; use a record of achievement with my own class.
• Inculcate a culture for self-assessment (for engineers).
• Propose bank of standard feedback comments, related to criteria and outcomes, to be used by (all?) staff (discussed, revised, negotiated....).
  - help students to understand criteria and improve learning and performance - and speed up feedback - without increasing the workload of staff.
• Encourage staff to create opportunities for students to share assignments and assessments.
• Use funding from the ASSHE Project to expand the repertoire of assessment methods that can be used with students.
  - variety of assessment methods to enliven their learning processes.

Conclusions

It is clear from the wealth of ideas expressed by delegates, and the emotive power often shown in the words they chose to express their views, that there is a groundswell of feeling that assessment needs to be changed to improve learning, and that many of the topics that the conference addressed had spurred delegates to make far-reaching resolutions regarding their own practice. It is to be hoped that as these plans and recommendations are put into practice, the gap between successful learning and fair assessment will be narrowed.