Graham Gibbs (1)

Who Should Judge Portfolios?

Graham Gibbs, Open University

Primary research evidence has been peer reviewed before it reaches a promotions panel.

When a promotions panel reviews documentation about research excellence, what it sees consists primarily of secondary evidence in the form of lists of published articles and research grants. The primary evidence, the articles or research proposals themselves, have been peer reviewed at an earlier stage - by journal editors and grant awarding bodies. The time and effort put into such prior peer judgment is considerable. Probably hundreds of hours of relatively sophisticated peer review have gone into the 'gatekeeping' which has produced my own list of publications. Some of this peer review greatly improved my publications and some of it made sure the dross remained unpublished. Promotion panels trust this peer review process to a considerable extent and may not even request to see primary evidence. The panel may or may not be sufficiently expert in the applicant's specialisms to make the kinds of judgment an editor might make about primary evidence, but they can judge the standing and credibility of the editors, the peer reviewers, and so make an informed judgment about the collection of secondary research evidence.

Teaching portfolios often consist of primary evidence which has not been peer reviewed before the panel sees it.

In contrast teaching portfolios, as commonly presented, often consist of primary evidence, or reflections upon it, which has not been peer reviewed before the panel sees it. The panel may be in a poor position to make valid judgments about such primary evidence, even if trained to do so. They probably do not have the time to undertake this peer review adequately and usually lack access to such crucial primary sources as classrooms, students and their assignments. What seems necessary if teaching portfolios are to have equivalent credibility, validity and status as lists of publications, is for extensive and rigorous peer review to have taken place before items in the portfolio reach the panel. And ideally this peer review will have taken place over a period of time so as to improve the quality of the evidence and weeded out the weak evidence. The credibility of these peer reviewers will be crucial as will the amount of time and effort put into the reviewing. Developing a comprehensive system of peer review is as important as developing the use of portfolios themselves, as without peer review portfolios cannot be meaningfully judged by panels. This is a massive task, but as Lee Shulman has argued, to be valuable, teaching has to be able to be valued, that is for its value to be judged. Just as junior scholars need to learn how to review articles and senior scholars need to learn how to become editors and chairs of research bodies, so junior teachers need to learn to review peers' teaching and senior teachers need to take on a supervisory quality assurance role in this peer review.