Students' views on graduate standards

James Wisdom, Educational Development and Support Service, London Guildhall University, Calcutta House, Old Castle Street, London E1 7NT; tel/fax 071 320 1096; J_Wisdom@lgu.ac.uk
October 1995

• Introduction
• Student Consultation Meetings
• The Findings from the Student Consultation reports
• The Programme of Meetings Overview of the Findings
• Question 1: What evidence do you use to make judgements about the quality of your course?
• Question 1: (for First Year students) What part did ideas about standards at different institutions and on different courses play while you were applying to and choosing your university course?
• Question 2: Do you think we should have specified standards for all British degrees
• Question 3: If so, what should such standards be? What elements would you include?
• Question 4: How would you define levels such as pass, satisfactory or excellent?

INTRODUCTION

This is a report into students' responses to the ideas being discussed in the Graduate Standards Programme. The enquiry was in two parts - a search through the archive of reports on student consultation meetings held in recent years, and meetings with first and final year students, structured around a formal consultation process.

STUDENT CONSULTATION MEETINGS

Rather than attempt an extensive questionnaire approach to gathering students' views, we decided to use the more qualitative approach of Student Consultation Meetings. These meetings have been shown to be effective methods of determining students' views on their educational experience. Since 1986 the author, with colleagues, has used this approach for reporting on 107 programmes of study at 11 institutions. The 4,300 students interviewed represent an average of 50.2% of those registered for these programmes. This has generated an archive of good quality student feedback which was used in the first instance to identify the likely outcomes of the exercise (seventeen reports were used). Five trial conversations were then attached to the normal programme of meetings, the wording of the questions was refined and finally thirteen meetings were held.

(In this report the term "programme (of study)" will be used to describe the whole collection of units, modules or courses taught by one or few staff and separately assessed which when taken together lead to the award of a degree. Students and many staff use "course" to mean both and during the meetings the usage was clarified when necessary.)

Student Consultation Meetings are a process of gathering student feedback using a knowledgeable but independent and neutral coordinator. They are usually commissioned by the teaching staff of a programme of study and then held with whole year groups of students. The agenda is set by the students around the statement "we are interested in anything which affecting the way you are learning this course (programme of study)". The meeting is run using a particular formal discussion process which ensures the consensus (or clarifies the disagreements) of the whole group, and the event produces a report on which teaching staff can make decisions about the development of the programme. The reports are only put into the public domain by the commissioning staff, often as part of a monitoring or review process. This approach was easily adaptable to the research required for the Graduate Standards Project.

THE FINDINGS FROM THE STUDENT CONSULTATION REPORTS:

The search through 17 reports in response to the question: "How do students appear to define standards?" produced the following six features:

1. In comparisons between units, modules or courses taught primarily by one member of staff; mainly by workload, intellectual pressure and the perceived level of difficulty in assessment.
2. In the relationship between theory and practice; students often comment that a course which is taught with many examples drawn from real life or which incorporates practice is easier to learn than one which is too theoretical; heavily theoretical courses are often described as going too fast, without time to assimilate the material, as having a heavy workload, or as having an overloaded syllabus.
3. By their previous experience; a particular feature of first year students' discussions.
4. By an awareness of what their chosen industry or profession will require; this includes skills development such as group or presentation work.
5. By the quality of teaching; the key words are enthusiasm, empathy, expectations and skills of organisation and planning.
6. By the reputation of the institution or the programme.

THE PROGRAMME OF MEETINGS

Between May and October, 18 meetings were held with 212 students in ten universities, in groups of between 3 and 38. The students came from ten discipline areas. Four of the groups were new to higher education; one group (and part of another) were in their second year; one was a postgraduate group and the rest were in their third or final year. 107 students were in post-1992 universities, 105 came from pre-1992 universities.

The final set of questions was:

1. What evidence do you use to make judgements about the quality of your course?
2. Do you think we should have specified standards for all British degrees?
3. If so, what should such standards be? What elements would you include?
4. How would you define levels such as pass, satisfactory and excellent?

If it was with a first year group, the first question was replaced with:

1A What part did ideas about standards at different institutions and on different courses play while you were applying to and choosing your university course?

OVERVIEW OF THE FINDINGS:

As we worked through the programme of meetings it became clear that a general consensus was emerging from the four questions. The rest of this report will lay out and develop the replies in greater detail, but as an overview it is possible to make the following points with some confidence:

Students claim to use six main ways of measuring the standards on their courses, of which the two most important are their personal development and the quality of the teaching and learning. Although they know what externally verifiable measurements might be, they do not appear to actively use them, though they would rely heavily on the career progress of recent graduates if they could discover it.

Most students welcome the idea of having specified standards, though at first they are not clear exactly what this means. The strongest motive for supporting this idea is to achieve visible equity between what they think ought to be similar experiences. Although amongst students of the post-1992 universities there is a desire to challenge the unfairness of the traditional hierarchy of reputation, the main drive for equity comes from students' perception of what other students might be doing, or not doing, on their courses. Many students expressed concern about the dangers of conformity which the specification of standards might bring.

When asked what those standards might be, most students appear to consider knowledge in terms of quantity rather than quality. They do not appear to have well developed ideas around the words "understanding" and "attainment". When it comes to the specification of the skills appropriate for their subject, students will claim a number of skills (although perhaps some are attitudes and values) which early in the discussions are assumed to be specific. They soon realise that the vast majority, however, are generic and it is then becomes quite hard to define what is specific about studying their chosen subjects. Students also want personal development and effort to be taken into account.

The least successful part of each meeting was the final part, in which students were asked to specify levels. While many groups had the words to characterise "excellence", these discussions did not seem to be as rooted in experience as the other parts of the meetings. It is possible that the ownership of assessment by academics has left students unsure and inexperienced in this area. However the students had ideas about how the process of specification might be achieved.

One theme ran through some of the meetings and is worth highlighting. It was that the central and defining characteristic of a graduate is that she or he is someone who has learned how to learn, and that the specification of standards for this area would be interesting, straightforward, common across all universities and useful for the many stakeholders in higher education. This point was neatly summarised by a final year history (mature) student in a post-1992 university who said (possibly in despair?): "When you start, you think graduation is an end; when you finish, you realise it is only a beginning".

QUESTION 1: What evidence do you use to make judgements about the quality of your course?

In explication of this question the students were told that the issue was not "How good is your course?" but "How do you know how good your course is?" They were asked to consider both the component parts and the overall course. Other prompts were: "How do you decide?" and "What standards do you have in your mind which you use to measure your experience on this degree programme?"

Students answered this in two ways - by dealing both with the courses (modules, units, subject areas leading to individual assessment, subject areas taught be one academic or by a small, identified group of academics) and by dealing with the whole programme of study, the department which administers it and the university in which they were studying.

1 Personal growth and development

This measure frequently appeared in the discussions. Students were measuring both courses and programmes by the changes they were able to estimate in their own personal growth and development. They described looking back over their work since they started and seeing the quality improve - by the increase in their knowledge and the development of their skills. They referred to how much they enjoyed a course or found it interesting. They also referred to how challenging a course might be, how much it encouraged them to go further on and discover things for themselves. In one group, the measure was how open-minded one had become; in another, whether they could talk with confidence to someone about the content of their course.

2 Quality of teaching and learning

This measure also frequently appeared and was described in a number of ways. When students described academic staff who were teaching well, they used such words as "enthusiastic", "thorough", "stimulating" and "accessible". Students welcomed motivated staff who could inspire them and make them want to engage. Although students recognised the academic expertise of their staff (and claimed to use it as a measure of the reputation of their programme) they often stated that teaching and communication skills were more important. Students also valued the quality of communication between them and staff, both in seminars and tutorials (and the amount of time for these could be measured) and in general in their departments (as in time for personal tutorials), as well as in the quality of the feedback on their work; the word "respect" was used in this context.

There was a strong theme in the meetings that one of the measures of quality was the relationship between theory and practice, with students favouring emphasis on the application of the theory they were learning, its relevance to future employment and being taught by staff who have practical experience.

Another measure, common amongst students of art and design who are in a position to see all their fellow students' work, is their personal judgement about its quality.

3 Quality of programme and course administration

Although this topic was mentioned less often, it lay behind many of the ideas which the students were putting. They knew when a course was well planned and structured, and when it was well organised in its delivery. They commented on the connections between the teaching and the coursework, the relationships between the staff in the programme team, and between them and the support staff, and on the management of resources. Students measured quality by how well integrated and complementary were the various elements in their studies. For students on modular programmes, or on linear programmes with choice, the availability of information (especially about content), the ease (and reality) of choice and the amount of control they had over their own study plans were also important.

4 Reputations

Reputation is a difficult subject. Although many groups referred to this as one of the ways they would make decisions about quality, it was hard to find evidence in the meetings that they actually did. Students commented on the reputation of their programme as a whole, determined by such things as contacts with the industry, references in magazines and newspapers, sponsorship from companies and the winning of public competitions. The success of ex-students in getting good jobs was also used as a measure. The reputation of the academic staff was another feature, determined by their research record, publications and how often they updated themselves. Students also claimed to use "research reputation" to determine the reputation of the university, and there was one reference to the use of league tables.

There was plenty of evidence that students discussed their experiences with other students, both within and outside their institutions. When choosing modules or units inside their universities, students used the opinions of those who had completed them, and then sometimes took the number of other students likely to study them into account. When comparing beyond their institutions, students claimed to discuss the content of their courses, the amount of contact time with tutors, the quality of their lecturers, the facilities, space and resources, the quality of the library, and how successful their course was. Students also discussed the difficulty of their course, defined by how much work they had to do and how hard were the exams.

5 By comparison with experience and expectations

Students have had experience of previous programmes and many have had experience of established employment; they measure their work in universities against these features. They also form expectations (from visiting the department, from the prospectus, from documents describing the programme or its individual units) and in many cases have a clear idea of their intended employment and what they will require to prepare for it.

6 By external measurement, assessment and performance indicators

In many meetings students suggested the same set of statistics to establish programme quality, and then made the same reservations. Students had up to four statistics in mind in these discussions: the pass rate (by which they meant the proportion of students passing the assessments and exams) and also a pass rate which showed quality by the proportions of firsts, upper seconds, lower seconds etc; they also used a failure rate (i.e. the number of students who failed or did badly in assessments and examinations) and a drop-out rate (sometimes known as a wastage rate) which described the proportion of students leaving the programme, usually because of failure at assessment or examination. In the conversations there was often confusion between number and proportion.

In no case would students accept that any of these rates could be used as figures on their own. The students thought that the award of many firsts could be because the standard of the programme was low - so it was also essential to see the quality of the work or to know the number of students getting good jobs. Similarly having many students drop out or fail might mean they (the students) were slack or that the programme was too hard - again, a second measure would be essential to make a good decision. Students familiar with modular schemes avowed that their flexibility made traditional drop-out rates unreliable.

Students thought that more reliable information could be derived from class and seminar size, by the entry qualifications of those on the programme of study and by the level of demand for it (as in first choice applications). The external evidence most frequently proposed by students was how many graduates got jobs from the programme and what happened to them - what progress they made in their careers.

QUESTION 1 (for First Year students): What part did ideas about standards at different institutions and on different courses play while you were applying to and choosing your university course?

51 students were asked this question directly in three meetings and from this it appears there were two key features in students minds when they were making their decisions. One was finding out in advance as much as possible about the nature of the programme, particularly in establishing how broad it was, what subjects were taught, how much choice was involved and how easy it was to change once they had started. The other was picking up on the reputation of the university - whether it was an ex-polytechnic, how well known it was, what quality of facilities it offered - and then visiting to find out if it was friendly. The one statistic which apparently would have influenced students is the success of employment after graduation.

QUESTION 2: Do you think we should have specified standards for all British degrees?

Only a few students thought that we should not have specified standards for all British degrees. Although the questions sounded like it was inviting a simple yes/no answer, many students offered reasons for their replies, and these are given below.

1 Yes - to challenge the perceptions of ranking between universities

A number of groups thought the specification of standards would act as a counter-weight to the assumptions (reputation, snobbery, élitism) which currently exist and which are perceived to unfairly benefit some graduates on the basis of unsubstantiated notional rank orderings. It would in particular assist the ex-polytechnics to become fully accepted. In a meeting in a "Robbins" university a small number of students wished to retain the current approach, perceiving themselves to be its beneficiaries.

However there was widespread scepticism that significant changes would follow from such a development, and certainly not rapidly. One reason was held to be tradition. The other was the likely effect of any inspection system that would follow from the specification of standards in which it was assumed that, while all universities would be expected to achieve them, some would achieve them more successfully than others. This would in turn produce new league tables which would affect of selection of applicants and thus over time would become as fixed as the present ones.

2 Yes - to inform employers

In some cases students thought the specification of standards would benefit them in their relationship with employers in a number of ways: by the recognition of transferable skills, by developing a more accurate perception of what a degree meant and was worth, and by enabling a candidate to be selected on personality (their knowledge having been taken for granted). One group developed the idea that as there was always going to be some training when they were taken on, the important element was in showing you had learned how to learn.

3 Yes - to get comparability with other subjects

In two areas students strongly supported any process which brought greater comparability between subjects. Some students had chosen to study on modular degrees and felt the modules should be "equal". It would also enable easier transfer between courses. Also it would reduce the possibility of confusion arising from a modular approach which allowed the fragmentation of named degrees. Finally they had realised that, as each student in a modular university can do a different degree, there is no longer a traditional basis for comparison.

The other commonly held belief was that, in other parts of any university, other students were getting away with a lighter load or were studying subjects (or combinations of subjects) which were not "proper". It was not difficult to find a lack of respect for other students' efforts. Many students seem to know of others who are following "mickey mouse" subjects - in one discussion we had the notion of a degree in "Catherine Cookson" because there has been references in the tabloid press to some classes just before the fieldwork. Most Engineering students seem to know someone who shares a flat with Arts students who are exhausted from their four hours of class contact a week. Interviews with students of Fine Art revealed the obverse - that a fully specified degree would help dispel the notion that their work is neither difficult nor proper. It may be that the particularism of staff, reinforced by a subject-dominated external examiner system, find echoes in the students' occasional lack of comprehension of the value of studying subjects other than their own.

4 Yes - For international comparison

Some students thought that national standards should be comparable internationally to enable them to compete effectively. In the course of the meetings, students from Spain, Italy and Germany made similar points - that in their countries the universities used mass lectures, hard exams and very high pass marks, all of which lead to widespread and frequent experience of failure. The British appeared to do two things differently - firstly, they attempted to teach their students; secondly, they bothered to define the levels of success in that whole area between their pass mark (typically 40%) and the European pass mark (typically 80% - 90%). These (few) European students also thought that the standard of excellence was much the same between their friends at home and the best in their British classes.

5 No - Danger of conformity

Many of the groups detected dangers in the specification approach, using words such as "conformity", "uniformity" and "too much standardisation". One, in distaste, likened it to the National Curriculum. The risk of occasional incompetence was seen as a lesser danger. There was a common belief that each university and department should develop its own identity, standards, specialisation and reputation - nothing should inhibit the striving for excellence. One group thought that the creative skill of teaching was seen as diverse and most at risk from the setting of standards.

6 No - Not possible

In some meetings the quality of undergraduate life - a social life away from home independently learning such "life skills" as cooking and handling human relationships - was regarded as being of similar importance to the learning of subject knowledge. Any attempt to specify this was deemed to be impossible, and could not span the differences between life in an Oxford college, coping alone in a room in a poor northern town, or living at home.

QUESTION THREE: If so, what should such standards be? What elements would you include?

A further question which helped to improve the discussion was: "What mixture of general and specific elements would you put together to define a graduate of your subject?"

1 Effort

Students saw their years as an undergraduate as evidence of a major personal commitment to a sustained period of hard work and therefore assumed that equity of effort would be a feature of any specification. This effort was characterised in a number of ways - attendance at lectures or in studios, workshops etc; amounts of coursework and assessment; amounts of examination; and time spent on private study.

2 Skills, Values and Attitudes

When asked directly, all groups of students were able to specify a range of skills which they felt were essential to the study of their particular subject, and which should be mastered by anyone claiming graduate status in that subject. In every case almost all the skills the students specified were generic, although in most cases they were sure their subject had a special, particular and exclusive claim to them. The specific skills were often only one amongst lists of up to ten. It is possible that further exploration would reveal subject-specific teaching and learning approaches which transform what appear to be generic skills into specific skills and which might explain why they are so often seen as particular.

When asked for appropriate skills, students also offered a range of personal characteristics, values and attitudes which they were expecting to develop during their studies. During the fieldwork we did not debate with students the differences between these, though we did ask if or how levels of performance could be distinguished. The students' lists are similar to those which have been discussed within the Enterprise and Capability initiatives. Many of the groups explained the importance of numeracy skills, communication and presentation skills, the successful use of information technology and the research skills implicit in the best use of a library.

The range of personal skills and qualities which the students offered as defining a graduate was very long, and unlikely to be found in one person. Students valued inter-personal skills such as being good at team work and having the ability to work with others. They also valued the more individual attributes of self-motivation, the ability to work hard, to use one's initiative, to help oneself and not wait for others to assist, and being adaptable. They thought very highly of good time management and being able, through organisation, planning, performance and efficiency, to get the job done on time. They recognised the importance of such social skills as knowing how to live with someone and being able to compromise; one group valued the contribution students made to other students' learning. Being confident and able to handle pressure was thought to be important, as were the skills of self-criticism and reflection.

The specific skills which were mentioned in these parts of the discussions were such things as: (for science students) competence to work in a laboratory, with appropriate lab skills and knowledge of procedures and equipment; (for physics students) the ability to apply maths to physical situations; (for engineering students) being able to model events mathematically; (for design students) communicating in visual terms, or through model making, and being able to explain design decisions.

3 Academic attainment

It was clear from the students' replies to this question that many of them carried a concept of academic attainment, but that they found it hard to define. In many cases the core of the definition lay in the quantity of knowledge which they had learned. If the definition was developed, it was in the direction of having an ability to do something with that knowledge, an ability to use it in whatever way was necessary.

In the discussions about skills the students suggested a number of desirable attributes which defined a graduate. Those which might be seen as related to "academic attainment" are offered here, selected from what were sometimes long lists of skills and qualities. A graduate was defined as someone with a well developed critical, sceptical and analytical approach, with the ability to originate and develop ideas and draw their own conclusions.

A graduate would be independent, with a logical mind and sound judgement. The ability to criticize and argue was also important. Finally a graduate was a creative person, able to find a problem and to go about solving it, someone who can spot the essential in a mass of data.

4 Teaching quality

In various meetings the students made modest suggestions about how one might define teaching quality to ensure the standards were achieved. The indicators they would be looking for were the educational and formal qualification levels of the staff, their teaching ability, their enthusiasm and interest, the staff student ratio (too large a group being overwhelming, too small offering insufficient diversity of opinion) and a specified minimum staff contact time for all courses - 10 hrs per week was suggested.

5 Personal development

Although students showed some enthusiasm for standardisation of processes, many of the groups discussed how personal development was the central experience of studying a subject to degree level.

The word that kept returning in this part of the discussions was "challenge", that all students should experience the self-improvement, growth and development that comes from being challenged, no matter what their starting ability and competence. Any external specification would have to encompass both a minimum and a maximum for something as varied and different as personal development.

On a few occasions the students suggested that no system could make decisions about personal development and that if any external measure was needed it should be in the hands of employers.

When asked what it meant to be described as a graduate, groups of students described the effort, motivation, dedication and perseverance which they thought was implicit in the name. But the sentence which returned in this part of the discussions was that graduates have learned to learn. One student summarised the dynamic in this way: "When you start, you think graduation is an end; when you finish, you realise it is only a beginning".

6 How such a specification should be done

Once students had seen that the discussion was opening up the possibility of some nationwide process of specifying the sorts of skills, attitudes, values and academic attainments which they though were so important to them as undergraduates, they were ingenious in describing how such a process might or should be conducted, or what conditions would need to apply to ensure success in the process.

Students would expect common national standards of marking, with the rules of assessment declared and degree classifications defined. They claimed this would ensure that all modules and courses had equivalent assignment and assessment procedures, that they would be of equal difficulty and be graded in similar ways. Well planned and well organised programme, course and unit handbooks would be compulsory, and prospectus details would be expected to be reliable. Students would expect courses, semesters and units to be the same length. As standards, course organisation and course content all went together, students thought it would be impossible to develop standards and content unless good quality organisation was in place.

While some students could see the advantage of a standard (or at least a minimum specification) syllabus for each module, others saw such a narrow specification as bringing danger in the long run. They thought syllabuses would have to be drawn so widely there might not be any advantage in the work. But many students saw the opportunity presented by modular schemes - which would allow common core modules with specialised options - as a possible way forward.

The sorts of tests which students suggested for the effectiveness of national specifications were that the level of comprehension should be the same, i.e. that people at one university should be able to understand what is being learnt at other universities; or that students might take exams from different places. They felt that if a student from one university should get a first they would get the same award at another university. As students were wary that universities might enhance quality and reputation by rejecting them during the programme (and thus ensuring high pass rates), they thought that the numbers of entrants should be monitored against both drop out and pass rates.

Other suggestions for running such a process were the equivalent of professional bodies drawing up national regulations in each subject, or an independent body researching the current hierarchy, or the equivalent of an ombudsman or parliamentary commission checking on standards in each institution.

QUESTION FOUR: How would you define levels such as pass, satisfactory or excellent?

Definitions of levels

This section of the meetings was the hardest for the students to work through satisfactorily. Although focused only on the three grades of pass, satisfactory and excellent, all seemed to find it hard to articulate a response that did justice to the rich complexity of learning, though they found it straightforward to measure knowledge in terms of quantity. This may have been a symptom of the absence in many higher education courses of a dialogue between staff and students designed to enhance students' understanding of the assessment process. In some discussions there were examples of a deep scepticism about the immutability of standards, about the nature of standards as a human construct, and about the whole process being cast in the form of a game with complex and unknowable rules - certainly unknowable to mere students. When it was discussed, students thought it was sensible for lecturers to specify assessment criteria, not so much because they would then know exactly what to work to, but because it would help clear up ambiguities in lecturers' minds.

Pass

Students viewed the pass as no more than an answer to a question, written well enough to be understood, showing a basic understanding but involving little effort.

Satisfactory

A satisfactory level of work was characterised with words such as "initiative" and "understanding", "expanded and reasoned arguments" and "wide reading". It would show evidence of having worked hard, and be well structured, using good language, grammar, punctuation and spelling.

Excellent

The students were able to describe excellent work with a much wider range of characterisation than for the other categories. The notion of a full, good or deep understanding was often linked to the ability to apply or use the knowledge. A second common descriptor was in the breadth and independence of reading and research. Originality and independence of thought were often linked. Fluency of expression, the clarity and flow of one's ideas, and the structure and presentation of ones' work were also commonly mentioned.

Sets of replies

In many cases the students were more comfortable in building their responses to the question about levels in a three-tiered layer of progressive quantity or sophistication. An example from Design students was : "The lowest level is just achieving the criteria; then doing more; up to a maximum; excellence is taking it so much further there is no room to go any further." A similar set from Education students was cast in terms of the amount of support required to be able to teach: "Ability to teach with support - pass; with less support - satisfactory; with very minimum support or without support - excellent". A third group suggested the bare minimum for a pass, with something (such as understanding) for a satisfactory and the ability to apply and use the knowledge for excellence. In many cases the grading was a reflection of how much of the student there was in the work - repeating knowledge back to lecturers was at one end and being original and creative was at the other.

Personal development

While students were prepared to set out definitions of external measures of the three levels, they also felt that the whole area of personal development should be included within any specification. This led to them to struggle with problems of personal performance and change which have external manifestations which are themselves unreliable as indicators. They thought that such elements as time in studying, effort and the amount of work should be included. Features deemed to be matters of personal opinion - such as confidence - were difficult but necessary, and instead of just having a grade, they suggested a record or profile of where in the programme someone had excelled and where they had done averagely - this was thought to be useful for an employer. In one meeting the students said that it was important that this was not seen as a judgement of the person, only of their performance at a particular moment.

In one meeting the view was put that a programme was good because it was not all based on examination, that it was good to be assessed widely. This was contrasted with what was assumed to be a stereotype of the pre-1992 universities which emphasised the measurement of the quantity of knowledge in examination - their students might indeed have developed skills, but they would not have had them measured.

A particularly interesting and valuable discussion was held with one group, which had its echoes in many other meetings. It was an exploration of what would be required of a university which was overtly emphasising that the central purpose of studying for a degree was to learn how to learn, to equip its students with the tools with which to learn. The students suggested that in such a regime the following personal characteristics would be developed: Enthusiasm; humility; preparedness to change; independence; perseverance; self-confidence; imagination; attitude.