R.D. Gregory, G. Harland and L. Thorley

Using a student experience questionnaire for improving teaching and learning

R. D. Gregory, Dr G. Harland and L. Thorley (University of Hertfordshire)

Reproduced with permission from Gibbs, G. (ed.) Improving Student Learning - Through Assessment and Evaluation. Oxford: Oxford Centre for Staff Development (1995)

Synopsis

This chapter describes the use made of a Student Experience Questionnaire in the School of Engineering at the University of Hertfordshire over the past two years in attempting to improve the quality of the student educational experience. The results of a survey conducted in 1993, already reported, and a new 1994 survey are compared. The process of investigation of the results involving both students and staff is described. Conclusions are drawn on the schemes of study surveyed and the usefulness of such questionnaires in assisting to improve the student learning is discussed.

Background

This chapter describes the process which has evolved in the School of Engineering at the University of Hertfordshire in attempting to use the results from the Ramsden Student Experience Questionnaire (Ramsden, 1990) to improve the learning experience of students. The questionnaire identifies five elements of the learning experience which are scored on a scale of 1-5 (i.e. Good Teaching, Clear Goals and Standards, Appropriate Workload, Appropriate Assessment and Student Independence). It has been used in surveys by the school over the past two years.

The first survey was carried out in May 1993 and reported in September 1993 (Gregory et al., 1993). A number of trends were identified for further investigation and the report concluded that priority should be given to investigating the apparent high student workload.

The trends and conclusion were generally consistent with anecdotal evidence and were - supported by the students. The survey results were also used by each Scheme Tutor as part of their annual monitoring and evaluation report to the School Academic Committee. The report on the questionnaire results was considered in the autumn 1993 term and the School Academic Committee agreed that:

1. the survey should be repeated in the spring term of 1994;
2. the reasons for the poor scores for Appropriate Workload should be investigated by the Scheme Tutors;
3. the results and recommendations for further action should be reported back;
4. the Scheme Committees should consider the implications of all the results and discussions

The second survey was carried out at a similar time of the year as the first; there were more schemes involved and the whole process took longer (i.e. most of the 1994 spring term). The survey this time covered the majority of the schemes of study in the school; more than in the first exercise. In the 1994 survey a total of 917 replies were received from 36 'cohorts' (e.g. first-year Civil Engineering BEng) representing a total 1512 students.

Joint Student-Staff Seminar

Student involvement is seen as a key to real improvements in teaching and learning. The students are currently involved in School Committees through elected representatives for each year of each scheme. These representatives serve on Scheme Committees and form the School Student Council. The Chair of the Council works closely with the Dean of School and the School's Learning Development Tutor. The findings of the 1993 questionnaire and the recommendations of the School Academic Committee were discussed by the Scheme Committees and the School Council. It was agreed to involve the students in the further investigation by organizing a joint student-staff seminar.

The seminar was conducted in two parts. First the students and staff met separately for a discussion; second the students-presented the result of their discussions. The students were asked to prepare a presentation based on a framework for discussion which included comments on the results of the questionnaire, comments on the strengths and weaknesses of their overall learning experience and suggestions for improvement.

Twenty student representatives attended the first part of the seminar and five went on to present their conclusions to the staff. The students responded with a confirmation that high workload was a significant problem and needed to be addressed. They identified the major problem not as being too much content but that scheduling and planning was either not carried out or not adhered to by staff.

Other issues of a more detailed nature were reported back to the staff or relevant Scheme Committees for action. The students had two other major concerns; the poor quality of a minority of lectures and the feeling that consultations with students were not being followed through by actions. It was agreed that these two points could be dealt with outside the meeting in different ways; the former by the new university-wide questionnaire which asked students opinions of individual lecturers which is then processed through the staff self-appraisal system; the latter by the Dean of School attending the next meeting the School Council and explaining the actions to be taken. This would be supported by further consultations with students to monitor the changes resulting from the seminar.

The seminar then focused on the issue of student workload. The student presentation was followed by a joint discussion in which staff gave various reasons for the low scores for Appropriate Workload. These included:

• students with lower entry grades taking longer to do the work;
• the quantity of material in the courses;
• group work demanding more time than individual work;
• the 'bunching' of coursework at certain times of the year;
• the lack of clarity in the standard required for high marks for coursework;
• insufficient feedback to staff on how long students are spending on coursework.

The meeting agreed that more information on student workload was required and that many of the issues raised needed to be addressed in detail by Scheme Committees. It was also agreed that all undergraduate Scheme Tutors should obtain more evidence from staff and students concerning the extent to which staff plan the coursework loading (timing and number of hours for completion), the schedules produced and how well they are both kept to.

Scheme Tutors' Investigations

Results for the 1994 survey were not available at the time of the student - staff seminar or in time for the Scheme Tutors' investigations. They will, however, be used as part of the next monitoring and evaluation reports to the Academic Committee. The Scheme Tutors each produced a single-page report based on their investigations. A number of issues were identified.

• There are large variations in the extent and practice of scheduling and planning.
• Good practice was evident which could be extended.
• Scheme Tutors often experience considerable difficulty in obtaining the required information from some staff and ensuring that schedules and plans are kept to.
• Evidence from student discussions indicated that the Appropriate Workload scores are sensitive to the time in the course when the survey takes place.
• Staff reported students spending too much time on coursework and identified the need for more coaching and information on how students should use their time effectively.
• Assessment procedures and practice need to reflect the time expected to be spent on a piece of coursework more accurately.

Two Schemes of study were identified as already making progress in addressing many of the above issues by using the concept of 'student learning hours'. A standard proforma is used for course planning which requires the specification of topics expected to be taught each week and the exercises, projects and tests that are to be set with all relevant dates. The 'student learning hours' are estimated and quoted for their guidance. After completion of the project/exercise the students feed back to their tutors the actual time spent.

The 'learning hours' quota for the courses is derived from the university's definition of a module as 90 hours of student effort for a 'good honours student'. This is used as the average 'learning hours' per module (a student takes ten 'modules worth' of courses per year).

A final report is now being compiled for the Academic Committee which will summarize the Scheme Tutors' investigations. It will recommend that the School adopt a common policy based on the good practice in scheduling and planning described above. It will also recommend that the number of 'learning hours' per module be reviewed and brought into line with a more realistic value in the region of 120 hours per module (i.e. 1200 hours per year).

The 1994 Survey

Table 1 shows the average element scores, their range and standard deviation, the number of 'cohorts' surveyed and student numbers involved for the 1993 and the 1994 surveys. Table 1 also shows the correlation coefficients for the 1993 and 1994 scores where a direct comparison can be made (18 'cohorts'). The element scores can range between 1 and S; a high score representing a good performance in that element. The results from Ramsden's work for engineering in Australian colleges are also included in the table for comparison (Ramsden, 1990)

Table 1: 1993 and 1994 Average element scores

The second survey generally supported the trends of the first. Poor scores are again seen in student workload and independence compared to other disciplines surveyed in Ramsden's work (Ramsden, 1990). The average element scores for the 1993 and the 1994 show remarkable consistency but there is less correlation between individual scores comparing 1993 to 1994 for particular 'cohorts'. The correlation coefficients for these 'cohorts', in Table 1, are in the region of 0.5 for three of the elements with Appropriate Workload being the highest at 0.57. It appears that these three individual elements have some degree of repeatability and could be used for monitoring purposes. The correlation coefficients for Student Independence and Good Teaching elements are, however, much lower and need to be treated with great caution on an individual year by year basis. The reasons for these variations in individual element scores is not known and little can be deduced from only two years of data. It appears that some elements are more sensitive than others to interference by other factors which change from year to year.

The stability of the average element scores, their agreement with Ramsden work for engineering, the difference in scores compared to disciplines such as humanities (Ramsden, 1990) and the engineering students seeming acceptance of the current quantity of material in their courses may suggest that the overall scores reflect a 'paradigm view' of engineering education. If this is true, then significant improvements would not just require minor changes in course design and management but a radical shift in the way the engineering profession perceives education. This will only be revealed by further work over a longer time scale.

Comments on the Process

It is interesting to note that the highest correlation coefficient came from the Appropriate Workload element which had the some of the lowest individual scores strongly supported by anecdotal evidence. It seems that in this element, where the feelings were stronger and more easily identified, the results were more stable. A detailed analysis of two cohorts with particularly low Appropriate Workload scores indicates that changes made, as a direct result of the survey, to assessments and scheduling appear to have contributed to improvements in these scores. This process needs, however, to continue since it has not yet made an impression on the much more reliable average element scores.

The original aim of using the questionnaire was to improve the student learning experience by providing an indication of the priority for investigation and change. The process has so far consisted of two school-wide questionnaires, a report to the Academic Committee of the school, a student-staff seminar and a detailed investigation and report on the workload issue by the Scheme Tutors. The outcomes so far have been involvement of the students in the evaluation process, a raised awareness of workload and a discussion by Scheme Tutors and Scheme Committees of the likely causes of the problem. The school has now issued standard procedures for course scheduling and planning including the specification of student workload in terms of 'learning hours'. The student representatives will be involved in monitoring the effectiveness of the changes and give feedback to the school's various committees.

The process has highlighted a number of issues in using the questionnaire as a development instrument. Change is slow and hence it is difficult to rigorously monitor since many other variables change with time. The element scores will always remain only indicative and a stimulation for investigation.

The evolving process has underlined the value and limitations of using questionnaire surveys actually to improve teaching and learning. It is easy, as happened during the discussions of these results, to take issue with the questionnaire itself and the questions used and miss the developmental aspects of the process. There are many interpretations of the same data. For example, the poor scores for final year cohorts were attributed by some staff to 'the student experience becoming worse' and by others to 'the students becoming more discerning' (i.e. success!).

Conclusions

The questionnaire has proved to be a very useful instrument in identifying overall areas of weakness which motivated further investigation. It can initiate the process of research and change which is at the heart of improvements in teaching and learning. It has resulted in significant 'awareness raising' and the formulation of procedures to improve and further monitor the situation. It is unlikely that this would have occurred on such a wide scale and action undertaken at school level without the stimulation of such a tool.

The questionnaire results appear to reflect the 'overall learning experience' of the school's students but are much less reliable when applied to individual 'cohorts'. The average element scores appear to have the potential of being sufficiently stable to identify improvements over a number of years on a school wide basis. The elements used are easily related to the teaching activity and hence of immediate relevance to practitioners. Student workload, an important element of teaching and learning, has been addressed and the process of improvement initiated and modelled. Students and staff have worked together on the problem with a mix of quantitative data, consultation, 'awareness raising' and formal directive procedures.

It is not yet clear whether the process described in this chapter will produce significant changes in teaching and learning in the engineering schemes or whether more radical changes are necessary. The problems, particularly seen with student workload and independence, may relate to the relative unpopularity of engineering schemes for undergraduate study. If true, this have important implications for scheme design and for the nature of engineering education which need to be addressed by schools/faculties and by the engineering profession and its institutions.

References

Ramsden, P. (1990). Report to the Higher Education Performance Indicators Project on the Course Experience Questionnaire Trial, NSW Centre for Technology and Social Change.

Gregory, R. D., Harland, G. and Thorley, L. (1993). Using a standard student experience questionnaire with engineering students - initial results. Improving Students Learning Conference, University of Warwick.