J.H.F. Meyer

A framework for evaluating and improving the quality of student learning

J. H. F. Meyer (University of Cape Town) and K. Scrivener (Imperial College)

Reproduced with permission from Gibbs, G. (ed.) Improving Student Learning - Through Assessment and Evaluation. Oxford: Oxford Centre for Staff Development (1995)

Introduction

Student learning is a complex multivariate phenomenon. There are many sources of variation, some of them unobserved and uncontrollable, that contribute to manifestations of learning behaviour and of learning outcome. Practitioners are often confused by the layered complexity found in competing conceptual models of student learning; they generally fail to appreciate the penalties that are incurred as the genuine complexity of student learning is approximated for modelling purposes. From a practitioner perspective there is thus an inherent tension in adopting a model that is conceptually simple and 'user friendly', and actually using that model in practice for the very serious business of evaluating and improving student learning. The simpler the model, the fewer the number of variables in it, and the more limited the sources of contributory (or explanatory) variation for the observed phenomenon. Part of the art of modelling lies in deciding what it is that needs to be understood or explained, and then using some common sense in selecting and using a model that is sufficiently detailed for that purpose. This is basically what this chapter is about; the application of a model to evaluate both static and dynamic aspects of student learning.

Individual Differences

At the most basic level it is recognized that students engage learning with differing intentions and motives (Biggs, 1979). This is not surprising, as intention and motive are two powerful forces that plausibly explain much observable human endeavour. This is especially so when the intention to do something is supported by a congruent motive; the intention to understand something, for example, is congruent with a motivation based on intrinsic interest. The intention could equally well be based on ambition; a desire to compete and excel. It could also be motivated by fear of failure. Alternatively, the intention to understand could be motivated by a combination of motives. Looked at in this way it is already apparent that some combinations of motive and strategy are more likely to realize the stated intention than others. There is thus a first layer of variation in student learning based on intention and motive.

The motive on its own, however, is no guarantee that the intention will be realized. At the very least, some method and process is involved, and these are likely to be further shaped to some degree by personal preference or habit (Entwistle and Ramsden, 1983). In short, it follows that the acquisition of conceptual understanding is also dependent on additional cognitive processes. Some of these processes address the manner in which students interact with the content of what they are learning; how they appraise, organize and reflect upon information in relation to what they already know. Contrasting intentions are clearly likely to be associated with different processes. For example, an intention to understand may invoke an iterative process of reflection and critical conceptual interaction, while an intention to memorize information for subsequent reproduction in an examination may simply be realized by a process of repeated rehearsal. Some students are also predisposed to do certain things (like solve problems) in a characteristic manner; such preferences introduce a 'style' distinction between divergent and convergent approaches (to solving problems), and they represent a further important source of inter-individual variation. There is thus a second layer of multivariate complexity in student learning that can substantively alter a perception that two individuals with the same declared intention and motive are similar.

A third layer of variation recognizes that learning cannot be decontextualized; it is purposefully shaped by perceptions of the learning environment, and especially by the demands that are perceived to regulate academic success and failure. From what has already been said, it is self-evident that substantial inter-individual variation can occur in the manner in which perceived learning demands are responded to. There are other perceptions of the learning environment that can contribute further to an explanation of individual differences; perceptions of the content of what is being learned, of learning materials, of the learning space, perceptions of self, and perceptions of causal attribution (Meyer, 1991; Lefcourt et al., 1979).

Decompositions of a Deep-Level Structure

There is considerable empirical evidence that the 'dimensionality' of variation in student learning is relatively stable across different studies in respect of what might be called a 'deep-level' structure, but that it is less stable in terms of other posited 'strategic-' or 'surface-level' ones. This evidence has accumulated from numerous group-level exploratory factor analyses of undifferentiated datasets based on models of varying layered complexity as outlined. A review of these quantitative studies by Richardson (1994) provides an excellent contemporary summary of this research (circa 1993).

Empirically, additional (factor analytic) dimensions of variation emerge when the fundamental assumption that a group adequately represents its constituent members is relaxed, and many of these dimensions do not conform to stereotyped conceptions of variation in learning behaviour. Under this relaxed assumption it can very easily be demonstrated that contemporary conceptual models of student learning, or admissible aspects of them, do not 'fit' some individual-similarity data structures, even at a relatively coarse level of individual similarity constituted on the basis of gender (Meyer, Dunne and Richardson, 1994; Meyer, 1995)

This observation is equally true in respect of analyses of student learning based on an individual-difference statistical model that is independent of correlational assumptions (Meyer and Muller, 1990). Conceptual stereotypes are, in fact, of very limited value in attempting to interpret individual differences. This observation has led to the exploratory modelling of conceptual differences in observed data (as opposed to empirically manifested differences), and has presented a promising framework for addressing the manifestation and consequentiality of inter-individual variation in learning behaviour (Meyer, 1993)

Categorisation

In essence, this approach admits the categorization of discrete conceptual decompositions of an 'ideal' deep-level learning behaviour structure. At an individual level, or in analyses of individual-similarity subgroups, stereotyped dimensions of variation in learning behaviour, or their admissible conceptual variants, are simply admitted as special cases. At the same time full justice is done to manifestations of aberrant or atypical 'high risk' patterns of learning behaviour that are associated with academic failure. The categorization, furthermore, carries with it an assignment of risk that is, in this case, informed by conceptual argument and the results of previous empirical modelling. (It should be noted that the concept of 'risk' is inherent in the definition of learning if students themselves define it in qualitatively contrasting terms.)

The instrument used as a (data yielding) basis for the categorization procedure is one that has proved useful in previous individual-difference studies. It is a modified and Extended version of the Approaches to Studying Inventory (EASI), further supplemented by a set of contextual perception, and causal attribution, variables. A subset of EASI variables that are relevant to this study are introduced further on.

Each individual student response is quantified in respect of the variables embedded in the Inventory, and the assigned values then determine a (symbolic) profile based on a ranking of the individually averaged item values. This profile is then treated as a preference structure (for analytical purposes) and is examined for distinguishing conceptual features that determine the categorization. This somewhat complicated procedure is fully explained elsewhere (Meyer, 1993; Meyer and Parsons, 1994), and the technical details are not further discussed here.

The categorisation procedure produces a risk category; this is simply a number (that presently ranges from 1 to 13) that is regarded for exploratory purposes as possessing conceptually ordinal properties. Notwithstanding the dangers inherent in the presumption of strict conceptual ordinality in the 13-way categorization, it remains true that, under ideal conditions, each category isolates a unique set of features in the individual response. It is these categories in a collapsed form (on a scale of 1 to 5) that have been used with some success in modelling learning outcomes, especially the relationship between 'high risk' learning behaviour and academic failure. The categorization thus provides a direct method, subject always to error variation, but amenable to independent verification (via interviews), of distinguishing between self-reported individual student responses.

Background to this Study

At the commencement of the 1993 academic year, students in the Department of Materials at Imperial College were asked to describe how they had previously engaged the studying of Science in their final secondary school year. Data thus obtained were used as a basis for initial risk assessment, and as a comparative base for establishing subsequent changes that would be presumed to be essentially attributable to the effects of the course. Three months later students were again requested to self-report their learning behaviour in the context of the Materials Science and Engineering course.

The 13-way risk distributions obtained prior to the commencement of undergraduate studies, and within the course, are shown in Table 1. Discrepancies in sample size are to missing data. (There were 53 students in the course at the time of completing the second Inventory.)

The data in Table 1 indicate that the within-course risk distribution is worse than the before-course risk distribution, notwithstanding the fact that the two distributions are derived from all complete (but not necessarily matched) data obtained on the two successive administrations of the Inventory. These categorical data form part of the framework of this study, the purpose of which is to demonstrate how Inventory data can be used to inform an evaluation of student learning in a static, dynamic and diagnostic sense.

Some Group-Level Learning Behaviour Dynamics

The first part of the framework illustrates some dynamic aspects of learning behaviour via empirical results that require conceptual interpretation.

A subgroup of individuals (n = 45) completed both Inventories and these matched sets of individual-response data are now considered. Table 2 contains the results of an analysis of differences in mean scores for the paired comparison data on the full variable set. Only differences that are significantly non-zero are tabulated, and they are listed in decreasing order of statistical significance. The sign of the mean (difference) indicates whether the shift is positive or negative relative to the before-course value on that variable. It is thus clear from Table 2 that an overall significant deterioration has occurred; all the positive shifts represent increases on the mean scores of theoretically undesirable variables, while all the negative shifts represent decreases on theoretically desirable variables.

Of further interest is whether there is an underlying empirical structure that can be used to interpret the shifts indicated in Table 2 in a conceptually coherent manner. To this end an exploratory (principal) common factor solution is presented in Table 3 under oblique rotation. The factors represent unobserved, and empirical, dimensions of variation across which all students contributing to the analysis can be expected to vary. While acknowledging the influence that the relatively small sample size may have on the stability of the factor structure, the challenge here is nevertheless whether these dimensions make sense conceptually, and whether they can contribute to an evaluation of the dynamic effect that the course is presumed to be having on the students.

The loadings on Factor 1 suggests a dimension of variation that is conceptually quite coherent; in the absence of both a deep approach (DA) and intrinsic motivation (IM), there is a manifestation of perceptions of a heavy workload (WL) and disorganized studying (DS) in the absence of an organizing principle with which to deal with new information (fragmentation: FA), with an added weak qualification in terms of over reliance on factual detail (improvidence: IP). This dimension of variation in learning behaviour is furthermore associated with an (external) attribution for academic success in terms of favourable circumstances (COS).

Factor 2 captures a contrasting, but complementary, dimension of variation compared to Factor 1. The narrow focus here, in the absence of both a deep intention to understand (DA) and making use of evidence (UE), is on relational aspects that is disorganized (DS); perceptions of the learning environment that facilitate the efficient transfer of information (LS) and a reluctance to expend intellectual effort beyond stated requirements (syllabus boundness: SB). This factor is associated with an (internal) attribution for academic failure in terms of a lack of effort (EFF).

The emphasis in Factor 3 is essentially motivational; fear of failure (FF), is linked to an absence of (internal) attribution for academic failure in terms of a lack of ability (ABF), to syllabus boundness (SB), and workload (WL). This dimension of variation is further weakly qualified by deep intention (DA), use of evidence (UE), and improvidence (IP).

Interim Discussion

Most first-year students come from a school environment where they were generally taught one broad subject (such as Physics) by one teacher, where homework requirements were clearly laid out, and where there was very little necessity for them to find out information for themselves (for example, from books). In their first year of university study they may have ten different lecture courses taught by different people. In each of these courses they are set work, often with complete disregard for the workload imposed by other courses. The whole style of information delivery is also very different to their previous experience. Compared to school, it thus appears reasonable to expect changes in the learning behaviour of first-year students to occur.

The practitioner's reaction to the shifts depicted in Table 2 is that they are consistent with common sense expectations. Students find it more difficult to organize their time (a problem exacerbated by being away from home and taking advantage of many new social opportunities); the subject appears fragmented without an overall structure, and they are overwhelmed with the amount of work that needs to be done. They don't really know how to take notes and tend to focus on the minutia of legibility and delivery of lectures, resulting in improvidence. All these pressures lead to an increased fear of failure and decrease their internal motivation. Attempts to understand (deep approach) are undermined. and their perception of the course content is superficial.

From a research perspective, one cannot infer causal relationships from the temporal proximity of events captured in a factor structure. However, interpreted against the findings of other studies, and within the underlying theory, the dimension of variation captured in Factor 1 is symptomatic of a classic syndrome: perceptions of a heavy workload, and an inability to cope with it in an organised manner signal the start of what can become a destructive cycle of deterioration in learning behaviour that is demotivating and is not directed towards conceptual understanding (Meyer and Sass, 1993). A broader interpretation of Factor 2, seen in relation to Factor 1, signals what may be the beginning of a strategic readjustment, or coping strategy within the course, while Factor 3 adds the motivational influence of fear of failure.

Some Individual-Level Learning Behaviour Dynamics

The analyses presented thus far essentially capture some of the group-level dynamics of changed, or changing, learning behaviour that may be attributed to the effects of the course (and the university environment in general). It now remains to address the dynamics of learning behaviour at an individual level. One way of doing this is to construct a frequency table of before- versus within-course categories as in Table 4 (based on a collapsed 5-way categorization of self-reported learning behaviour; 1 = high risk, 5 = low risk).

The contents of Table 4 communicate a wealth of information. The totals at the bottom, and on right hand side, of Table 4 indicate the before- and within-course risk distributions respectively. The diagonal cell entries represent subgroups of individuals manifesting apparently stable (categorical) patterns of learning behaviour; for example, five individuals self-reportedly appear to be in the 'high-risk' category, while none of the five individuals coming into the course with a 'low-risk' categorization maintained that categorization.

The upper triangle of cell entries (those that lie to the right of the diagonal) indicate subgroups of individuals who have apparently deteriorated relative to their initially self-reported learning behaviour, while the cell entries in the lower triangle indicate subgroups that have apparently improved upon their initial categorisations.

It is self-evident that the frequency table is asymmetric about the diagonal (and significantly so in terms of the McNemar statistic); there is, as expected, more deterioration than improvement. In addition, there is moderate association between the two sets of categories as evidenced by the various summary statistics. The ultimate value of Table 4, however, lies in the fact that the cell entries represent subgroups of individuals whose identities (in this case) are known on a voluntary basis.

The Evaluation of Learning

There are now two complementary sets of data that are amenable to evaluation; one represents the dynamics of learning at an aggregate level, the other represents intra- and inter-individual learning dynamics. The former data can be used to compare patterns of evolving learning behaviour across different courses, or within the same course over time, or across successive versions of the same course.

An interest here is in individual differences and their consequentiality. The presentation that follows is based on one set of examination results, relating to one fifth of the first-year course. The association of these results with the categorisations of self-reported learning behaviour are good, but no better than the association with the performance at 'A-level' before entry. However, the data on which the categorisations are based have diagnostic as well as predictive properties; the point is that, at an individual level, these data can inform a variety of counselling and intervention activities. In both cases there are some pronounced exceptions to the general trend, although these exceptions are generally different students in the two cases.

The academic progress of students who self-reportedly had either remained in, or had shifted to, a 'high-risk' category of learning behaviour is of particular interest here. The before-course categorisations identified seven students with 'high-risk' learning behaviour patterns (see Table 4); five of them maintained a relatively stable condition and two of them manifested some improvement. These seven students were informally observed more closely during the course of their studies.

Subjectively, two of these students were clearly problematic. They appeared to have little, if any, grasp of the basis of 'scientific method' and, when questioned, could recall little (if any) material delivered in lectures only hours previously. Both students, however, managed to scrape along by transferring segments of information from lecture notes and text books into laboratory write-ups with sufficient editorial skill to obtain adequate marks. Many of the staff independently identified these students as having bizarre learning strategies which emerged clearly from this study. Both of these students maintained a 'high-risk' pattern of learning behaviour in terms of the within-course categorization, and performed poorly in-the examination, obtaining 37 per cent and 43 per cent respectively.

A second initially identified 'high-risk subgroup consisted of four students who were clearly weak, and who were observed to be having difficulty in understanding things at a conceptual level. However these students had clearly evolved strategies which allowed them to cope. All were able to provide good answers to questions of a type previously encountered, and performed comparatively well in the examination.

The remaining initially identified 'high-risk' student is the most interesting case; this student had good entry qualifications and performed very well (72 per cent) in the examination. This student was interviewed (as part of a separate study aimed at designing teaching software) in order to establish how some of the essential course concepts were being understood. The interview data clearly indicated that this student had developed a successful coping strategy. The basis of this strategy rested on an almost perfect ability to recall key definitions and explanations given in lectures. However, when questioned further about the meaning of some of these, this student commented along the lines of 'You don't want to try and understand things too deeply or you will just get confused; some things should just be accepted without question.'

The general pattern of within-course shifts was viewed by the practitioner as being rather depressing; the learning behaviour of the majority of students appeared to have deteriorated. Of particular concern was the subgroup of seven students who initially had not manifested 'high-risk' patterns of learning behaviour, but who subsequently did so within the course. The categorisations associated with this subgroup dropped to 1 from a previous 3 or 4. Three of these students had independently been perceived to be having some difficulties with the course, and two of them did badly in the examination. Of the remaining four students, one had a very poor attendance record for the course, but the other three appeared to be doing well and also performed well in the examination. It was felt by the practitioner that some of these students' learning behaviour may have deteriorated only temporarily, or that their within-course responses may have been biased by particular sections of the course with which they were having difficulty at the time, but not to the extent of affecting their overall performance.

Improving Learning

The framework that has been outlined produces a wealth of diagnostic data, some of which can clearly be used to 'improve' learning in a number of respects; the choice however depends on whether the students, the course, or both, require attention. For example, a similar exercise carried out within the same framework on a first-year engineering course has been reported by Meyer and Sass (1993) and has contributed to a major revision of that course. The focus here, however, is on the students.

There are, firstly, static 'snapshots' of individual patterns of learning behaviour before students enter the course, and when they are in the course. The frequency of within-course snapshots can obviously be increased and can be used for monitoring purposes. The value of the first snapshot lies in being able to identify individuals potentially manifesting 'high risk' forms of self-reported learning behaviour before they even commence their undergraduate studies. The danger is that, without assistance, these individuals may fail to adjust quickly enough, or may adjust inappropriately, to the demands of undergraduate study. In some individuals, 'high-risk' patterns of learning behaviour appear to endure, with a consequent expectation of academic underachievement or failure. There is some evidence that early detection of this condition, and explicit forms of intervention intended to improve it, can meet with limited success if attempted at an intensive individual level, but not on a large impersonal scale (Cliff and Dunne, 1994).

Of equal concern are the individuals who deteriorate to 'high-risk', or nearly so, patterns of learning behaviour within the course. In another similar study to this one, but carried out on a larger scale, it has been observed that some previously 'good' students will 'recover''; this 'recovery' may take the form of a strategic readjustment to course demands (Meyer and Sass, 1993). However, the 'worse' the previous learning history, the less likely the 'recovery'. The problem of intervention here for students who fail to 'recover' (or readjust) is that when the deterioration has been detected, it may be too late to do anything about it. However, the major sources of variation associated with the condition (at an aggregate level of analysis), as exemplified in the interpretation of the data in Table 2 and Table 3, represent 'early warning signals' that can be of strategic value in forestalling the condition for at least some individuals.

Such an intervention has recently been attempted in the form of a learning 'Hot Seat' (Meyer and Kaschula, 1994). This is basically a walk-in learning counselling service on student learning that operates within a specific course. Students are informed at various points in the course of what the 'early warning signals' are, and are asked to seek counselling assistance if they feel that they are, for example, being overwhelmed by the workload, are experiencing difficulty in organizing their studying, are working hard and not doing very well, and so on. In the context of this study a suitably paraphrased description of the data contained in Table 2 substantively constitutes such a set of 'early warning signals'.

The practitioner's overall reaction to the analyses presented is that it has confirmed and illuminated what should have been known; that a major cause of the deterioration in learning behaviour is the difficulty in adjusting from the highly regulated school environment to the less highly regulated working environment of the university. It is also conceded that the coursework component in this particular course is too high. Another contributing factor to what has been observed is that first-year courses in general concentrate on providing the basic tools of a discipline that will only be integrated at a later stage. Students may thus experience difficulty in seeing the relevance of what they are attempting to learn in the overall context of the subject. This is, in fact, frequently the reason given for disillusionment by students who consider leaving the course around the first year.

Concluding Discussion

The framework for evaluating student learning that has been outlined is a versatile one that has been used in a number of different contexts. It produces data that is both diagnostic and strategic, and that can directly inform academic practice and learning counselling.

The question of whether a knowledge of school-leaving learning behaviour can be used for selection purposes also arises. From the limited data available there appears to be no prospective benefit in selection based on self-reported learning behaviour versus selection based on A-level results as at present. However there might be some potential benefit in terms of selecting students with a comparatively poor at A-level performance but who nevertheless manifest learning behaviours that are likely to enable them to succeed at university.

References

Biggs, J.B. (1979). Individual differences in study processes and the quality of learning outcomes. Higher Education, 8, 381-94.

Entwistle, N.J. and Ramsden, P. (1983). Understanding Student Learning (London, Croom Helm).

Lefcourt, H.M., Von Baeyer, C.L., Ware, E.E. and Cox, D.J. (1979). The Multidimensional-Multiattributional Causality Scale: the development of a goal specific locus of control scale, Canadian Journal of Behavioural Science, 11, 286-304.

Meyer, J.H.F. (1991). Study Orchestration: the manifestation, interpretation and consequences of contextualised approaches to studying. Higher Education, 22, 297-316. .

Meyer, J.H.F. (1993). The individual-difference modelling of student learning: I - Static and dynamic aspects of causal attribution. Paper presented at the Fifth European Association for Research on Learning and Instruction Conference, Universite de Provence, Aix-en-Provence, France, 31 Aug - Sept 5.

Meyer, J.H.F. (1995). Gender-group differences in the learning behaviour of entering first year university students. Higher Education, 29, 201-215.

Meyer, J.H.F., Cliff, A. and Dunne, T.T. (1994). Impressions of disadvantage. II - monitoring and assisting the student at risk, Higher Education, 27, 95-117.

Meyer, J.H.F., Dunne, T.T. and Richardson, J.T.E. (1994). A gender comparison of contextualised study behaviour in higher education, Higher Education, 27, 469-85.

Meyer, J.H.F. and Kaschula, W. (1994). Helping engineering students to learn better: the concept and creation of a learning 'hot seat'. In Smith, A.J. (ed.) Engineering Education, Increasing Student Participation, Sheffield Hallam University, Sheffield, 294-300.

Meyer, J.H.F. and Muller, M.W. (1990). Evaluating the quality of student learning. I - an unfolding analysis of the association between perceptions of learning context and approaches to studying at an individual level, Studies in Higher Education, 15, 131-54.

Meyer, J.H.F. and Parsons, P. (1994). Conceptually at risk students: diagnostic and intervention strategies based on individual differences. In Gibbs, G. (ed.), Improving Student Learning: Theory and Practice, OCSD, Oxford.

Meyer, J.H.F. and Sass, A.R. (1993). The impact of the first year on the learning behaviour of engineering students, International Journal of Engineering Education, 8, 328-35.

Richardson, J.T.E (1994). Cultural specificity of approaches to studying in higher education. A literature survey Higher Education, 27, 449-468.