A student-generated questionnaire for the evaluation of teaching in psychology
Date
1993
Authors
Müller-Clemm, Werner Johannes
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
An important part of program evaluation research is to investigate methods for the production of reliable instruments with at least face validity. The evaluation of teaching in the university classroom is particularly challenging owing to its complexity and the variety of the activities involved, and also to the lack of generally accepted criteria for "good teaching." The most common way in which the problem has been approached has been to produce sets of descriptive statements or questions having to do with conformity to normally accepted standards of teaching practice and student reactions to teaching performance. Such rating instruments are commonly generated by the teaching or administrative staff.
The present research approached the problem within this format, but deviated from normal practice by seeking a way in which the questionnaire items could be generated by the students themselves. Of particular interest was to see whether the concerns expressed by the students regarding what items they felt ought to be included on a teaching evaluation questionnaire would differ significantly from those expressed by standard instruments. The research began by repeating an earlier attempt to proceed in this manner known as the "ECHO" technique (Schaefer, Bavelas, & Bavelas, 1980). Finally, a streamlined version of the earlier procedures was developed that appeared to deal more adequately with the problems of the reliability of instrument production.
A randomly selected group of students was asked to generate as many statements describing the value of courses and teaching as they wished. A second group was asked to categorize these items according to similarity of content, choose the most representative item of each category, and then rank the importance of each representative item.
The second procedure proved to be much to cumbersome. The focus was narrowed to only those items having to do with teaching performance. A further, less time-consuming, innovation was to provide a "discard" category into which items judged to be unimportant could be discarded. This procedure for sorting and ranking was then repeated by ten additional groups. The consistency of each item and its average ranked importance were calculated following the ten sorts. These provided the basic criteria according to which the final instruments were constructed.
Using a criterion of consistency of ten out of ten sorts, only three items in two categories remained. A reduction of the consistency criterion to eight out of ten sorts (the statistically reliable minimum based on binomial probabilities) yielded 15 items, representing nine distinct categories. These bore a remarkable resemblance to the item-content of the standard questionnaire currently used by the Psychology Department.
Applying the criterion of judged importance required a relaxation of the consistency criterion to five out of ten sorts yielding a set of 16 items where again, 9 distinct items with high rankings for importance were identified. The content of these questions was somewhat different from those produced by the consistency (reliability) criterion.
Two nine-item questionnaires produced by this procedure, one (EC) based on a criterion of consistency of items and one based on judged perceived importance of item content (EI), were then presented to an independent sample of students from the same population, along with the standard departmental questionnaire (MD), reduced to nine items for comparability, to be preferentially rated for their expression of students' concerns. Using a paired-comparison procedure, the EC and MD forms showed no significant difference. The EI form, however, was judged as significantly more representative of student interests than either the EC or MD forms.
Implications for the practicability of student-generated teaching-evaluation questionnaires are discussed.
Description
Keywords
UN SDG 9: Industry, Innovation, and Infrastructure