The Impact on Teachers of Language Variation as a Course Component
METHODS
Questionnaire Design
The questionnaire was designed to elicit ratings for all the courses offered to the participants as part of their MA program, partly in order to reveal how language variation was rated in relation to other course components, but also to disguise the true focus of the study. The meaning of questionnaire responses is notoriously hard to interpret. The ‘leading question effect’ is a particular source of unreliability; the wording of questions can easily predispose respondents to answer in certain ways. Asking subjects to comment on our teaching of language variation, without setting that topic in a broader context, would almost certainly be inadequate because perceptions of the worth of any course element are bound to be relative. In the light of these well-known problems with questionnaire research, our aim here was to disguise the focus on language variation by burying questions about it in a larger survey of responses to our courses, and many students did respond as if to a course evaluation exercise, in spite of it being clearly labelled ‘research’!
The questionnaire comprised three sections: Section A, which asked for demographic information; Section B, which elicited a general course evaluation, and Section C, which itemized the detailed components of the selected course for rating against various criteria (sociolinguistics, with 10 sub-components including five language variation topics).
We designed our questionnaire to be completed on a web site. This medium has advantages, not only for busy people living thousands of miles away but just as importantly for us, because the responses for analysis can be downloaded automatically to a database (Hewson et al., 2003:43). The practicalities of this web-based questionnaire were solved by a helpful computer officer, but there were a number of difficulties along the way. Computerization does not eliminate problems of partial or erroneous completion, and we have had to take the usual decisions about whether to include incomplete returns. We have not included partial returns in the quantitative analysis of data for Section C, but we have used some of the comments that help to illuminate issues from the 26 partially completed questionnaires. Validity of data is a concern, since remote administration undermines the ability of the researcher to judge the sincerity or genuineness of responses (Hewson et al., 2003:44), and the mechanics of selecting responses from ‘pull-down’ lists can result in inadvertent mis-selection. In Section A, six people gave incorrect program dates, almost certainly caused by ‘mis-clicking’ possible responses. Given that much of the questionnaire used the same mechanism, all the results should be interpreted with the awareness that around five per cent of responses may not be the ones the respondents intended.
Despite these caveats, the web-based questionnaire yielded a good body of information, and we see this as a positive step forward in course evaluation. Furthermore, to maximize return rates, we incorporated a backup medium in the form of a word-processed version of the questionnaire that could be completed electronically and emailed back to us, or printed, completed by hand, and faxed. This procedure helped us to remove any sample bias caused by lack of internet access or inadequate system requirements on the part of otherwise willing respondents.
