Introduction/Overview

Testing your questionnaire is absolutely essential for ensuring a high quality survey. There has been a proliferation of question testing methods (both new methods and variations of existing ones). This course covers 21 different question testing methods. It combines information from the survey methodological literature with practical advice and hands on practice of some of the methods.

Learning objectives

By the end of the workshop, participants will:

• Be familiar with the 21 methods to test survey questions
• Understand the pros and cons of the different methods
• Have ideas for multi-method and simple testing schemes

Topics

This course includes:

• Methods for testing web, postal or interview questionnaires – standard pilot, expert review, systematic forms appraisal (including two online programmes), respondent debriefing (including vignettes and web probing), cognitive interviewing (very short introduction because the SRA offers a course in cognitive interviewing), focus groups, split ballot tests, and usability testing,
• Two methods just for testing interview questionnaires - interviewer rating form and variations, and behaviour coding (classical, sequence-based and shortened),
• A look at newer or unusual techniques which can be applied when collecting pilot data (e.g., eye tracking or latent class analysis for identifying error rates) or alternatives to collecting pilot data (e.g., computational linguistics and crowd sourcing) and soft launch, as well as traditional procedures such as examination of item nonresponse, and response distributions of pilot data. (How to measure reliability and validity, use record check studies and paradata on pilot data are described in the course material as an appendix.)

After exploring all the methods, the day ends with suggestions for different testing schemes aimed to fit both needs and budget.

Who will benefit

The course intended for participants with knowledge of questionnaire design and survey research in general.

Learning outcomes

Participants will:

• Recognise the importance of testing questionnaires,
• Be familiar with the 21 methods to test survey questions and their pros and cons,
• Have the tools to critique new or existing surveys and
• Know how to combine one or more methods into a testing scheme suited to your needs.

Course tutor

Dr Pamela Campanelli is a Survey Methods Consultant and runs The Survey Coach business. She is also a Chartered Statistician, Chartered Scientist and Fellow of the Academy of Social Sciences. She has worked at the University of Michigan, the U.S. Bureau of the Census, ISER at the University of Essex, and NatCen Social Research. She has both led and been a team member on ESRC grants (one on survey nonresponse and one on measurement error in mixed mode surveys). She regularly teaches short courses in the UK and abroad for government departments, survey research companies, universities, as well as for various other institutions and businesses.

  This course contributes 6 hours to the MRS CPD programme

n.b. This course runs over two consecutive afternoons:

Part 1 - 3 November - 12.30 pm to 4.30 pm

Part 2 - 4 November - 12.30 pm to 4.30 pm