So this unit is a lot about pretesting techniques and making sure that your questionnaire is in the best possible format. We'll start out with a subset of techniques, but let me first tell you why we do pretesting in the first place. Overall, we try to do pretest to identify and reduce any possible errors on the test. Now any, that's a hot claim. Remember, it took them 400 years to measure the longitudes. So don't expect that you can build a good measurement instrument now within two weeks or the six weeks of this course, but these techniques that we present here are a good step to get better at identifying specification error. Seeing if your decomposition has worked if you really tap into all the concepts that we discussed at the beginning. It will help you to identify operationalization error. Is the question really measuring the construct, do you see variability across the respondents, the interviews, the times? Issues with measurement in that regard? And then measurement error in general, remember the sensitive questions, do you have question characteristics that are leading the respondents to answer a certain way? And there are typical interviewer characteristics that interfere with the certain question, and issues of that nature. So all of that, hopefully, you can detect in pretests. It is rather uncommon that you would be able to administer a survey perfectly, measurement error free, without any pretesting, so you know, one of these techniques you should use, ideally, a multitude of these techniques. We'll show a comparison of the techniques at the end of the segment, and you will see they're not suitable for all settings. And you certainly get the best picture if you use several techniques at once. The first thing, expert reviews, focus group, cognitive interviews, they are all more qualitative techniques. And then we have behavior coding and other statistical methods that, I guess behavioral coding can be done on the qualitative interviews as well, but all of the others are sort of assuming that you have a larger set of paces for which you test whether they measure questions and the answer match what you're trying to measure. So, let's start with the expert reviews. First thing you need to ask yourself is, who's an expert? Who the questionnaire designated expert in that matter, that's one set. But also, you should probably enroll subject matter experts to review your questionnaire to make sure that it also matches the concept that the subject matter expert told you to begin with. Questionnaire administration expert is another good set for trying to enlist into review those would be interviews. And then there are some computer based systems, expert systems like QUAID that have been developed and the link is available here, you can test that out. What do these experts do? They identified potential response problems and make recommendations for improvement. Now, how do they do that? Are they individual or in a group setting? Sometimes you have open-ended comments, sometimes you can solicit codes for a particular problem that would be in form of an appraisal system. In the end, you hopefully have a report with comments that help you to improve, noted problems and revisions, a summary of report that shows you the distribution of the problems you want to definitely have a few experts review this and not just relying on the answers from one. They're really qualitative in nature, though, so that you won't be able to make any statistics off the problems you found. Now, experts are good at identifying problems related to the data analysis question. They are good at looking at issues that pop up. In a quantitative analysis of comparing expert reviews with more sofisticated techniques like latent class analysis, we've found considerable agreement, but we'll come back to that point once we look closer at these LCAs. What's definitely good is they're relatively cheap and fast, and what's not so good is the quality of various in practice much a lot, we don't have a lot of literature on this and there is this one thing known, large inconsistency and disagreement between experts. So with that, let's move to the next technique, which are focus groups. Focus groups are small groups, five to ten people, that you bring together to investigate a research topic. You try to find out what people think when they hear this topic, when they hear a particular question, so you can mix with an open discussion, and an evaluation of survey questions themselves. Try to figure out how people about the vocabulary, about the terms used, about any key concepts that appear in the questionnaire. And keep in mind interviewer and respondents don't always agree on this context. You know let's say for example sources of income. A questionnaire administered in a low SES neighborhood, the investigator and as an extension the interviewer, they might think of job salary, interest income, dividends, whereas the respondent would think of ad hoc work, illicit activities, drug use, prostitution and gambling, there you have it. So the key task of the focus group is to explore what the respondent thinks and how the respondent thinks about the topic, maybe both. You have to think about recruiting, what the moderator does, identify good moderators, decide on whether to tape, and videotape, but at least you should audio record, it's very difficult to take sufficient notes, and then have a report written. A little bit more on recruitment. You really would like to target participants that are also part of your target population in the survey. You want to decide if you want to have a homogeneous group or a more heterogeneous group. If you have the means to have several groups, it could be good to vary that a little bit. You will get a different group discussion whether you do one or the other. But in general, it's probably a good idea if you only have one focus group to have it diversified, but not too much. So each group has a little bit of a mixture in it. As for the moderator, you want to give the moderator a guide, where you make clear what the purpose is that they study here. You can talk about the flow where you write out questions with open ends and then you can see what the respondents will answer those. And keep in mind what problem should be solved and what information you search for. Here's an example for points that should be mentioned in the moderator guide. Flow of ground rules, introduction, open questions, in-depth investigation, and some statements on how they can close the focus group. It's also good to give them a guideline for good questions inside the focus group. Tell them that they should ask short questions to get long answers. They should ask questions that are easy to say, they should address one issue at a time, use a conversational tone, ask open-ended questions, and ask positive before negative ones. A focus group is very good at providing qualitative information as we had earlier. They're good in providing a range of information, lots of a variety. But what you learn there, it's hardly generalizable. So it's qualitative just like the expert reviews. The advantage is they're efficient and small. The disadvantage is they're more costly, because you have to recruit, you have to incentivize, you often have to rent a facility, in particular if you want to have outside service through one way mirror and things of that nature. There are lots of good textbooks on focus groups, a couple of them listed here, more are listed on the syllabus on Coursera. Our next segment will move into cogmented interviewing another qualitative technique that can be used to test questionnaires.
