In this video, we're gonna talk about running studies in-person. And to give you a preview, in the next video, we're going to talk about running studies online. The web has made it much easier to run controlled experiments and I think that there's a lot of people who run studies that they wouldn't have been able to previously do. But there's also a lot that you can get from watching people. Don't miss out on that. In particular, at watching people live, you can talk to them about what their doing, get their feedback, see points of confusion, and in general, have a much higher bandwidth engagement with the participants in your studies, then you would get just out of a click stream online. So, today we're going to talk about planning, executing, and analyzing in-person studies. The first step is to make clear goals, for example, say, you were building a room reservation system for a Computer Science department. One strategy would be to put all the information on one page. Another strategy would put different pieces on different pages. And you might be particularly interested in if we split things up. Does that change whether people will book a right size room or not? So, before running your study, lay out its scope. You don't need to cover an entire system, especially if you're working on a huge piece of software. It's okay to have a narrow scope. And that scope will be guided by the purpose of your study, what you hope to learn. I encourage you to come up with a hypothesis ahead of time and figure out a way to know whether your hypothesis is true or not. Set up your study in way so that you can see whether or not people's behavior supported your hypothesis. Book an appropriate place for your study. If the interface is something that will be used in a quiet room, book a quiet room. If, on the other hand, it's gonna be used at a train station, test it at a train station. Figure out who you're gonna recruit and how, and how many people. And come up with scenarios that you give people when they come in. You want to m ake sure that these scenarios are realistic and ideally something that these participants can be motivated to care about. At least put themselves in the shoes of the actual user of the system. So, here we have some questions, like will people book the right sized room and for the right length of time. And this implies to measures. The size of room that they booked, how long it was booked for. You can also get measures like task completion time, how long it took and especially if you have realistic users, you can see how this task interweaves with other things they do. So, if people are actually booking a room for an actual meeting that they need to have, you might see whether people makes notes to themselves to also order food or whether they need to coordinate with people over e-mail first to, for example, get an approximate attendance count. And if you think things like screen size might affect how users behave, have a screen that's appropriate for what they'll have so. The simplest thing to do obviously is to have a stock setup for everybody that is pretty normal. One danger that you can see is that developers often have fancier computers than normal users. So, don't test people on the latest greatest fanciest thing if what people are going to be looking at your site with is a 5-year-old laptop. If you can, try to get at least two people to be present for the study. That way you can have one person who is the facilitator and another one who can take notes. It's important to have concrete tasks among other things that lets you compare the performance between people. And I encourage you to write them down as opposed to speaking them so that everybody gets the same experience, making it even easier to compare. Here's an example of a creativity task asking people to draw creatures from a novel planet. And you can see that the instructions are pretty clear. And you can see that the instructions are pretty specific, even though the output of the task is really open ended. And here's an example of the meeting room t ags. You can see how it gives some concrete details and context. I've been amazed and heartened by how seriously most of the participants in our studies take what they do. And the flip side of that is that sometimes people have left in tears when they've been unable to perform well. In studies, people can feel like they're being put under a microscope. And you have a responsibility to alleviate that stress. For starters, make consent voluntary and make sure that consent is informed. Avoid pressuring people to participate and let them know that they can stop at anytime. And maybe the most important rule of all for minimizing stress and increasing the quality of the interaction that you have with participants is remind people multiple times that you're testing the site or the user interface, the system, not them and that's an important frame change because when you're testing the system, if the user is unable to complete the task, the fault is the system's fault. If participants believe that they are the ones being tested, then for example, if I am being unable to perform a task that reflects poorly on me and my abilities and I may be stressed out. If, on the other hand, I, as a participant believe that I am helping the designer test the system, then when I encounter a roadblock, it's not a poor reflection of my abilities. It's I've done something good. I've found a bad part of the user interface that I can then help the designer make better. There's several important considerations in a study like this. For example, what's going to be your order of tasks? One good approach to this is to start out with a really simple task to get people warmed up. And then you can move to more complex tasks as you go through the study. Other times you may shuffle the order so there's no effect of one of the tasks on another one. How much training, if any, are you going to provide? It depends on how the real system's going to be used. If you have a ticket machine, not so much. If you have a surgical system, maybe a little bit more. What are you gonna do if somebody doesn't finish? Probably the answer shouldn't be lock them in the room until they do. For a lot of tasks, I recommend deciding in advance what a reasonable upper band for time is. And you can let participants know that. Another strategy is that if somebody hits a small stumbling block, you may elect to help them along so that they can get to the rest of the user interface so that you're not purely stalled out on one point. And finally, pilot your study. A pilot participant is somebody that runs through your studies before the actual participants. And the main goal of a pilot participant is to work out the kinks in the study design. Also for a usability study, pilots can find really obvious useability bugs that you wouldn't want to burn real participants on. In fact, I recommend doing two pilots. One with a colleague who may know the interface well just as a way of getting all of your materials ready. Figuring out what the tasks are, all that kind of stuff. Then, do another pilot with one real user and that will help you see a bunch of these kinks that you didn't see ahead of time. I promise you that if you run a pilot, you'll discover things that otherwise would have screwed up the study. And how are you gonna capture the results? There's a number of ways of getting feedback from people. At the very least I recommend having a paper notebook or a laptop or other note taking device handy so that you can note down critical incidents, either aha moments that you have about how to improve the design, things that work really well, stories that you wanna share with other stakeholders or problems that you'll need to bring back to the design team to try and figure out how to fix. You might record video and a great reason to record video is it will let you grab those moments of either success or real challenge and let you share those with other people. Depending on what you're doing, screen recording can serve much the same purpose. It's going to depend on whether you want to gather the user's facial expression or the crisp contents of the user interface. Are you going to interrupt participants? If you have a question to ask them about why they are doing what they are doing, if you want to talk to them about some feature of the user interface or if you'd like to help them get past a stumbling block so that they can get onto the next thing. There's no universal answer to this question. It really depends on what you're hoping to get out of the study. And finally, are you going to ask users to think aloud while they go through the study? The think aloud method can be a really powerful way of getting feedback about a user interface. And here's what I mean by think aloud. Say, for example, you are studying how somebody changes the staples in a stapler. If I were a participant in that study [noise] and I were using the think-aloud method, I might start out by saying, well, I'm changing the staples in the stapler because it, it seems to be out. It's not, it's not working anymore. So, and see I haven't used this stapler before. I'm going to, oh, there we go. Okay, so I grabbed the top. I thought that might work and that opened it up. And I have my staples here. And I'm gonna put those right in there. And make sure this part goes forward. Oops, that didn't go. There we go. And then try and close it without getting a staple out. There we go. That's think aloud. And it helps you get a window into how people are thinking about the task they're doing. It's not something that most of us are used to doing. And so you'll need to prompt people at the beginning of the study to think as they're going. And then you'll need to offer prompts during the study as well. Prodding people to remember to say what they're thinking or what they're trying to do or any questions that arrives. Or even what it is they're reading on the screen. So, remember to prompt people to keep talking. And decide ahead of time what things you will help on and what things you won't. If you can, write that down. Consistency is h elpful. Is thinking aloud always going to give you the right answers? No. Among other things, if you gotta talk while you're doing the task, that may change how you do the task. In some cases, it may mean that you're more effective because by trying to bubble up what it is that's the issue, that may help you get through it. In another case, it may make you less effective because all of a sudden you have to do two separate things, work the interface, and speak up simultaneously. And at the very least it's almost certainly going to slow you down. And you shouldn't take what people say as a veridical representation of what they're thinking. We can almost always get a rationale or reason for something even if it isn't at all the actual reason. In studies like this, it's usually not because people are being malicious or trying to hide anything. It's usually just because, it's usually just because they're trying to help you out and generate an answer. Oftentimes those answers are useful for your design process. But they may or may not be what's actually going on in your doodle. When people arrive, the first thing that you want to do is greet them. So, the facilitator is going to welcome the participant, explain what the setup is, explain the overall narrative of the scenarios that they'll do. And get them setup and trained, if necessary, with the interface. During the studies, you wanna collect both processed data and bottom-line data. The processed data tends to be more qualitative and tends to have real insights to it. The bottom-line data tends to be more of the numbers that the, you will then use for more quantitative analysis. Bottom-line numbers are great for things like task completion time. Did my interface speed people up or increase completion rate? But, don't combine it with think aloud. The reason, think aloud slows you down and it slows different people down, different and so, the variance in task completion time, introduced by think aloud means that it just doesn't make that much sense anymore. So, if yo u care both about the numbers and about the think aloud data, you may wanna run those on separate studies. At the end of the study, debrief the participant, bot, so they learn what your goals were and they can find out more information about what you're trying to do. And also, so that you can learn more holistically from tham after the completion of everything, what it is that they're thinking. And now that you're through this study, in the next video we'll move on to analyzing the data.
