Welcome back as we prepare to wrap up our course on evaluating user interfaces. I wanted to pause for a moment and talk about quantitative project management often driven evaluations, specifically usability goals and measures. To understand why we have this goals and measures, let's go back to the whole notion of the interface development that we've been learning. It's very much a prototype and iterate model. Keep iterating as we learn how make the tool better until the product is good enough. And evaluate along the way so that we're assessing regularly when is it good enough? Heck, what is good enough? Well, to answer that question we need to set some goals. These goals will probably relate to our tasks, they relate to our users. But the challenge here is to understand what kinds of goals can we define that might say this is ready to go. Now we know in software engineering generally, there are often goals defined for product functionality. We don't ship with major bugs that would cause the system to crash. We don't ship with major bugs that would cause a loss of customer data. But what do we do for usability? So remember if we started with this notion of casual iteration, let's find major problems, missing features, user confusion, poor interaction. We would start by trying that interface with specific tasks and observing the usage. We're doing this with an open supportive approach, where we're not defending the interface or biasing the tests. We're exploring, we're capturing new tasks. Every time we find a problem, we look at different ways we might improve the interface to fix that problem, or eliminate what caused the problem in the first place. But that's never going to say, stop, it's never going to say, it's not worth spending more money. And frankly, if you're in an organization that's worried about the bottom line, this kind of, wait, people are still not quite clear on our interface, probably doesn't justify delay for your products. So that's where we get to these quantitative measures. Let me give you a few examples of the types of things that we're going to measure. We'll give a few examples of the kinds of measures. We're not going to give you a magic formula for developing these measures. Because that's outside the scope of designing the interface, that's really in the scope of managing the project. But you may be able to do that in the role you have understanding the project as a whole. So the kinds of measures we might look at, learning time. How quickly can a particular set of users get to a particular level of proficiency? Sometimes that's get up and started and using. Sometimes that's complete certain tasks. Use time for particular tasks and users. I want someone to be able to pay a bill to an existing vendor. Shouldn't take more than 60 seconds. We can measure that. The users may differ. I may care more about a particular type of user. A newcomer, an experienced user, a user of my competitor's product. I may care about error rates. How long does it take error free? How many people are error free? What's the number and severity of errors I'm willing to allow? And what even is an error in the context of my particular application? If I'm writing a drawing program, I don't know that I would call it an error if somebody draws a line and says, no, that's not where I want it and moves it. That's just exploration. But if they were asked to create something and the thing is wrong at the end, that might be an error. I may also have measures of user satisfaction. Some of these are survey measures. Would you use this product again? Would you choose this product over another one? Would you upgrade? Some of the maybe much more targeted there are standard questionnaires for user interface satisfaction. I may also have comparative usability goals. How do I compare with the previous version of this product, or its competitor, or companion product that I'm trying to move people too? Come back to those in a minute. So the other thing I would say is when you're talking about quantitative. Be realistic, 100% is almost never realistic, if you comeback and say, 100% of first time users should be up and running within two minutes you will discover there are people up there who are not. They get distracted, they ran into something that you could never anticipate. Maybe your goal should be 95%, maybe it should be 85%, for somethings maybe 99. But think about what's a realistic number and if you're going to do it you need to test to it. Recognize that many of your goals extend beyond just the application user interface. If you have any form of training, help system, manuals, those may be part of achieving a goal. Doesn't make it a bad goal, it just means you can't necessarily come up with that goal and test it early if those things aren't complete. The other thing is there are many goals and the first priority for these goals is to make a good business decision. The second priority is to get some value out of testing. Ideally if you fail a goal, if you fail one of these quantitative measures, you should learn something about what's wrong with the interaction that will help you improve the interfacing you do pass. A measure that's so coarse grain that you don't know what's going on if all you measure is after 20 hours of use, people say they're willing to pay at least $99 to upgrade? And then people say, no, you don't know why, you don't know how to fix it. Having goals that include measures that allow you to understand how to fix it are important. So here are a couple of examples, this is from a specific online game app. 85% of first-time visitors should be able to start playing within 90 seconds. That's the kind of goal you can go out there and say, you know what? If we can't achieve this too many people are going to abandon this before they can play. 90 seconds might or might not be the right amount of time. You would test this as you're going and figure out what is a reasonable time after which people abandon installing or playing this particular game. For a mailing list management program, this is actually one that I run into quiet recently. 75% of mailing list administrators should be able to change the moderation settings on their mailing list in three minutes or less without googling for help or looking it up in the manual. You want people to find it in the settings. This actually came up because I was helping people who administered Google groups mailing lists. And one of the issues that came up was people could figure it out, but very frequently they couldn't figure it out without basically going into Google and typing how do I change something? And that's why there's a second goal here 95% should be able to change them in five minutes or less using available help. Because that larger context is actually very valuable. For most people, the fact that they could find the answer on Google, which pointed back into the website that had the manual, was a good enough answer. Manuals have changed with the Internet. Here's a third one. 75% of users changing their password should select a password that's valid the first time. 95% should do so the second time. Now if you remember the video where I did the Twitter password change, Twitter was really nice about this. It will tell me if I'm coming up with a password that's not very secure, But it leaves the decision to me. Many of the programs that I run from online networks to more secure things like bank accounts and email systems, have very strict rules for what the password is. But they don't always tell you what those rules are. And so this is something that says, gee, if more than a quarter of people can't looking at the screen where they type in their password come up with one that's valid? That's probably a bad thing, and I've been on systems that have all sorts of stuff that there I am, I'm using upper case, lower case, numbers. They come back and say, well, wait a minute, you also have to have punctuation. So I come up with punctuation. It says, but that was your password three times ago. You can't use that again. Well, if I remembered that was my password three times ago I wouldn't have changed it so many times since. But so many of these systems make it hard that this might be a useful usability goal. So what other features might you find. Rates of complaint, errors we mentioned, negative feedback. And then you get into things like very specific user populations. So, as I've been taping this, H&R Block has been doing a lot of advertising about their online tax preparation, their big competitor in that space is TurboTax. There are others but they mention TurboTax even in the ads. One goal they might want is that 85% of people who filed taxes with TurboTax in a previous year, can file with Block's software in the same amount of time with no more errors and without major complaints. That would be a nice selling point that says look, there's a few people who might have find this a little more difficult but most people found it as easy or easier, that would be pretty cool. That's also frankly very ambitious. It may even be unobtainable given that the software that people have used before is what they're already very comfortable with. But it's the type of thing you might consider as you're evaluating what percentage can you achieve. So you'll notice that I've always made these quantitative, if you're managing a project, if you're doing summative evaluation, an assessment that says, is it time to ship? Is this ready? Is this worth shipping with our name on it? You need to have some sort of a number. And if you just say, we're going to look at the percentage of people who do this. That's great then it comes out and says, well, 28% of people have trouble and you're back down to 28% too much. If you don't set thresholds at a certain point, you have trouble making decisions. And that's why we're doing these as summative assessments. They're focused on, is this good enough or not? It doesn't mean you can't get formative value, how can I improve it? But that's not its primary value. And again, if you're doing software engineering, if you're doing software project management, determining what these are is something you may do very early in your project stage. Where you've determined who's the audience, who are the decision makers, the points of purchase, the points that you need to keep happy. And then you're going to negotiate and develop a set of these goals to match. Hope you found that useful. In our next lecture, Lauren will be here to tie everything together. And then we'll bring you to the end of evaluation. Look forward to seeing you next time. [SOUND]
