User-focused Web design and sample size
Google “Web site usability,” “user-centric Web design,” “user-focused Web design,” … and you’ll get an endless list of definitions, agencies offering expertise, descriptions of how to achieve true user-centric design. I’m all for user-focused (or whatever label you want to give it) Web design. With the success of Web 2.0, the only way Web site owners will be able to attract and engage users is to design from the users’ experiences and perspectives.
For anyone who isn’t familiar with or has not yet experienced user-focused Web design, Web guru and “father of Web usability” Jacob Nielsen offers several resources.
Several years ago, I learned about Web site usability when I was asked to be a participant. At the time, I was a bit surprised that the hired consultants had engaged three groups of six to eight participants. A few years ago, my agency hired consultants to conduct usability assessment on our agency site. Again, I was surprised by the small audience sample from which dramatic conclusions were drawn. This past month, a collegue indicated that his agency had just completed their preliminary site analysis with customer representatives. Yup, you guessed it. Very small sample size.
Having been involved in and conducted many audience assessments related to a variety of communication programs, I understand the importance of establishing statistical validity. One way you do that is by obtaining response numbers that are very carefully calculated. What makes Web usability assessments different? Why are such small samples acceptable?
August 16th, 2007 at 10:53 am
Nielsen answers this himself at http://www.useit.com/alertbox/20000319.html
August 16th, 2007 at 12:11 pm
It all depends on what the research team was trying to learn. Sample sizes for different tests will vary greatly.
You can learn a lot about how easy a site is to use by testing as few as five users IF you:
1. Prepare the test correctly
2. Recruit the right type of user (Do they accurately reflect your most important user profile for your site, etc.) and
3. Collect and interpret the collected data correctly.
That being said, such a small sample size is only acceptable for certain types of tests.
The link that commenter Tim Hicks points to above is a great reference. (www.useit.com/alertbox/20000319.html)
A detailed follow-up is worth noting here from Kath Straub at Human Factors International, (www.humanfactors.com/downloads/may042.htm#susan)
Basically Nielsen says that you only need five subjects for a usability task test. He makes a compelling point that there is a diminishing level of return for the number of “problems” found. However, testing only 5 users has its own set of risks. Would you find more “problems” if you tested more people? Most certainly, but once you get beyond 10 or 12 test subjects, you experience a very small return on your investment.
Going beyond 5 test subjects will help cover your bases should a test subject fail to show up (it happens more than you may think), and help ensure that you collect a more representative sample of your target user profile.
For example, say you wanted to see how intuitive or easy to use an e-commerce transaction is on your site. A “classic” usability test would be to ask a test subject to complete a transaction and then observe what they do to complete the transaction. In this case, the research team is looking for where the “problems” are or inefficiencies in completing the tasks. You really don’t need to test 50 users to uncover where the difficulties are. You really only want to discover usability problems rather than quantify how frequently users would encounter those problems. Observing a small sample of users is enough to give a design/development team enough information to make decisions and set priorities on how to improve their site.
The other curveball here is that you need at least 5 and no more than 12 test subjects for EACH user profile. Most websites have more than one target profile for site visitors. For example, if you had a home improvement site, you may have a audience segment of trade professionals and another audience segment for novice do-it-yourself-ers, those segments will have different expectations for your site. You’ll need to run the test for both segments. Defining these profiles and segments carefully is essential for an effective site.
So how many user profiles do you need for your site? That’s an entirely different question for another post to cover in detail. (But I will give you a hint: if you have more than 3, you may be over-thinking things.)
In a mature and effective production environment, end-user tests are being run quickly and efficiently throughout the entire production phase. Though a test subject volunteer may only experience a single test, the best production teams are collecting end-user data in multiple ways and continuously through the project cycle.
August 16th, 2007 at 6:05 pm
Mr. Irwin worked hard on his commentary.
Any idea how many of the 14,000 IABC members think the usability of IABC.com is attractive enough to bring them to his words?
Aside from that — a good PR person could write a list of questions, go visit a site, and do a serious and valuable usability test all by himself, or, in the case of a woman, herself.
As Todd alludes to in his original posting above, there are definitions at play here. There’s usability, and then there’s usefulness, and then there’s attractiveness (in the sense of whether anyone even wants to go to the site), and then there’s readability.
BAK