Any idea how many of the 14,000 IABC members think the usability of IABC.com is attractive enough to bring them to his words?

Aside from that — a good PR person could write a list of questions, go visit a site, and do a serious and valuable usability test all by himself, or, in the case of a woman, herself.

As Todd alludes to in his original posting above, there are definitions at play here. There’s usability, and then there’s usefulness, and then there’s attractiveness (in the sense of whether anyone even wants to go to the site), and then there’s readability.

BAK

You can learn a lot about how easy a site is to use by testing as few as five users IF you:
1. Prepare the test correctly
2. Recruit the right type of user (Do they accurately reflect your most important user profile for your site, etc.) and
3. Collect and interpret the collected data correctly.

That being said, such a small sample size is only acceptable for certain types of tests.

The link that commenter Tim Hicks points to above is a great reference. (www.useit.com/alertbox/20000319.html)

A detailed follow-up is worth noting here from Kath Straub at Human Factors International, (www.humanfactors.com/downloads/may042.htm#susan)

Basically Nielsen says that you only need five subjects for a usability task test. He makes a compelling point that there is a diminishing level of return for the number of “problems” found. However, testing only 5 users has its own set of risks. Would you find more “problems” if you tested more people? Most certainly, but once you get beyond 10 or 12 test subjects, you experience a very small return on your investment.

Going beyond 5 test subjects will help cover your bases should a test subject fail to show up (it happens more than you may think), and help ensure that you collect a more representative sample of your target user profile.

For example, say you wanted to see how intuitive or easy to use an e-commerce transaction is on your site. A “classic” usability test would be to ask a test subject to complete a transaction and then observe what they do to complete the transaction. In this case, the research team is looking for where the “problems” are or inefficiencies in completing the tasks. You really don’t need to test 50 users to uncover where the difficulties are. You really only want to discover usability problems rather than quantify how frequently users would encounter those problems. Observing a small sample of users is enough to give a design/development team enough information to make decisions and set priorities on how to improve their site.

The other curveball here is that you need at least 5 and no more than 12 test subjects for EACH user profile. Most websites have more than one target profile for site visitors. For example, if you had a home improvement site, you may have a audience segment of trade professionals and another audience segment for novice do-it-yourself-ers, those segments will have different expectations for your site. You’ll need to run the test for both segments. Defining these profiles and segments carefully is essential for an effective site.

So how many user profiles do you need for your site? That’s an entirely different question for another post to cover in detail. (But I will give you a hint: if you have more than 3, you may be over-thinking things.)

In a mature and effective production environment, end-user tests are being run quickly and efficiently throughout the entire production phase. Though a test subject volunteer may only experience a single test, the best production teams are collecting end-user data in multiple ways and continuously through the project cycle.