usability study methodology

14 Jun 2005 - 1:15pm
9 years ago
2 replies
509 reads
Vishal Subraman...
2005

I'm conducting a usability study of a U.S army corps of engineers website.
The most important component is a task analysis. In this case we sit along
with the subject and have a dialogue with them as they go along with the
task (talk to them about what they are thinking, try getting their thoughts
out), and give them hints if they are stuck etc, until they get to the
result (all info is captured in the coding sheets).

Another research I recently noticed used a diametrically different approach.
They just give the task to the subjects and left them on their own. In this
case there would be success/failure to get to the result etc. No hints were
given and the users could give up when they wanted.

Which of these approaches would you prefer? In the first case, the data
collected would be really rich and deep...but it isn't natural becauase of
all the hints and dialogues. Any other de/merits of either methodology?

Vishal Iyer
http://www.blog.vishaliyer.com

Comments

15 Jun 2005 - 9:13am
Mitchell Gass
2004

At 11:15 AM 6/14/2005, Vishal Subramanian Iyer wrote:
>I'm conducting a usability study...we sit along with the subject and have
>a dialogue with them as they go along with the task (talk to them about
>what they are thinking, try getting their thoughts out), and give them
>hints if they are stuck etc, until they get to the result...

This is the most common approach for diagnostic testing, where you're
looking for ways to improve a design.

>...I recently noticed used a diametrically different approach. They just
>give the task to the subjects and left them on their own. In this case
>there would be success/failure to get to the result etc. No hints were
>given and the users could give up when they wanted.

This is what's needed for comparative testing, where you're measuring the
usability of a product, usually when it's finished, in a way that lets you
make statistically significant comparisons with other versions of the same
product or with other similar products. You do this when you're looking for
numbers, not insights. For an overview of this, see

http://www.useit.com/alertbox/20010121.html

In comparative testing, having participants think aloud is generally
avoided because it can affect success measures, in particular the time it
takes to complete a task.

Mitchell Gass
uLab | PDA: Learning from Users | Designing with Users
Berkeley, CA 94707 USA
+1 510 525-6864 voice
+1 510 525-4246 fax
http://www.participatorydesign.com/

21 Jun 2005 - 10:47am
Ted Boren
2005

Sorry I'm late jumping in on this one, but there's a good article on this topic by Howard Tamler. See "How (much) to Intervene in a Usability Testing Session" at: http://www.htamler.com/papers/intervene/ .

Ted Boren

*----------
I'm conducting a usability study of a U.S army corps of engineers website.
The most important component is a task analysis. In this case we sit along
with the subject and have a dialogue with them as they go along with the
task (talk to them about what they are thinking, try getting their thoughts
out), and give them hints if they are stuck etc, until they get to the
result (all info is captured in the coding sheets).

Another research I recently noticed used a diametrically different approach.
They just give the task to the subjects and left them on their own. In this
case there would be success/failure to get to the result etc. No hints were
given and the users could give up when they wanted.

Which of these approaches would you prefer? In the first case, the data
collected would be really rich and deep...but it isn't natural becauase of
all the hints and dialogues. Any other de/merits of either methodology?

Vishal Iyer
http://www.blog.vishaliyer.com

------------------------------------------------------------------------------
This message may contain confidential information, and is
intended only for the use of the individual(s) to whom it
is addressed.
------------------------------------------------------------------------------

Syndicate content Get the feed