Scenarios and Task in usability testing

24 Mar 2008 - 3:43pm
6 years ago
9 replies
301 reads
Lada Gorlenko
2004

> I do not have a particular
> strategy to make scenarios, and scenarios and tasks are the same for me.

Typically, SCENARIOS are treated as end-to-end user activities that lead
to achieving a specific user goals and TASKS are individual actions
within these activities. For example:

Goal: Feed myself before going to work

Scenario: Make a breakfast
Task 1: Pour water into a kettle
Task 2: Put the kettle on
Task 3: Put cereal in a bowl
Task 4: Get milk from a fridge

... and so on.

> I only look for what users would like to do in my Website, and adapt my
> scenarios for these tasks.

What are the goals that users are trying to achieve on your website?
Potential scenarios:

Scenario 1. Get information about a product
Task 1: Find if the product is featured on this site
Task 2: Find where the necessary product info is located
Task 3: Check if the info is enough
... and so on.

Scenario 2: Buy a product
Task 1: Find if the product is available
Task 2: Find product info necessary to make buying decision
Task 3: Complete paykment form
... and so on.

> What could be the right way to find potentials errors in an interface?

This is a VERY abridged response; you should understand that a remotely
sensible reply is worth many pages.
1. Identify target user audiences of your website.
2. Identify Top 5 user scenarios for each target audience
3. Break the top scenarios down into logical tasks
4a. Get users of relevant profiles to perform the scenarios
4b. If you cannot do user evaluations, recruit usability experts instead
to do expert evaluations
5. Get results on (at least):
- Task success rate: what proportion of users have succeded in
performing scenarios and individual tasks
- Time on task: how long it took users to perform each task (vs. how
long they should spend on a task on this kind)
- Satisfaction: subjective data on whether your users were satisfied
witrh the experience (they can been objectively successful but hate the
process)

Lada

>
>
>
> Thanks,
>
> Chiwah
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help

Comments

24 Mar 2008 - 4:00pm
Marijke Rijsberman
2004

Chiwah,

It's useful to make a distinction between scenario and task if you associate
not only goals with the scenario but also "fit"--fit with the user's goal/s,
context, and desires.

At the task level it can be appropriate to figure out whether people are
able to do X and it's easy to think of quantitative measures to capture
their ability to do X. At the scenario level, which is in most cases a
million times more important, quantitative measures are more of a stretch.
You can ask people to rate things on a Likert scale, of course, but the
richness comes from the qualitative data. Better to have your team observing
than to try to translate findings into something quantitative.

Marijke

24 Mar 2008 - 7:16pm
Chauncey Wilson
2007

Lada had some excellent comments about task and scenario. Kuutti
(1995) notes that the key aspects of all scenarios are that (1) they
describe a process leading to a goal and (2) they explain a system
from the perspective of a user or other stakeholder. While we are
often focused on user scenarios, stakeholder scenarios (customer, IT
manager, tech support within a company) can also be quite useful. One
additional note is that there are different categories of scenarios.
Here is a list that I extracted from the CHI literature.

1. Misuse scenarios. Misuse scenarios document threats toward a
system that involve:
-- Hostile actors (vandals who put superglue in vending machine
slots, for example)
-- Safety problems
-- User goals that are not compatible with system goals
2. Exception scenarios (these looks at the rare events or interactions)
3. What-if scenarios (to explore future possibilities)
4. Alternative scenarios - different ways to do the same thing.
5. Making-sense scenarios -- These scenarios show people trying to
understand how to use, troubleshoot, and maintain a system
6. Technology scenarios -- Scenarios that highlight possibilities for
novel technology like haptic interfaces or smart badges.
7. Concept of operation -- A concept of operation is a set of
high-level scenarios that describes in conceptual terms (without
details of the operating system or specific technologies) the general
operation of a complex system (for example, a rail system or
powerplant).
8. Alternative world scenario -- An imaginary situation; sometimes
used as input to a simulation.
9. Organizational Scenarios -- Organizational scenarios are those that
depict potential consequences for groups with different people in
multiple roles. For example, what happens when you introduce a
technology that eliminates some jobs or puts different people in a
position of power.
10. Scenario Timelines - McGraw and Harbison (1997) describe a
technique, the scenario timeline, for visualizing and analyzing
activities involving multiple actors over time.

Some excellent references are:

Alexander, I. (2004). Negative scenarios and misuse cases. In I. F.
Alexander, & N. Maiden (Eds.) Scenarios, stories, use cases through
the systems development life cycle. New York, NY: Wiley, pp. 119-139.
Alexander, I. F., & Maiden, N., (Eds.) (2004). Scenarios, stories, use
cases through the systems development life cycle. New York, NY: Wiley.
Carroll, J. M. (Ed.) (1995). Scenario-based design: Envisioning work
and technology in system development. New York, NY: Wiley.
Carroll, J. M. (1997). Scenario-based design. In M. G. Helander, T. K.
Landauer, and P. V. Prabhu (Eds.). Handbook of Human-Computer
Interaction (Second Edition). Amsterdam, The Netherlands: Elsevier.
Pp. 383-406.
Carroll, J. M. (2000). Making use: Scenario-based design of
human-computer interactions. Cambridge, MA: The MIT Press.

Chauncey

On Mon, Mar 24, 2008 at 5:00 PM, Marijke Rijsberman
<marijke at interfacility.com> wrote:
> Chiwah,
>
> It's useful to make a distinction between scenario and task if you associate
> not only goals with the scenario but also "fit"--fit with the user's goal/s,
> context, and desires.
>
> At the task level it can be appropriate to figure out whether people are
> able to do X and it's easy to think of quantitative measures to capture
> their ability to do X. At the scenario level, which is in most cases a
> million times more important, quantitative measures are more of a stretch.
> You can ask people to rate things on a Likert scale, of course, but the
> richness comes from the qualitative data. Better to have your team observing
> than to try to translate findings into something quantitative.
>
> Marijke
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

25 Mar 2008 - 7:14pm
Anonymous

Hello,

I don't have a lot of experiences about user testing, but shouldn't it be
better if the finding are somewhat qualitative ?
I mean, if a finding councern only one user and never happen to any other
users. Would that still be usefull to take it into account ?

Chiwah

2008/3/24, Marijke Rijsberman <marijke at interfacility.com>:
>
>
> You can ask people to rate things on a Likert scale, of course, but the
> richness comes from the qualitative data. Better to have your team
> observing
> than to try to translate findings into something quantitative.
>
>
> Marijke
>
>

26 Mar 2008 - 2:46am
Anonymous

Thank you for this answer. I have the feeling that scenarios are the key for
a successful usability test, and I really would like to have all the tools
to succeed my usabilities tests.

I still have a lot of questions about the right way to do my scenarios and
to analyze them.

- - For example, I usually start most of my scenarios from the Home
page, but when my users have 5 scenarios to do and start 5 times from the
home page, this might constitute a biasis.

- - Or the measure of the satisfaction, should I measure it at the
end of each scenario, or at the end of all scenarios?

- - Should scenarios be contextual? For example, rather than saying
"get information about a product" should I say "you would like to go to
holiday with your family and you are looking for the cheapest ticket…"

- - Is there any validity about the time on task if the user is
doing a "thinking aloud" ?

Or maybe there is some litterature about scenarios methodology? Then I won't
ask too much about this on this mail list.

Thank you

Chiwah

26 Mar 2008 - 2:47am
Anonymous

> Hello,
>
> Thank you for your answer. I take the opportunity to ask you some
> methodological aspect of scenarios
>
> - because we have time and budget constraints, we have group of user
> (about 6-8 users) that performs together scenarios, each user have his own
> computer and a facilitator gives the instruction to the users. I know it's
> very quick and dirty, but is there any value to this kind of scenarios ?
>
> - If I try to convince my team that an 1 on 1 scenario (1 user and 1
> facilitator at a time) would be a lot better, even if it is much more time
> consuming, would that worth the change ?
>
> Thank you a lot,
> Chiwah
>
>

26 Mar 2008 - 9:45am
Eva Kaniasty
2007

Chiwah,

Are all the participants in one room at the same time? The problem that you
will run into is twofold:

1) it will be difficult for all the users to do think-aloud at the same
time; if you have a group discussion, then you will have to contend with
group dynamics, and will likely get different results than you would if you
spoke to participants one on one.
2) Even if you videotape the session, it will be much harder for you to
analyze the results, as you will not be able to see each participants'
screen as they are performing the tasks.

I would recommend that you do try to convince your client to do a one-on-one
test. Otherwise you may find that there is limited value to the data that
you gather.

As you are finding out, designing and moderating usability tests is a
complex subject. If there is anybody in your company who is familiar with
usability testing, then they could be an invaluable resource in advising
you on the best approach and/or methodology.

For further reading, I would recommend the Joe Dumas book below as a
starting point.

http://www.amazon.com/Practical-Guide-Usability-Testing/dp/1841500208/ref=sr_1_2?ie=UTF8&s=books&qid=1206542052&sr=1-2

--
Eva Kaniasty
http://www.linkedin.com/in/kaniasty

26 Mar 2008 - 11:19am
Marijke Rijsberman
2004

Chiwah,

There's always a tendency to express usability findings in quantitative
terms, which lends the results a factitious air of rigor and reliability.
However, if 1 participant out of 10 has a certain problem in a usability
test, you really don't know how common it is among the target audience. And
if 10 participants out of 10 experience the problem you don't know how
common it is either.

Neither the numbers nor the recruiting procedures in the standard usability
test support generalizations about the population at large or any subset of
it. And the testing situation itself is normally sufficiently different from
real life that you can't be sure that what you're seeing in the lab isn't
your own creation. All of which means you have to use your judgment and work
with your team to reach some kind of consensus on what you're going to
change in the product once you've seen some people interact with it. Is the
problem that one user ran into serious and plausible enough to expend
resources to fix it?

When trying to answer that question, you might consider the following:

- How seriously did this issue impact the overall experience? is it
a minor irritant or does it make your product useless altogether whenever it
occurs?

- Is the problem part of a common task or scenario or is it more of
an exception?

- Did the participant strike you as a normal, plausible person who
was reasonably representative of your users or did you have some reason to
suspect the person was an outlier?

You might of course always decide you need more data before you do anything
about it, especially if it isn't obvious how to fix the problem without
giving rise to other problems..

Marijke

--------------------

>I don't have a lot of experiences about user testing, but shouldn't it be
better if the finding are somewhat qualitative ?
>I mean, if a finding councern only one user and never happen to any other
users. Would that still be usefull to take it into account ?

Chiwah

26 Mar 2008 - 5:22pm
Anonymous

Hello Marijke,

I have recently seen a Microsoft methodology for user testing, it is called
RITE : Rapid Iterative Testing and Evaluation.
It is like an usual test with scenarios, but they test only one user at a
time. They test one user and immediatly from the finding they adapt the
design. And they do it once every 2 or 3 days.

So it's not 5-10 users testing once, it's 1 user testing then iterate 5-10
times. I am very afraid about false positive here.

Have you ever tried it ? what do you think about this methodology ?

Chiwah

2008/3/26, Marijke Rijsberman <marijke at interfacility.com>:
>
>
> However, if 1 participant out of 10 has a certain problem in a usability
> test, you really don't know how common it is among the target audience. And
> if 10 participants out of 10 experience the problem you don't know how
> common it is either.
>
>
>

26 Mar 2008 - 5:31pm
Anonymous

Hello Eva,

It is effectively all the participants in one room at the same time. It
allow both user testing and focus group... to minimize the cost, the time
necessary to analyze the data and maximize the amount of information. Even
if it is a little biaised.

I call it a "Quick, cheap and dirty" test. We have a bit of everything than
only usability test or perception test.

I will meet them tomorrow to discuss about the pro and the con for each
methodology. I think that time, cost, quality of finding will be at the
center of our discussion.

Thank you,
Chiwah

2008/3/26, Eva <kaniasty at gmail.com>:
>
> Chiwah,
>
> Are all the participants in one room at the same time?
>

Syndicate content Get the feed