Usability Testing Challenge

18 Feb 2009 - 11:21am
5 years ago
7 replies
834 reads
Bender
2009

Hi all,
I am a UI designer for a start up company and I'm conducting a
usability test for our web product for the first time.

I've read a few books including "handbook of usability testing"
and "a practical guide to usability testing", and I've read a ton
of resources online.

The product we are testing is a web widget which is going to be
integrated in other websites (blogs, portals, etc.). We have a
working prototype where we have embedded the widget in a fully
functional mocked-up website.

We are interested in testing two things:
1- Whether or not people will notice our widget at all and will
interact with it
2- Knowing what the widget is, how easily can they use the features
the widget offers

The success of the product largely depends on the first point. The
problem is that its almost impossible to write tasks for that.

So I have designed the test to have 2 parts: in the first part, I
want to give the user about 10 minutes to free-roam the website, and
think-out-loud, to see how long it would take for them to notice the
widget. I also want to observe how long it will take for them to
interact with the feature, and see if they understand (and use) the
features of the widget on their own.

In part 2, I will give them particular tasks related to features of
the widget and see how well they can interact with the UI (if they
haven't figured out by now what the widget is, I will tell them)

The only issue is, they might have already done some of the tasks in
part 2 when they were free-roaming in part 1, and I'm not sure how
to handle that.

So, for the usability experts here, do you think this is a good
approach? I would definitely appreciate your advice and comments.

Comments

18 Feb 2009 - 11:50am
krushford
2008

Hello,

I work on similar usability labs and my task lists always start with
some exploratory tasks. Even if they may do some of the widget
related tasks during that period, I would still ask the specific tasks
associated with the goals of the widget.
Hope that helps and good luck!
Kaden

On Wed, Feb 18, 2009 at 11:21 AM, pendar <legofish at legofish.com> wrote:
> Hi all,
> I am a UI designer for a start up company and I'm conducting a
> usability test for our web product for the first time.
>
> I've read a few books including "handbook of usability testing"
> and "a practical guide to usability testing", and I've read a ton
> of resources online.
>
> The product we are testing is a web widget which is going to be
> integrated in other websites (blogs, portals, etc.). We have a
> working prototype where we have embedded the widget in a fully
> functional mocked-up website.
>
> We are interested in testing two things:
> 1- Whether or not people will notice our widget at all and will
> interact with it
> 2- Knowing what the widget is, how easily can they use the features
> the widget offers
>
> The success of the product largely depends on the first point. The
> problem is that its almost impossible to write tasks for that.
>
> So I have designed the test to have 2 parts: in the first part, I
> want to give the user about 10 minutes to free-roam the website, and
> think-out-loud, to see how long it would take for them to notice the
> widget. I also want to observe how long it will take for them to
> interact with the feature, and see if they understand (and use) the
> features of the widget on their own.
>
> In part 2, I will give them particular tasks related to features of
> the widget and see how well they can interact with the UI (if they
> haven't figured out by now what the widget is, I will tell them)
>
> The only issue is, they might have already done some of the tasks in
> part 2 when they were free-roaming in part 1, and I'm not sure how
> to handle that.
>
> So, for the usability experts here, do you think this is a good
> approach? I would definitely appreciate your advice and comments.
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

18 Feb 2009 - 12:33pm
Dana Chisnell
2008

Hi Pendar,

> I've read a few books including "handbook of usability testing"
> and "a practical guide to usability testing", and I've read a ton
> of resources online.

I am so psyched that you've read Handbook of Usability Testing. I hope
it was the second edition. I was one of the authors. (Oh gawd, I hope
I give you good advice.)

> We are interested in testing two things:
> 1- Whether or not people will notice our widget at all and will
> interact with it
> 2- Knowing what the widget is, how easily can they use the features
> the widget offers

These are great questions. Setting up an exploratory test is exactly
what I would do.

> So I have designed the test to have 2 parts: in the first part, I
> want to give the user about 10 minutes to free-roam the website, and
> think-out-loud, to see how long it would take for them to notice the
> widget. I also want to observe how long it will take for them to
> interact with the feature, and see if they understand (and use) the
> features of the widget on their own.
>
> In part 2, I will give them particular tasks related to features of
> the widget and see how well they can interact with the UI (if they
> haven't figured out by now what the widget is, I will tell them)
>
> The only issue is, they might have already done some of the tasks in
> part 2 when they were free-roaming in part 1, and I'm not sure how
> to handle that.
>
> So, for the usability experts here, do you think this is a good
> approach? I would definitely appreciate your advice and comments.

If your participants do the tasks that you have in mind for part 2 in
part 1 without your prompting them, I'd say your site and widget are
really successful. Don't try to force-fit the test. Go with the flow.

If that happens, you're still finding out what you want to know, which
I think are these hidden questions (tell me if I'm wrong):

Is the idea of the widget useful?
Is it in the right place to be found?
Do people understand what it is when they see it?
Is the call to action of the widget strong enough that people click
through?

I hope you've recruited participants who are motivated to do what you
want to observe -- that is, they've demonstrated somehow that they've
done something before that is related to what you're exploring in this
test, or they've volunteered a desire for the feature that the widget
offers.

Although you're talking about "how long it takes," I would think twice
about actually timing people for this test. Time won't tell you a lot
at this point. Might be better to track what participants click on
their way to interacting with the widget. That data can help you think
about where to position the widget or entry points to it. Also, if
you're asking people to think aloud, talking will slow them down (not
necessarily a bad thing). I've written about this in the blog that
accompanies Handbook of Usability Testing, here: http://usabilitytestinghowto.blogspot.com/2007/04/when-to-ask-participants-to-think-out.html

Hope this helps. Let us know how it turns out.
Dana

:: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: ::
Dana Chisnell ~ Be good. UsabilityWorks.
desk: 415.392.0776
mobile: 415.519.1148

dana AT usabilityworks DOT net

www.usabilityworks.net
http://usabilitytestinghowto.blogspot.com/

18 Feb 2009 - 12:35pm
Paul H Greasby
2009

I would urge you to not focus so much on the 'time' element unless
they take excessive time to notice or do things. You're more
interested in their interpretation and actions based on the
'think-aloud' method, and you need to focus on this more than time
along IMO.

Allowing users time to explore is good. This is a compromise though.
If you subsequently give them a task, they will be more familiar with
the interface and will be looking at it with a more analytical eye
than a regular user. Therefore they may more easily discover a
feature because they had more time to fully explore it than in an
operation scenario.

Your approach however looks good and sensible. Good luck

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=38904

18 Feb 2009 - 12:48pm
Ron Perkins
2007

You've got the right approach. After the open ended part of the session,
you'll know whether the widget stands out enough to be noticed on it's own
merit.

Then stop and ask whether the test participant ever does 'x' where x is the
thing that the new widget does. 'x' is not the widget name, only the
essence of what it does...

After you establish whether 'x' is interesting, then ask them now to do 'x'.

If they still cannot find it, that's a problem.

After that even if you have them repeat things that the widget does, it's
often good to see if they use it as designed and understand it or if they
just got lucky on the first try.

Best of luck,
Ron

Ron Perkins, Principal
DesignPerspectives.com
Usability and Interaction Design Consulting

-----Original Message-----
From: krushford [mailto:krushford at gmail.com]
Sent: Wednesday, February 18, 2009 11:50 AM
To: pendar
Cc: discuss at ixda.org
Subject: Re: [IxDA Discuss] Usability Testing Challenge

Hello,

I work on similar usability labs and my task lists always start with some
exploratory tasks. Even if they may do some of the widget related tasks
during that period, I would still ask the specific tasks associated with the
goals of the widget.
Hope that helps and good luck!
Kaden

On Wed, Feb 18, 2009 at 11:21 AM, pendar <legofish at legofish.com> wrote:
> Hi all,
> I am a UI designer for a start up company and I'm conducting a
> usability test for our web product for the first time.
>
> I've read a few books including "handbook of usability testing"
> and "a practical guide to usability testing", and I've read a ton of
> resources online.
>
> The product we are testing is a web widget which is going to be
> integrated in other websites (blogs, portals, etc.). We have a working
> prototype where we have embedded the widget in a fully functional
> mocked-up website.
>
> We are interested in testing two things:
> 1- Whether or not people will notice our widget at all and will
> interact with it
> 2- Knowing what the widget is, how easily can they use the features
> the widget offers
>
> The success of the product largely depends on the first point. The
> problem is that its almost impossible to write tasks for that.
>
> So I have designed the test to have 2 parts: in the first part, I want
> to give the user about 10 minutes to free-roam the website, and
> think-out-loud, to see how long it would take for them to notice the
> widget. I also want to observe how long it will take for them to
> interact with the feature, and see if they understand (and use) the
> features of the widget on their own.
>
> In part 2, I will give them particular tasks related to features of
> the widget and see how well they can interact with the UI (if they
> haven't figured out by now what the widget is, I will tell them)
>
> The only issue is, they might have already done some of the tasks in
> part 2 when they were free-roaming in part 1, and I'm not sure how to
> handle that.
>
> So, for the usability experts here, do you think this is a good
> approach? I would definitely appreciate your advice and comments.
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org Unsubscribe
> ................ http://www.ixda.org/unsubscribe List Guidelines
> ............ http://www.ixda.org/guidelines List Help
> .................. http://www.ixda.org/help
>
________________________________________________________________
Welcome to the Interaction Design Association (IxDA)!
To post to this list ....... discuss at ixda.org Unsubscribe ................
http://www.ixda.org/unsubscribe List Guidelines ............
http://www.ixda.org/guidelines List Help ..................
http://www.ixda.org/help

No virus found in this incoming message.
Checked by AVG.
Version: 7.5.552 / Virus Database: 270.10.25/1957 - Release Date: 2/17/2009
7:07 AM

18 Feb 2009 - 2:14pm
Bender
2009

Thank you krush and pgreasby for the wonderful feedback. As a
usability testing virgin I do appreciate the tips.

Dana, it is a wonderful surprise to read your reply. Indeed I read
the second edition and found the book very insightful, using some of
its templates in conjunction with other to develop a test plan (still
in progress).

You are on the spot regarding the questions we are trying to address

We still havent recruited the users, we're doing that this week. Our
target user group is very broad (pretty much anyone who visits
blogs/portals/etc.), so I am only making a distinction between light
and power internet users.

We're also going to test two versions of the prototype (A & B). So
my strategy is to use 12 testers, 6 light users and 6 power uers
(nicely varied age group). And then within each group, have 3 users
test version A and 3 users test version B. So at the end, we would
have 6 users who have tested A and 6 who have tested version B.

I will definitely take note of your advice about not worrying too
much about tracking time. I chose that as a measure rather
arbitrarily, mostly because I thought if most users don't notice it
within 30-40 seconds, then in a real-life scenario people would have
probably gone on to another page already and have totally missed the
widget.

Anyway, thanks again for your replies, they are extremely helpful

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=38904

18 Feb 2009 - 2:43pm
Bender
2009

Thank you Ron, very useful insight.

My strategy was to keep the communication between the user and the
test facilitator fairly open during the open-ended session and have
the facilitator probe the user as the user does the stuff. For
example, immediately after the user first interacts with the widget,
I was going to pose the question "what do you think the widget
does?", and so on.

After reading your insight I'm wondering if it's better pose the
questions after the open-ended part is done, and before starting the
task-based part of the test.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=38904

18 Feb 2009 - 11:19pm
Ron Perkins
2007

I like to ask as few questions as possible when someone is interacting
to keep them from artificially paying attention and thinking about
things-- doing is more revealing than talking.
Ron

Sent from my iPhone

On Feb 18, 2009, at 11:43 AM, pendar <legofish at legofish.com> wrote:

> Thank you Ron, very useful insight.
>
> My strategy was to keep the communication between the user and the
> test facilitator fairly open during the open-ended session and have
> the facilitator probe the user as the user does the stuff. For
> example, immediately after the user first interacts with the widget,
> I was going to pose the question "what do you think the widget
> does?", and so on.
>
> After reading your insight I'm wondering if it's better pose the
> questions after the open-ended part is done, and before starting the
> task-based part of the test.
>
>
>
>
>
>
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Posted from the new ixda.org
> http://www.ixda.org/discuss?post=38904
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help

Syndicate content Get the feed