Using a Survey to Asses Usability

10 Jun 2008 - 8:03am
6 years ago
11 replies
787 reads
tamlyn
2008

The company I'm working for is redesigning their account control
panel. We'll be releasing the functionality in stages and running it
as a beta alongside the existing control panel.

I've been set a task to write a series of questions that users will be
given the option to answer after using the new control panel to asses
"usability, accessibility & design". I've told them that what users
say and what they do are often not the same and that the only way to
do it is with user testing (which we will also be doing) but they seem
dead set on the survey idea.

Can a survey/questionnaire yield useful results in this kind of
situation? I think a simple text box for feedback and bugs would be
better. Any suggestions?

Cheers, Tamlyn.

Comments

10 Jun 2008 - 8:13am
SemanticWill
2007

You are right....

If the first commandment of UX is:
You are not the user

then the second is
The user always lies. (I am a huge fan of House :-)

Ok... that is a bit bold, but it has been proven over and over that even
when users are not being untruthful outright, they often times do not even
know what they have done - even if they did it only 10 minutes ago.

A survey combined with user observation will end up providing the most
useful and actionable information. A simple survey after completing a set of
tasks will most certainly be almost useless. Not as useless as not doing an
testing whatsoever - but certainly not provide the kind of valuable,
actionable data needed.

I highly recommend you quickly run and get "Observing the User Experience"
-- accept that book is a bit big - perhaps for something more reasonable -
get Steve Mulder's "The User is Always Right" which provides some great
arguments to use with stakeholders and decision makers about using
qualitative and quantitative user research.

- W

On Tue, Jun 10, 2008 at 9:03 AM, Tamlyn Rhodes <tamlyn at tamlyn.org> wrote:

> The company I'm working for is redesigning their account control
> panel. We'll be releasing the functionality in stages and running it
> as a beta alongside the existing control panel.
>
> I've been set a task to write a series of questions that users will be
> given the option to answer after using the new control panel to asses
> "usability, accessibility & design". I've told them that what users
> say and what they do are often not the same and that the only way to
> do it is with user testing (which we will also be doing) but they seem
> dead set on the survey idea.
>
> Can a survey/questionnaire yield useful results in this kind of
> situation? I think a simple text box for feedback and bugs would be
> better. Any suggestions?
>
> Cheers, Tamlyn.

10 Jun 2008 - 8:33am
Todd Warfel
2003

On Jun 10, 2008, at 9:03 AM, Tamlyn Rhodes wrote:

> I've told them that what users say and what they do are often not
> the same and that the only way to
> do it is with user testing (which we will also be doing) but they
> seem dead set on the survey idea.

A survey will only assess one aspect of usability — satisfaction. The
other aspects, those of time and effort will not be addressed through
a survey. In short, you're only addressing 1/3 of the usability puzzle
with a survey.

Cheers!

Todd Zaki Warfel
President, Design Researcher
Messagefirst | Designing Information. Beautifully.
----------------------------------
Contact Info
Voice: (215) 825-7423
Email: todd at messagefirst.com
AIM: twarfel at mac.com
Blog: http://toddwarfel.com
Twitter: zakiwarfel
----------------------------------
In theory, theory and practice are the same.
In practice, they are not.

10 Jun 2008 - 10:58am
Brooke Baldwin
2008

Tamlyn
A survey can be helpful, and it's not uncommon for a company to prefer
this mechanism of collecting data. You're right that it generally won't
paint as clear a picture as observations - but what is the goal that you
(or the business) are trying to accomplish by collecting this data?

Is it to determine if the user likes the changes? Perhaps questions should
be answerable with a qualitative response, i.e., 'Hated it', 'Hated it a
Little', 'Like it a little', and 'Liked it a lot'. I'd encourage you to
NOT have a middle value, but force a selection of bias with an even number
of possible answers.

If you're trying to determine something else, like was the task completed
within a reasonable period of time and to successful completion with x% of
errors - then you might want to push for testing with users.

Good luck!
brooke

Tamlyn wrote:

> Can a survey/questionnaire yield useful results in this kind of
> situation? I think a simple text box for feedback and bugs would be
> better. Any suggestions?

10 Jun 2008 - 12:03pm
Alexander Baxevanis
2007

Just sat (on the observation side of the mirror) on a couple of user
tests for the quoting flow in an insurance website. There were quite a
few issues with the forms: validation messages being missed,
incomprehensible questions, help not immediately available etc. Yet
all the users who struggled, even those who didn't manage to get a
quote in the end, also said that they were quite happy with the whole
process. Hope that answers your question :)

Cheers,
Alex

On Tue, Jun 10, 2008 at 2:03 PM, Tamlyn Rhodes <tamlyn at tamlyn.org> wrote:
> The company I'm working for is redesigning their account control
> panel. We'll be releasing the functionality in stages and running it
> as a beta alongside the existing control panel.
>
> I've been set a task to write a series of questions that users will be
> given the option to answer after using the new control panel to asses
> "usability, accessibility & design". I've told them that what users
> say and what they do are often not the same and that the only way to
> do it is with user testing (which we will also be doing) but they seem
> dead set on the survey idea.
>
> Can a survey/questionnaire yield useful results in this kind of
> situation? I think a simple text box for feedback and bugs would be
> better. Any suggestions?
>
> Cheers, Tamlyn.
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

10 Jun 2008 - 12:07pm
SemanticWill
2007

Go quickly sir, get LukeW's book on Form Design, and enlighten the heathen
masses! Good, actionable discussions on form design, design principles, and
design patterns -- all focused on form design. Worth the cash!

- W

On Tue, Jun 10, 2008 at 1:03 PM, Alexander Baxevanis <
alex.baxevanis at gmail.com> wrote:

> Just sat (on the observation side of the mirror) on a couple of user
> tests for the quoting flow in an insurance website. There were quite a
> few issues with the forms: validation messages being missed,
> incomprehensible questions, help not immediately available etc. Yet
> all the users who struggled, even those who didn't manage to get a
> quote in the end, also said that they were quite happy with the whole
> process. Hope that answers your question :)
>
> Cheers,
> Alex
>
>

10 Jun 2008 - 12:15pm
SemanticWill
2007

I should have been more explicit - if you go to
http://www.rosenfeldmedia.com/, you can buy the book and have an electronic
version in 5 minutes - go to Chapter 2 "Form Organization," - which focuses
on just those issues you brought up - also read Caroline Jarrett's side
discussion on Keep, Cut, Postpone, and Explain. The chapter will take you 20
minutes to read - and will save a lot of time, headache, and especially
money in trial and error.

--
~ will

"Where you innovate, how you innovate,
and what you innovate are design problems"

---------------------------------------------------------------------------------------------
Will Evans | User Experience Architect
tel +1.617.281.1281 | will at semanticfoundry.com
twitter: https://twitter.com/semanticwill
---------------------------------------------------------------------------------------------

On Tue, Jun 10, 2008 at 1:07 PM, Will Evans <will at semanticfoundry.com>
wrote:

> Go quickly sir, get LukeW's book on Form Design, and enlighten the heathen
> masses! Good, actionable discussions on form design, design principles, and
> design patterns -- all focused on form design. Worth the cash!
>
> - W

10 Jun 2008 - 11:24am
Jeremy White
2008

It can certainly help you figure out some blatant issues, but that's
really only helpful if the product is clearly poorly designed. Why
not just fix those blatant issues yourself without having the user
confirm the obvious?

When creating the usability surveys, it's important to be as
non-leading as possible (don't tell them any of the steps needed to
perform that task). You've got to balance that vagueness with short
enough tasks so that they don't forget what they did and don't have
trouble trying to explain their actions to you.

This isn't hard to do for one person you know well, but for creating
a balanced survey for more than one person it's just impossible since
the balance will be different for everyone.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=30086

10 Jun 2008 - 12:56pm
tamlyn
2008

Thanks everyone! Lots of good advice.

Will: I've been meaning to buy LukeW's book since it came out but your
reminder pushed me to action :) I'll give the download a read this
evening.

Diane: we do have some fairly sophisticated web analyitics software
but I'm not sure anyone knows how to use it!

Cheers, Tamlyn.

10 Jun 2008 - 6:18pm
Steve Baty
2009

Tamlyn,

I'd reinforce the comment from Todd that your survey will only reliably
measure satisfaction; for the rest I'd suggest putting in place some
measurements around time-to-completion and abandonment - on both the old &
new versions - and run some analysis over the data in a week or so
(depending on volumes you could do it after a day). I'm assuming the reason
for the new design is to get more people to complete the process which is
why I've focused on those two metrics, but if your goal is something
different you might need different measurements.

Regards
Steve
----------------------------------------------
Steve 'Doc' Baty B.Sc (Maths), M.EC, MBA
Principal Consultant
Meld Consulting
M: +61 417 061 292
E: stevebaty at meld.com.au

UX Statistics: http://uxstats.blogspot.com

Member, UPA - www.upassoc.org
Member, IA Institute - www.iainstitute.org
Member, IxDA - www.ixda.org
Contributor - UXMatters - www.uxmatters.com

11 Jun 2008 - 7:12am
SemanticWill
2007

Definitely Tamlyn -

It's one of the few books for IxD folks that you can download the electronic
version and within an hour you have actionable principals, patterns, best
practices to put into your designs immediately...

For other people - I am trying to finish a book review today and will post
back to the group.

- Will

On Tue, Jun 10, 2008 at 1:56 PM, Tamlyn Rhodes <tamlyn at tamlyn.org> wrote:

> Thanks everyone! Lots of good advice.
>
> Will: I've been meaning to buy LukeW's book since it came out but your
> reminder pushed me to action :) I'll give the download a read this
> evening.
>
> Diane: we do have some fairly sophisticated web analyitics software
> but I'm not sure anyone knows how to use it!
>
> Cheers, Tamlyn.
>

17 Jun 2008 - 7:39am
SemanticWill
2007

I think this post by Seth Godin (who is not a UX person - he's a
marketing person, even if he sometimes thinks he is a UX person -
snark!),

http://tinyurl.com/c2lyt

"All customers are smarter than average...

In study after study, respondents rate themselves as less racist than
average, smarter than average, more generous than average.

And though they are never asked, I'm pretty certain that your
customers also believe that they are righter than average as well.

At the airport yesterday, a woman at security said to the TSA
official, "I'm a regular traveler, a frequent flyer and I know the
rules. I want the fast line." A moment later, it was determined that
the woman had two huge bottles of shampoo in her very large carry on.
"No one told me that there was a restriction on liquids! Where does
it say that?" she snarled, as she stood in front of the sign that
said that...

Any time you ask customers to self-segregate, they will put
themselves in the best line.

And just about any time you ask a customer to acknowledge that they
were wrong, you will fail."

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=30086

Syndicate content Get the feed