iPhone Usability Test

21 Sep 2007 - 9:06pm
7 years ago
7 replies
2035 reads
Jack L. Moffett
2005

Has anyone read about the iPhone vs. HTC Touch vs. Nokia N95
usability study performed by Perceptive Sciences?

http://tinyurl.com/yojmyr

While I believe the final result is likely correct (the iPhone wins
handily), I question the integrity of the study. The way they define
"usability" and the categories in which they rank the phones give me
the impression that the consulting firm is not one of the "country's
top...user-experience firms" as they claim.

Looking at their website, they seem to be a whole lot of marketing
research with a few usability/UCD buzzwords thrown in for good measure.

From their webpage explaining their approach to design:
> Does your user interface help sell your product or service? Is it
> helping or hindering your brand? Can your customer effectively and
> efficiently use your product? Will the usability of your product
> positively or negatively affect future buying decisions?
>
> Our design services focus on assembling data and recommending
> designs that answer these questions.
>
http://www.perceptivesciences.com/services/design.php

I don't know anything else about them, nor do I know if any of their
employees are on the list. I don't want to offend anyone, but from
what I read in the article, I'm not impressed with their study.

Jack

Jack L. Moffett
Interaction Designer
inmedius
412.459.0310 x219
http://www.inmedius.com

You could design a process to catch
everything, but then you're overprocessing.
You kill creativity. You kill productivity.
By definition, a culture like ours that
drives innovation is managed chaos.

-Alex Lee
President, OXO International

Comments

21 Sep 2007 - 9:34pm
.pauric
2006

Dont know if this is relevant.. in my industry we have a
"Independent, Authoritative, Unbiased" firm The Tolly Group:
http://www.tolly.com/

Independent in the sense that vendors pay them to test their
products, we get to name who's products we get compared against, the
parameters of the tests as well as the features we dont want tested.
Once thats spelled out, the results a completely out of our control.

I'm not saying thats whats going on here, just that I guess there's
an existing business model and maybe a need for it in the Usability
arena..

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the improved ixda.org
http://beta.ixda.org/discuss?post=20740

21 Sep 2007 - 11:45pm
Dave Malouf
2005

hmmm? Jack, I think you need to be more specific. Personally that
description of the company sounds like one that is right on target
for real UX and not just usability.

Let me put that another way. It spoke to me in a positive way.

I think I'd need more from you about why you think the test is
problematic. It just sounds like you don't like something in the
company's messaging. So can you explain what you didn't like about
the test? Personally, I don't think there is enough to go on to
judge (and there seldom is).

I had problems with other publicized iPhone tests done as well by
"good companies".

I actually really appreciate that in the article itself it
acknowledges the difference between short term out-of-the box
usability and long term user experience. Tells me that they get an
important part of the puzzle.

-- dave

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the improved ixda.org
http://beta.ixda.org/discuss?post=20740

22 Sep 2007 - 3:52pm
Jack L. Moffett
2005

> I think I'd need more from you about why you think the test is
> problematic. It just sounds like you don't like something in the
> company's messaging. So can you explain what you didn't like about
> the test? Personally, I don't think there is enough to go on to
> judge (and there seldom is).

That's a good point, and I suppose that I should be satisfied that
they went into as much depth about the tests as they did. Perhaps
it's the fact that there is more detail than would usually be given
in a news article of this kind that lead me to be more critical.

> I actually really appreciate that in the article itself it
> acknowledges the difference between short term out-of-the box
> usability and long term user experience. Tells me that they get an
> important part of the puzzle.

True, but I don't quite agree with the way they expressed it.

""People can eventually learn to use any device," Ballew said. "But
that's not true usability. We wanted to see how long it took to
figure out how to use the phones. That's the difference between
learnability and usability."

This makes it seem as though usability equates with intuitiveness—
that usability is only good for first impressions. I would consider
learnability to be an important part of usability. I don't mind that
they focused on how easy it was to use for the first time, just their
explanation.

Then they had a rating category titled "Usability/Information
Architecture".

"Usability/IA is similar to global navigation, but it more
specifically refers to how easy and fun the interface is."

I think measuring how easy the interface is to use, and how enjoyable
it is to use, are both important measures. But I would label that
Interaction Design, or would expect to see it called Interface
Design. If the whole test is a usability test, why is one section
labeled "usability"? To be fair, Ballew does talk about labels and
the file structure, which does fit under IA.

Towards the beginning of the article, it specifically points out,
"It's important to remember that these are usability tests, not tests
of functionality." Then, the final category on which the phones were
rated was functionality.

Having read it a second time now, I suppose most of my issues are
more with the way in which the test has been communicated in the
article. Perhaps I should be less critical of Perceptive Sciences,
and chalk it up to David Haskin's interpretation.

As for the descriptions from Perceptive Science's website, they
seemed to have a strong marketing focus. Not that there is
necessarily anything wrong with that, but it raised a flag for me.

Jack

Jack L. Moffett
Interaction Designer
inmedius
412.459.0310 x219
http://www.inmedius.com

You could design a process to catch
everything, but then you're overprocessing.
You kill creativity. You kill productivity.
By definition, a culture like ours that
drives innovation is managed chaos.

-Alex Lee
President, OXO International

23 Sep 2007 - 7:23am
Todd Warfel
2003

I'd have to agree with Jack on this. To me there are three levels of
usability:
1. Intuitive - no severity, user understands it, can use it natively
from the beginning.
2. Learnable - low severity, user may experience some initial
confusion the first use, but subsequent use becomes easier and
acceptable.
3. Unusable - high severity, user experiences frustration and
confusion every time they use the product with little or no ability
to overcome the problem. This is a problem every time the customer
uses the product.

So, I would agree that intuitive does not directly equal usable. It's
only one grade of usable. At least, that's the real world.

On Sep 22, 2007, at 5:52 PM, Jack Moffett wrote:

> ""People can eventually learn to use any device," Ballew said. "But
> that's not true usability. We wanted to see how long it took to
> figure out how to use the phones. That's the difference between
> learnability and usability."
>
> This makes it seem as though usability equates with intuitiveness—
> that usability is only good for first impressions. I would consider
> learnability to be an important part of usability. I don't mind
> that they focused on how easy it was to use for the first time,
> just their explanation.

Cheers!

Todd Zaki Warfel
President, Design Researcher
Messagefirst | Designing Information. Beautifully.
----------------------------------
Contact Info
Voice: (215) 825-7423
Email: todd at messagefirst.com
AIM: twarfel at mac.com
Blog: http://toddwarfel.com
----------------------------------
In theory, theory and practice are the same.
In practice, they are not.

23 Sep 2007 - 10:00am
Barbara Ballard
2005

On 9/23/07, Todd Zaki Warfel <lists at toddwarfel.com> wrote:
> I'd have to agree with Jack on this. To me there are three levels of
> usability:
> 2. Learnable - low severity, user may experience some initial
> confusion the first use, but subsequent use becomes easier and
> acceptable.

"Learnable" is an odd duck. For many kiosks, "learnable" could be the
kiss of death. For an iPod, "learnable" has done fabulously well.

I just joined Facebook for the first time, with absolutely no prior
exposure. There's lots that is not "intuitive", but looks (and is)
"learnable" ... and I enjoy the learning process.

For a highly personal device, or for a game, learnable could be a very
good thing. Learning the product is part of the experience, and
expertise in the product is a bit of pride.

What's critical, to my view, is not that everything is intuitive, but
that a critical mass of the product is intuitive. The rest should be
learnable in an enjoyable, discoverable, intuitive fashion.

--
Barbara Ballard
barbara at littlespringsdesign.com 1-785-838-3003

23 Sep 2007 - 10:39am
Dave Malouf
2005

I wrote up a bit of a anecdotal experience I had w/ my iPhone that I
think expresses these points.

My father was visiting form Israel ... I don't think they have hope
of an locked (authentic) iPhone for quite some time so the press and
the pop culture have no clue of what is going on. Basically, he never
heard of it before.

He saw how much I used it for email and texting and surfing and was
really impressed with its functionality but because he didn't really
see "how" I was doing anything, he just assumed that it would be
"too hard" for him to use.

I handed him the phone in the off position and told him to call my
home number. While it took a bit mainly b/c he kept saying "wow!"
every so often, I'd say that it was a complete success and then I
said, "Send me an email", and "go to the NYTimes".

Now the 2nd thing worked well, but going to the NYTimes didn't work.
Why? b/c Apple put "Safari" as hte label instead of "Web" like
most phones do and unless you are an avid Mac user "Safari" is a
completely foreign application to you. The icon wasn't enough.

so I guided him there, but still had to teach him how to zoom pan.

But after I was done with the lesson he was so incredibly impressed
and so able to just continue going that he wouldn't give me my phone
back for quite awhile.

Tasks 1 & 2: intuitive
Task 3: learnable

I totally agree with Barbara that "learnable" is sometimes a
quality that as a designer we can't afford to use. Kiosks is a great
example. The Metrocard (subway fare) system in NYC is a device that
has been lauded and criticized, but whether completely successful or
not, it was designed to be intuitive and not learnable. Having
"Start" be the first and only button guides the user away from the
physical buttons which probably pulls their attention and focuses
them on the screen. Then having very limited choices in each screen
is another way to reduce learnability in favor of intuitive, but that
also means making it more difficult for "advanced" users. I mean now
for the NYC Subway most native residents could probably use a system
that is much faster. I.e. when refilling a card, wouldn't it be
great to just be able to put your card in the slot and it assumes a
workflow based on reading that card and giving you appropriate
choices.

-- dave

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the improved ixda.org
http://beta.ixda.org/discuss?post=20740

23 Sep 2007 - 11:17am
Mark Schraad
2006

"I totally agree with Barbara that "learnable" is sometimes a
> quality that as a designer we can't afford to use."

The thing is... and I am going to shift over and put a marketing/
experience hat on for a moment... for many early adopters learn-
ability is a very powerful component of satisfaction and post
purchase behavior. There is truly a seduction through the discover
process for people that make purchases of 'new toys'. It could be an
old house (the history), a car (so many feature I forgot all about
the ski/trunk door) or a cool phone, but that discovery process is
often what keeps the user engaged beyond utility. It can drive
customer satisfaction, and in fact drives much of apples brand
loyalty well beyond the initial purchase and early use. Further, it
is often what solicits the next conversation (spreading the word) and
fanaticism. In the early adopter phase, technology can be too simple
- too intuitive.

I think what is interesting, and hardly talked about, is managing and
planning the layers of feature discovery. An initial interface that
is very intuitive... allowing the user to find utility immediately.
Later comes the delight of discovering a capability that was
unexpected. Hackers and third party developers can be a very
important component, adding yet more layers of functionality (I can
record a lecture with my iPod? - cool).

Mark

On Sep 23, 2007, at 5:39 AM, dave malouf wrote:

> I wrote up a bit of a anecdotal experience I had w/ my iPhone that I
> think expresses these points.
>
> My father was visiting form Israel ... I don't think they have hope
> of an locked (authentic) iPhone for quite some time so the press and
> the pop culture have no clue of what is going on. Basically, he never
> heard of it before.
>
> He saw how much I used it for email and texting and surfing and was
> really impressed with its functionality but because he didn't really
> see "how" I was doing anything, he just assumed that it would be
> "too hard" for him to use.
>
> I handed him the phone in the off position and told him to call my
> home number. While it took a bit mainly b/c he kept saying "wow!"
> every so often, I'd say that it was a complete success and then I
> said, "Send me an email", and "go to the NYTimes".
>
> Now the 2nd thing worked well, but going to the NYTimes didn't work.
> Why? b/c Apple put "Safari" as hte label instead of "Web" like
> most phones do and unless you are an avid Mac user "Safari" is a
> completely foreign application to you. The icon wasn't enough.
>
> so I guided him there, but still had to teach him how to zoom pan.
>
> But after I was done with the lesson he was so incredibly impressed
> and so able to just continue going that he wouldn't give me my phone
> back for quite awhile.
>
> Tasks 1 & 2: intuitive
> Task 3: learnable
>
> I totally agree with Barbara that "learnable" is sometimes a
> quality that as a designer we can't afford to use. Kiosks is a great
> example. The Metrocard (subway fare) system in NYC is a device that
> has been lauded and criticized, but whether completely successful or
> not, it was designed to be intuitive and not learnable. Having
> "Start" be the first and only button guides the user away from the
> physical buttons which probably pulls their attention and focuses
> them on the screen. Then having very limited choices in each screen
> is another way to reduce learnability in favor of intuitive, but that
> also means making it more difficult for "advanced" users. I mean now
> for the NYC Subway most native residents could probably use a system
> that is much faster. I.e. when refilling a card, wouldn't it be
> great to just be able to put your card in the slot and it assumes a
> workflow based on reading that card and giving you appropriate
> choices.
>
> -- dave

Syndicate content Get the feed