Measuring Usability

13 Jun 2005 - 6:12am
9 years ago
9 replies
550 reads
Sachendra
2005

Hi,

Is there a method to measure usability?

Regards,
-Sachin

Comments

13 Jun 2005 - 6:29am
Lada Gorlenko
2004

SY> Is there a method to measure usability?

Yes. Quite a few dozens of them, depending on what, when, where and
why one wants to measure. Also on who and how will be measuring. Also
on how long and how much one can spend.

Measuring usability is a biiig area indeed.

Lada

13 Jun 2005 - 6:32am
Sachendra
2005

I am working on mobile UI, we've created some new features, followed
the well known process, created personas, created prototypes, did
usability testing. I just want to make sure what goes out of our lab
is at par with best in the world. How can I make sure of that.

On 6/13/05, Lada Gorlenko <lada at acm.org> wrote:
> [Please voluntarily trim replies to include only relevant quoted material.]
>
> SY> Is there a method to measure usability?
>
> Yes. Quite a few dozens of them, depending on what, when, where and
> why one wants to measure. Also on who and how will be measuring. Also
> on how long and how much one can spend.
>
> Measuring usability is a biiig area indeed.
>
> Lada
>
> _______________________________________________
> Welcome to the Interaction Design Group!
> To post to this list ....... discuss at ixdg.org
> (Un)Subscription Options ... http://discuss.ixdg.org/
> Announcements List ......... http://subscribe-announce.ixdg.org/
> Questions .................. lists at ixdg.org
> Home ....................... http://ixdg.org/
>

--
Sachendra Yadav
9886376231

13 Jun 2005 - 6:45am
Tadej Maligoj
2004

By following well known process, you must have determined criteria.
You measure usability by them.

Tadej

On 6/13/05, Sachendra Yadav <sachendra at gmail.com> wrote:
> [Please voluntarily trim replies to include only relevant quoted material.]
>
> I am working on mobile UI, we've created some new features, followed
> the well known process, created personas, created prototypes, did
> usability testing. I just want to make sure what goes out of our lab
> is at par with best in the world. How can I make sure of that.
>
> On 6/13/05, Lada Gorlenko <lada at acm.org> wrote:
> > [Please voluntarily trim replies to include only relevant quoted material.]
> >
> > SY> Is there a method to measure usability?
> >
> > Yes. Quite a few dozens of them, depending on what, when, where and
> > why one wants to measure. Also on who and how will be measuring. Also
> > on how long and how much one can spend.
> >
> > Measuring usability is a biiig area indeed.
> >
> > Lada
> >
> > _______________________________________________
> > Welcome to the Interaction Design Group!
> > To post to this list ....... discuss at ixdg.org
> > (Un)Subscription Options ... http://discuss.ixdg.org/
> > Announcements List ......... http://subscribe-announce.ixdg.org/
> > Questions .................. lists at ixdg.org
> > Home ....................... http://ixdg.org/
> >
>
>
> --
> Sachendra Yadav
> 9886376231
> _______________________________________________
> Welcome to the Interaction Design Group!
> To post to this list ....... discuss at ixdg.org
> (Un)Subscription Options ... http://discuss.ixdg.org/
> Announcements List ......... http://subscribe-announce.ixdg.org/
> Questions .................. lists at ixdg.org
> Home ....................... http://ixdg.org/
>

--
_______________________________
Tadej Maligoj, Information Architect
e1: tadej.maligoj at gmail.com
e2: studio at maligoj.com
m: 031 306 986
w: www.maligoj.com

13 Jun 2005 - 7:45am
Lada Gorlenko
2004

SY> I am working on mobile UI, we've created some new features, followed
SY> the well known process, created personas, created prototypes, did
SY> usability testing. I just want to make sure what goes out of our lab
SY> is at par with best in the world. How can I make sure of that.

If you did usability testing, that should have been the place to
measure usability parameters. Which specific parameters you measure
depends on many aspects. The typical basic four measures are:
- efficiency (how fast the key user tasks are performed)
- effectiveness (how well the interface satisfies the key tasks)
- ease of learning (how steep and erroneous the learning curve is)
- satisfaction (how excited the users are about the product)

There are plenty of other very useful and insightful metrics, but
these are your plain vanilla benchmarks. Each measure has its own
weight in the overall picture, depending on the goals of the
interface. For example, for walk-up-and-use interfaces ease of
learning is particularly important. For interfaces which are used
often and intensively, efficiency can be critical. For
leisure/pleasure interfaces satisfaction may be of prime concern,
and so on. It's up to you (or rather your business) to decide which
goals you try to achieve with the new set of features.

Ideally, goals should have been set prior to the development and
initial measurements on the same group of tasks done on the old
version of the interface. Comparing the two sets of measurements
(before and after) gives you the improvement delta.

If you want to compare with "best in the world", run a competitive
evaluation with one or two products you consider you prime
competitors. Pick the same set of tasks, use similar user groups, test
on your product and competitors' interfaces.

It's a bird's eye overview, of course. Spend half and hour browsing
Usability Views site (http://www.usabilityviews.com/), it contains
lists of articles from main usability publications on the web, such as
Boxes and Arrows, Nielsen's Alertbox, and others. It should give you
some original ideas on what you can do next.

Lada

13 Jun 2005 - 7:30am
Scott Weiss
2005

Usability measurement is a proven discipline, as long as the analyst
making the measurements is consistent between usability sessions. In my
firm's "Media Download Usability" we conducted 20 hour-long sessions on
each of 10 mobile telephone handsets, asking respondents to shop for,
download, and install ring tones, games, and wallpapers. We measured the
following:
- Time to complete
- Success rates (did they succeed within a pre-determined amount of time?)
- Number of attempts (how many times did the participant return to the
idle screen while attempting a task?)
- Perceived usability, on a five-point scale (really difficult,
difficult, neither/nor, relatively easy, really easy)
- Perceived time (in minutes, or on a five-point scale)

These values can be crossed to track performance in two dimensions, as well.

Logging software makes these measurements easier.

We built a custom database to store the logging software results, and
use SPSS to analyze the data. We built our own charting software to
present the data our own way. Excel's charting features are ugly and
awkward, but will do for a small project.

You can read more about our study at http://usableproducts.com. I'll be
presenting this study's operations and design along with a subset of the
results at HCII.

Best of luck,
-scott

--
Scott Weiss
Principal, Usable Products Company: usableproducts.com (212.929.8599)
Author, "Handheld Usability": handheldusability.com

13 Jun 2005 - 10:58am
Mitchell Gass
2004

At 04:32 AM 6/13/2005, Sachendra Yadav asked:
>I am working on mobile UI...I just want to make sure what goes out of our
>lab is at par with best in the world. How can I make sure of that.

Sachendra,

There is a useful overview of comparative usability testing at

http://www.useit.com/alertbox/20010121.html

and details in a number of books, one of the best of which is A Practical
Guide to Usability Testing by Dumas and Redish

http://www.amazon.com/exec/obidos/tg/detail/-/1841500208/

There is also a standard for reporting the results which was just approved
by ISO (ISO 25062). There's information about the standard at

http://zing.ncsl.nist.gov/iusr/

The standard is available for download at

http://www.techstreet.com/cgi-bin/detail?product_id=918375

Best,

Mitchell Gass
uLab | PDA: Learning from Users | Designing with Users
Berkeley, CA 94707 USA
+1 510 525-6864 voice
+1 510 525-4246 fax
http://www.participatorydesign.com/

13 Jun 2005 - 11:10pm
Sachendra
2005

Thanks guys, this was really helpful.
My dilemma was that these features are not in the market yet (I'm sure
competitors must be working on them as well) so I can't really do a
competitve analysis, and wanted to get it as right as I could in the
first cut itself.

Scott's techniques and Usabilityviews.com were very helpful. I'm sure
the ISO standard will help achieve the last mile.

Thanks,
-Sachin

On 6/13/05, Mitchell Gass <mitchell at participatorydesign.com> wrote:
> At 04:32 AM 6/13/2005, Sachendra Yadav asked:
> >I am working on mobile UI...I just want to make sure what goes out of our
> >lab is at par with best in the world. How can I make sure of that.
>
> Sachendra,
>
> There is a useful overview of comparative usability testing at
>
> http://www.useit.com/alertbox/20010121.html
>
> and details in a number of books, one of the best of which is A Practical
> Guide to Usability Testing by Dumas and Redish
>
> http://www.amazon.com/exec/obidos/tg/detail/-/1841500208/
>
> There is also a standard for reporting the results which was just approved
> by ISO (ISO 25062). There's information about the standard at
>
> http://zing.ncsl.nist.gov/iusr/
>
> The standard is available for download at
>
> http://www.techstreet.com/cgi-bin/detail?product_id=918375
>
> Best,
>
> Mitchell Gass
> uLab | PDA: Learning from Users | Designing with Users
> Berkeley, CA 94707 USA
> +1 510 525-6864 voice
> +1 510 525-4246 fax
> http://www.participatorydesign.com/
>
>
>

--
Sachendra Yadav
9886376231

14 Jun 2005 - 10:31am
Ted Boren
2005

A couple of keywords to guide your search for information on this topic:
"benchmark" usability testing, "summative" (as opposed to formative) usability
testing. I'll also second the plug for the Dumas & Redish book cited below.

Best of luck!

Ted Boren

>>> "Mitchell Gass" <mitchell at participatorydesign.com> 6/13/2005 9:58:28 AM
>>>
At 04:32 AM 6/13/2005, Sachendra Yadav asked:
>I am working on mobile UI...I just want to make sure what goes out of our
>lab is at par with best in the world. How can I make sure of that.

Sachendra,

There is a useful overview of comparative usability testing at

http://www.useit.com/alertbox/20010121.html

and details in a number of books, one of the best of which is A Practical
Guide to Usability Testing by Dumas and Redish

http://www.amazon.com/exec/obidos/tg/detail/-/1841500208/

There is also a standard for reporting the results which was just approved
by ISO (ISO 25062). There's information about the standard at

http://zing.ncsl.nist.gov/iusr/

The standard is available for download at

http://www.techstreet.com/cgi-bin/detail?product_id=918375

Best,

Mitchell Gass
uLab | PDA: Learning from Users | Designing with Users
Berkeley, CA 94707 USA
+1 510 525-6864 voice
+1 510 525-4246 fax
http://www.participatorydesign.com/

------------------------------------------------------------------------------
This message may contain confidential information, and is
intended only for the use of the individual(s) to whom it
is addressed.
------------------------------------------------------------------------------

14 Jun 2005 - 1:11pm
Richard Zobarich
2004

{ At 04:32 AM 6/13/2005, Sachendra Yadav asked:

I am working on mobile UI...I just want to make sure what goes out of our lab is at par with best in the world. How can I make sure of that.

Sachendra

}
Hi,

I suggest doing a Comparative Usability Test. Use the results of the usability test to: (a) Shape the level of buy-in, i.e. all those that have a stake in the success of the mobile UI, in favour of (b) Capital D. Inform, inspire, and shape both User Experience Research and User Experience Design, to be THE 007 of Innovation.

Some great books on Usability:

Dumas, J. and Redish, J. A Practical Guide to Usability Testing, Revised Edition, Intellect, 1999. ISBN: 1-84150-020-8.
:: http://www.amazon.com/exec/obidos/ASIN/1841500208/ref=pd_sxp_elt_l1/103-0860078-0064663 ::

Rubin, J. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. Wiley: New York, NY, 1994.
:: http://www.amazon.com/exec/obidos/ASIN/0471594032/stcusabilitsig/103-0860078-0064663 ::

On User Experience:

Kuniavsky, M. Observing the user experience: A practitioner's guide to user research. Morgan Kaufmann, San Francisco, CA, 2003
:: http://www.amazon.com/exec/obidos/ASIN/1558609237/ref=pd_sxp_elt_l1/103-0860078-0064663 ::

Finally, I recommend checking out the UPA bookshelf:
:: http://www.stcsig.org/usability/resources/bookshelf/ ::

Sincerely,
Richard Zobarich
>

Syndicate content Get the feed