Yes. Quite a few dozens of them, depending on what, when, where and
why one wants to measure. Also on who and how will be measuring. Also
on how long and how much one can spend.
I am working on mobile UI, we've created some new features, followed
the well known process, created personas, created prototypes, did
usability testing. I just want to make sure what goes out of our lab
is at par with best in the world. How can I make sure of that.
On 6/13/05, Lada Gorlenko <lada at acm.org> wrote:
> [Please voluntarily trim replies to include only relevant quoted material.] > > SY> Is there a method to measure usability? > > Yes. Quite a few dozens of them, depending on what, when, where and > why one wants to measure. Also on who and how will be measuring. Also > on how long and how much one can spend. > > Measuring usability is a biiig area indeed. > > Lada > > _______________________________________________ > Welcome to the Interaction Design Group! > To post to this list ....... discuss at ixdg.org > (Un)Subscription Options ... http://discuss.ixdg.org/ > Announcements List ......... http://subscribe-announce.ixdg.org/ > Questions .................. lists at ixdg.org > Home ....................... http://ixdg.org/ >
By following well known process, you must have determined criteria.
You measure usability by them.
Tadej
On 6/13/05, Sachendra Yadav <sachendra at gmail.com> wrote:
> [Please voluntarily trim replies to include only relevant quoted material.] > > I am working on mobile UI, we've created some new features, followed > the well known process, created personas, created prototypes, did > usability testing. I just want to make sure what goes out of our lab > is at par with best in the world. How can I make sure of that. > > On 6/13/05, Lada Gorlenko <lada at acm.org> wrote: > > [Please voluntarily trim replies to include only relevant quoted material.] > > > > SY> Is there a method to measure usability? > > > > Yes. Quite a few dozens of them, depending on what, when, where and > > why one wants to measure. Also on who and how will be measuring. Also > > on how long and how much one can spend. > > > > Measuring usability is a biiig area indeed. > > > > Lada > > > > _______________________________________________ > > Welcome to the Interaction Design Group! > > To post to this list ....... discuss at ixdg.org > > (Un)Subscription Options ... http://discuss.ixdg.org/ > > Announcements List ......... http://subscribe-announce.ixdg.org/ > > Questions .................. lists at ixdg.org > > Home ....................... http://ixdg.org/ > > > > > -- > Sachendra Yadav > 9886376231 > _______________________________________________ > Welcome to the Interaction Design Group! > To post to this list ....... discuss at ixdg.org > (Un)Subscription Options ... http://discuss.ixdg.org/ > Announcements List ......... http://subscribe-announce.ixdg.org/ > Questions .................. lists at ixdg.org > Home ....................... http://ixdg.org/ >
SY> I am working on mobile UI, we've created some new features, followed
SY> the well known process, created personas, created prototypes, did
SY> usability testing. I just want to make sure what goes out of our lab
SY> is at par with best in the world. How can I make sure of that.
If you did usability testing, that should have been the place to
measure usability parameters. Which specific parameters you measure
depends on many aspects. The typical basic four measures are:
- efficiency (how fast the key user tasks are performed)
- effectiveness (how well the interface satisfies the key tasks)
- ease of learning (how steep and erroneous the learning curve is)
- satisfaction (how excited the users are about the product)
There are plenty of other very useful and insightful metrics, but
these are your plain vanilla benchmarks. Each measure has its own
weight in the overall picture, depending on the goals of the
interface. For example, for walk-up-and-use interfaces ease of
learning is particularly important. For interfaces which are used
often and intensively, efficiency can be critical. For
leisure/pleasure interfaces satisfaction may be of prime concern,
and so on. It's up to you (or rather your business) to decide which
goals you try to achieve with the new set of features.
Ideally, goals should have been set prior to the development and
initial measurements on the same group of tasks done on the old
version of the interface. Comparing the two sets of measurements
(before and after) gives you the improvement delta.
If you want to compare with "best in the world", run a competitive
evaluation with one or two products you consider you prime
competitors. Pick the same set of tasks, use similar user groups, test
on your product and competitors' interfaces.
It's a bird's eye overview, of course. Spend half and hour browsing
Usability Views site (http://www.usabilityviews.com/), it contains
lists of articles from main usability publications on the web, such as
Boxes and Arrows, Nielsen's Alertbox, and others. It should give you
some original ideas on what you can do next.
Usability measurement is a proven discipline, as long as the analyst
making the measurements is consistent between usability sessions. In my
firm's "Media Download Usability" we conducted 20 hour-long sessions on
each of 10 mobile telephone handsets, asking respondents to shop for,
download, and install ring tones, games, and wallpapers. We measured the
following:
- Time to complete
- Success rates (did they succeed within a pre-determined amount of time?)
- Number of attempts (how many times did the participant return to the
idle screen while attempting a task?)
- Perceived usability, on a five-point scale (really difficult,
difficult, neither/nor, relatively easy, really easy)
- Perceived time (in minutes, or on a five-point scale)
These values can be crossed to track performance in two dimensions, as well.
Logging software makes these measurements easier.
We built a custom database to store the logging software results, and
use SPSS to analyze the data. We built our own charting software to
present the data our own way. Excel's charting features are ugly and
awkward, but will do for a small project.
You can read more about our study at http://usableproducts.com. I'll be
presenting this study's operations and design along with a subset of the
results at HCII.
At 04:32 AM 6/13/2005, Sachendra Yadav asked:
>I am working on mobile UI...I just want to make sure what goes out of our >lab is at par with best in the world. How can I make sure of that.
Sachendra,
There is a useful overview of comparative usability testing at
Thanks guys, this was really helpful.
My dilemma was that these features are not in the market yet (I'm sure
competitors must be working on them as well) so I can't really do a
competitve analysis, and wanted to get it as right as I could in the
first cut itself.
Scott's techniques and Usabilityviews.com were very helpful. I'm sure
the ISO standard will help achieve the last mile.
Thanks,
-Sachin
On 6/13/05, Mitchell Gass <mitchell at participatorydesign.com> wrote:
> At 04:32 AM 6/13/2005, Sachendra Yadav asked: > >I am working on mobile UI...I just want to make sure what goes out of our > >lab is at par with best in the world. How can I make sure of that. > > Sachendra, > > There is a useful overview of comparative usability testing at > >http://www.useit.com/alertbox/20010121.html > > and details in a number of books, one of the best of which is A Practical > Guide to Usability Testing by Dumas and Redish > >http://www.amazon.com/exec/obidos/tg/detail/-/1841500208/ > > There is also a standard for reporting the results which was just approved > by ISO (ISO 25062). There's information about the standard at > >http://zing.ncsl.nist.gov/iusr/ > > The standard is available for download at > >http://www.techstreet.com/cgi-bin/detail?product_id=918375 > > Best, > > Mitchell Gass > uLab | PDA: Learning from Users | Designing with Users > Berkeley, CA 94707 USA > +1 510 525-6864 voice > +1 510 525-4246 fax >http://www.participatorydesign.com/ > > >
A couple of keywords to guide your search for information on this topic:
"benchmark" usability testing, "summative" (as opposed to formative) usability
testing. I'll also second the plug for the Dumas & Redish book cited below.
Best of luck!
Ted Boren
>>> "Mitchell Gass" <mitchell at participatorydesign.com> 6/13/2005 9:58:28 AM >>> At 04:32 AM 6/13/2005, Sachendra Yadav asked:
>I am working on mobile UI...I just want to make sure what goes out of our >lab is at par with best in the world. How can I make sure of that.
Sachendra,
There is a useful overview of comparative usability testing at
Mitchell Gass
uLab | PDA: Learning from Users | Designing with Users
Berkeley, CA 94707 USA
+1 510 525-6864 voice
+1 510 525-4246 fax http://www.participatorydesign.com/
------------------------------------------------------------------------------
This message may contain confidential information, and is
intended only for the use of the individual(s) to whom it
is addressed.
------------------------------------------------------------------------------
I am working on mobile UI...I just want to make sure what goes out of our lab is at par with best in the world. How can I make sure of that.
Sachendra
}
Hi,
I suggest doing a Comparative Usability Test. Use the results of the usability test to: (a) Shape the level of buy-in, i.e. all those that have a stake in the success of the mobile UI, in favour of (b) Capital D. Inform, inspire, and shape both User Experience Research and User Experience Design, to be THE 007 of Innovation.
Get the feed
Comments
SY> Is there a method to measure usability?
Yes. Quite a few dozens of them, depending on what, when, where and
why one wants to measure. Also on who and how will be measuring. Also
on how long and how much one can spend.
Measuring usability is a biiig area indeed.
Lada
I am working on mobile UI, we've created some new features, followed
the well known process, created personas, created prototypes, did
usability testing. I just want to make sure what goes out of our lab
is at par with best in the world. How can I make sure of that.
On 6/13/05, Lada Gorlenko <lada at acm.org> wrote:
> [Please voluntarily trim replies to include only relevant quoted material.]
>
> SY> Is there a method to measure usability?
>
> Yes. Quite a few dozens of them, depending on what, when, where and
> why one wants to measure. Also on who and how will be measuring. Also
> on how long and how much one can spend.
>
> Measuring usability is a biiig area indeed.
>
> Lada
>
> _______________________________________________
> Welcome to the Interaction Design Group!
> To post to this list ....... discuss at ixdg.org
> (Un)Subscription Options ... http://discuss.ixdg.org/
> Announcements List ......... http://subscribe-announce.ixdg.org/
> Questions .................. lists at ixdg.org
> Home ....................... http://ixdg.org/
>
--
Sachendra Yadav
9886376231
By following well known process, you must have determined criteria.
You measure usability by them.
Tadej
On 6/13/05, Sachendra Yadav <sachendra at gmail.com> wrote:
> [Please voluntarily trim replies to include only relevant quoted material.]
>
> I am working on mobile UI, we've created some new features, followed
> the well known process, created personas, created prototypes, did
> usability testing. I just want to make sure what goes out of our lab
> is at par with best in the world. How can I make sure of that.
>
> On 6/13/05, Lada Gorlenko <lada at acm.org> wrote:
> > [Please voluntarily trim replies to include only relevant quoted material.]
> >
> > SY> Is there a method to measure usability?
> >
> > Yes. Quite a few dozens of them, depending on what, when, where and
> > why one wants to measure. Also on who and how will be measuring. Also
> > on how long and how much one can spend.
> >
> > Measuring usability is a biiig area indeed.
> >
> > Lada
> >
> > _______________________________________________
> > Welcome to the Interaction Design Group!
> > To post to this list ....... discuss at ixdg.org
> > (Un)Subscription Options ... http://discuss.ixdg.org/
> > Announcements List ......... http://subscribe-announce.ixdg.org/
> > Questions .................. lists at ixdg.org
> > Home ....................... http://ixdg.org/
> >
>
>
> --
> Sachendra Yadav
> 9886376231
> _______________________________________________
> Welcome to the Interaction Design Group!
> To post to this list ....... discuss at ixdg.org
> (Un)Subscription Options ... http://discuss.ixdg.org/
> Announcements List ......... http://subscribe-announce.ixdg.org/
> Questions .................. lists at ixdg.org
> Home ....................... http://ixdg.org/
>
--
_______________________________
Tadej Maligoj, Information Architect
e1: tadej.maligoj at gmail.com
e2: studio at maligoj.com
m: 031 306 986
w: www.maligoj.com
SY> I am working on mobile UI, we've created some new features, followed
SY> the well known process, created personas, created prototypes, did
SY> usability testing. I just want to make sure what goes out of our lab
SY> is at par with best in the world. How can I make sure of that.
If you did usability testing, that should have been the place to
measure usability parameters. Which specific parameters you measure
depends on many aspects. The typical basic four measures are:
- efficiency (how fast the key user tasks are performed)
- effectiveness (how well the interface satisfies the key tasks)
- ease of learning (how steep and erroneous the learning curve is)
- satisfaction (how excited the users are about the product)
There are plenty of other very useful and insightful metrics, but
these are your plain vanilla benchmarks. Each measure has its own
weight in the overall picture, depending on the goals of the
interface. For example, for walk-up-and-use interfaces ease of
learning is particularly important. For interfaces which are used
often and intensively, efficiency can be critical. For
leisure/pleasure interfaces satisfaction may be of prime concern,
and so on. It's up to you (or rather your business) to decide which
goals you try to achieve with the new set of features.
Ideally, goals should have been set prior to the development and
initial measurements on the same group of tasks done on the old
version of the interface. Comparing the two sets of measurements
(before and after) gives you the improvement delta.
If you want to compare with "best in the world", run a competitive
evaluation with one or two products you consider you prime
competitors. Pick the same set of tasks, use similar user groups, test
on your product and competitors' interfaces.
It's a bird's eye overview, of course. Spend half and hour browsing
Usability Views site (http://www.usabilityviews.com/), it contains
lists of articles from main usability publications on the web, such as
Boxes and Arrows, Nielsen's Alertbox, and others. It should give you
some original ideas on what you can do next.
Lada
Usability measurement is a proven discipline, as long as the analyst
making the measurements is consistent between usability sessions. In my
firm's "Media Download Usability" we conducted 20 hour-long sessions on
each of 10 mobile telephone handsets, asking respondents to shop for,
download, and install ring tones, games, and wallpapers. We measured the
following:
- Time to complete
- Success rates (did they succeed within a pre-determined amount of time?)
- Number of attempts (how many times did the participant return to the
idle screen while attempting a task?)
- Perceived usability, on a five-point scale (really difficult,
difficult, neither/nor, relatively easy, really easy)
- Perceived time (in minutes, or on a five-point scale)
These values can be crossed to track performance in two dimensions, as well.
Logging software makes these measurements easier.
We built a custom database to store the logging software results, and
use SPSS to analyze the data. We built our own charting software to
present the data our own way. Excel's charting features are ugly and
awkward, but will do for a small project.
You can read more about our study at http://usableproducts.com. I'll be
presenting this study's operations and design along with a subset of the
results at HCII.
Best of luck,
-scott
--
Scott Weiss
Principal, Usable Products Company: usableproducts.com (212.929.8599)
Author, "Handheld Usability": handheldusability.com
At 04:32 AM 6/13/2005, Sachendra Yadav asked:
>I am working on mobile UI...I just want to make sure what goes out of our
>lab is at par with best in the world. How can I make sure of that.
Sachendra,
There is a useful overview of comparative usability testing at
http://www.useit.com/alertbox/20010121.html
and details in a number of books, one of the best of which is A Practical
Guide to Usability Testing by Dumas and Redish
http://www.amazon.com/exec/obidos/tg/detail/-/1841500208/
There is also a standard for reporting the results which was just approved
by ISO (ISO 25062). There's information about the standard at
http://zing.ncsl.nist.gov/iusr/
The standard is available for download at
http://www.techstreet.com/cgi-bin/detail?product_id=918375
Best,
Mitchell Gass
uLab | PDA: Learning from Users | Designing with Users
Berkeley, CA 94707 USA
+1 510 525-6864 voice
+1 510 525-4246 fax
http://www.participatorydesign.com/
Thanks guys, this was really helpful.
My dilemma was that these features are not in the market yet (I'm sure
competitors must be working on them as well) so I can't really do a
competitve analysis, and wanted to get it as right as I could in the
first cut itself.
Scott's techniques and Usabilityviews.com were very helpful. I'm sure
the ISO standard will help achieve the last mile.
Thanks,
-Sachin
On 6/13/05, Mitchell Gass <mitchell at participatorydesign.com> wrote:
> At 04:32 AM 6/13/2005, Sachendra Yadav asked:
> >I am working on mobile UI...I just want to make sure what goes out of our
> >lab is at par with best in the world. How can I make sure of that.
>
> Sachendra,
>
> There is a useful overview of comparative usability testing at
>
> http://www.useit.com/alertbox/20010121.html
>
> and details in a number of books, one of the best of which is A Practical
> Guide to Usability Testing by Dumas and Redish
>
> http://www.amazon.com/exec/obidos/tg/detail/-/1841500208/
>
> There is also a standard for reporting the results which was just approved
> by ISO (ISO 25062). There's information about the standard at
>
> http://zing.ncsl.nist.gov/iusr/
>
> The standard is available for download at
>
> http://www.techstreet.com/cgi-bin/detail?product_id=918375
>
> Best,
>
> Mitchell Gass
> uLab | PDA: Learning from Users | Designing with Users
> Berkeley, CA 94707 USA
> +1 510 525-6864 voice
> +1 510 525-4246 fax
> http://www.participatorydesign.com/
>
>
>
--
Sachendra Yadav
9886376231
A couple of keywords to guide your search for information on this topic:
"benchmark" usability testing, "summative" (as opposed to formative) usability
testing. I'll also second the plug for the Dumas & Redish book cited below.
Best of luck!
Ted Boren
>>> "Mitchell Gass" <mitchell at participatorydesign.com> 6/13/2005 9:58:28 AM
>>>
At 04:32 AM 6/13/2005, Sachendra Yadav asked:
>I am working on mobile UI...I just want to make sure what goes out of our
>lab is at par with best in the world. How can I make sure of that.
Sachendra,
There is a useful overview of comparative usability testing at
http://www.useit.com/alertbox/20010121.html
and details in a number of books, one of the best of which is A Practical
Guide to Usability Testing by Dumas and Redish
http://www.amazon.com/exec/obidos/tg/detail/-/1841500208/
There is also a standard for reporting the results which was just approved
by ISO (ISO 25062). There's information about the standard at
http://zing.ncsl.nist.gov/iusr/
The standard is available for download at
http://www.techstreet.com/cgi-bin/detail?product_id=918375
Best,
Mitchell Gass
uLab | PDA: Learning from Users | Designing with Users
Berkeley, CA 94707 USA
+1 510 525-6864 voice
+1 510 525-4246 fax
http://www.participatorydesign.com/
------------------------------------------------------------------------------
This message may contain confidential information, and is
intended only for the use of the individual(s) to whom it
is addressed.
------------------------------------------------------------------------------
{ At 04:32 AM 6/13/2005, Sachendra Yadav asked:
I am working on mobile UI...I just want to make sure what goes out of our lab is at par with best in the world. How can I make sure of that.
Sachendra
}
Hi,
I suggest doing a Comparative Usability Test. Use the results of the usability test to: (a) Shape the level of buy-in, i.e. all those that have a stake in the success of the mobile UI, in favour of (b) Capital D. Inform, inspire, and shape both User Experience Research and User Experience Design, to be THE 007 of Innovation.
Some great books on Usability:
Dumas, J. and Redish, J. A Practical Guide to Usability Testing, Revised Edition, Intellect, 1999. ISBN: 1-84150-020-8.
:: http://www.amazon.com/exec/obidos/ASIN/1841500208/ref=pd_sxp_elt_l1/103-0860078-0064663 ::
Rubin, J. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. Wiley: New York, NY, 1994.
:: http://www.amazon.com/exec/obidos/ASIN/0471594032/stcusabilitsig/103-0860078-0064663 ::
On User Experience:
Kuniavsky, M. Observing the user experience: A practitioner's guide to user research. Morgan Kaufmann, San Francisco, CA, 2003
:: http://www.amazon.com/exec/obidos/ASIN/1558609237/ref=pd_sxp_elt_l1/103-0860078-0064663 ::
Finally, I recommend checking out the UPA bookshelf:
:: http://www.stcsig.org/usability/resources/bookshelf/ ::
Sincerely,
Richard Zobarich
>