Difference between Heuristic Evaluation and Expert Review

17 Jun 2009 - 9:55pm
3 years ago
19 replies
5935 reads
kishor
2009

Hi all,

In Heuristic evaluation, evaluator evaluates design with reference of
design principles to find out the usability issues.
How “Expert Review” is different from this? Is there any major
difference or it’s the same?

Thanks in advance.

Regards,
Kishor
http://perceivedesign.wordpress.com/

Comments

17 Jun 2009 - 10:36pm
Anonymous

Hi Kishor,

It's a pretty fine line.

For a Heuristic Evaluation, an interface is judged against some
established rules of thumb (e.g. "visibility of system status" and
"match between system and the real world").

For an Expert Review, the reviewer judges the interface against
heuristics and their own personal experience/opinions on interface
design.

So pretty similiar. Hope this helps,

Cheers, Suze.

Suze Ingram
User Experience Consultant

suze.ingram at gmail.com
@suzeingram
http://suzeingram.blogspot.com/
http://www.linkedin.com/in/suzeingram

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42932

17 Jun 2009 - 11:54pm
kishor
2009

Thanks Suze for response.

Had some reading on this and got understanding that in Heuristics,
evaluator does not perform any task or evaluates any flow as such,
rather he just apply usability principles to find out usability
issues, whereas in expert review expert performs task, task
optimization, etc (a cognitive walkthrough).

So both approaches seems to be different.

Regards,
Kishor
http://perceivedesign.wordpress.com/

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42932

18 Jun 2009 - 2:37am
Shrikant Ekbote
2008

I don't think that's a correct differentiation between the two methods.
Principally both methods are same. Just two names. During heuristics
evaluation also an evaluator has to perform tasks and evaluate flows by
applying usability principles to find out if the end users is likely face
any issues while performing those tasks.

Regards,
Shrikant

On Thu, Jun 18, 2009 at 4:24 AM, Kishor Sonawane <cskishor at gmail.com> wrote:

> Thanks Suze for response.
>
> Had some reading on this and got understanding that in Heuristics,
> evaluator does not perform any task or evaluates any flow as such,
> rather he just apply usability principles to find out usability
> issues, whereas in expert review expert performs task, task
> optimization, etc (a cognitive walkthrough).
>
> So both approaches seems to be different.
>
> Regards,
> Kishor
> http://perceivedesign.wordpress.com/
>
>
>
>
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Posted from the new ixda.org
> http://www.ixda.org/discuss?post=42932
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

18 Jun 2009 - 4:45am
devesh jagatram
2005

Hi Kishor,

Although both are methods to evaluate any product on the aspects of
usability, interaction, design, navigation etc. But they still are two
completely different approaches to achieve this.

In Heuristic Evaluation, the product is simply evaluated against a set of
given rules, which are more or less best practices, and gives a checklist or
sometimes quantitative evaluation report on how the product's design will
work. It is more of a benchmarking for the product against a given set of
heuristics.

Whereas, an Expert Review is more of a qualitative evaluation. In this an
expert will step into user's shoes, go through whole product as a user might
go, and identify and document what works in the product and what doesn't. An
Expert Review is done with main focus to identify issues pertaining to
design in any product and identify specific areas where these issues occur.
Expert Review is widely used to evaluate a product when it is to be
redesigned. The findings are then used to come up with a proposed new
design. In an expert review, the reviewer brings in their expertise in
usability and design, domain experience, and also sometimes their personal
choices/biases. This is also a disadvantage of an expert review. There is a
possibility that the outcome of this evaluation is also based on the
evaluator's personal choices, or understanding of the product and its
domain.

I hope this helps.

Regards,
Devesh

On Thu, Jun 18, 2009 at 2:07 PM, shrikant ekbote
<shrikant.ekbote at gmail.com>wrote:

> I don't think that's a correct differentiation between the two methods.
> Principally both methods are same. Just two names. During heuristics
> evaluation also an evaluator has to perform tasks and evaluate flows by
> applying usability principles to find out if the end users is likely face
> any issues while performing those tasks.
>
> Regards,
> Shrikant
>
> On Thu, Jun 18, 2009 at 4:24 AM, Kishor Sonawane <cskishor at gmail.com>
> wrote:
>
> > Thanks Suze for response.
> >
> > Had some reading on this and got understanding that in Heuristics,
> > evaluator does not perform any task or evaluates any flow as such,
> > rather he just apply usability principles to find out usability
> > issues, whereas in expert review expert performs task, task
> > optimization, etc (a cognitive walkthrough).
> >
> > So both approaches seems to be different.
> >
> > Regards,
> > Kishor
> > http://perceivedesign.wordpress.com/
> >
> >
> >
> >
> > . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> > Posted from the new ixda.org
> > http://www.ixda.org/discuss?post=42932
> >
> >
> > ________________________________________________________________
> > Welcome to the Interaction Design Association (IxDA)!
> > To post to this list ....... discuss at ixda.org
> > Unsubscribe ................ http://www.ixda.org/unsubscribe
> > List Guidelines ............ http://www.ixda.org/guidelines
> > List Help .................. http://www.ixda.org/help
> >
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

18 Jun 2009 - 7:21am
Shrikant Ekbote
2008

Devesh,
Even while evaluating the product simply against heuristics the reviewer
will have to walk through the product and identify potential problem areas
so the method of evaluation remains the same.

While I agree that while doing Expert review the reviewer looks at the
product more holistically keeping in mind other aspects which are beyond
heuristics ( more as User Experience than just usability) but the reviewers
judgment of what might work for users and what will not is again based on
the heuristics.

So we can say that these two are not completely different approaches but
'heuristic evaluation' is a sub set of 'expert review'

Regards,
Shrikant

On Thu, Jun 18, 2009 at 4:15 PM, Devesh Jagatram
<devesh.jagatram at gmail.com>wrote:

> Hi Kishor,
>
> Although both are methods to evaluate any product on the aspects of
> usability, interaction, design, navigation etc. But they still are two
> completely different approaches to achieve this.
>
> In Heuristic Evaluation, the product is simply evaluated against a set of
> given rules, which are more or less best practices, and gives a checklist
> or
> sometimes quantitative evaluation report on how the product's design will
> work. It is more of a benchmarking for the product against a given set of
> heuristics.
>
> Whereas, an Expert Review is more of a qualitative evaluation. In this an
> expert will step into user's shoes, go through whole product as a user
> might
> go, and identify and document what works in the product and what doesn't.
> An
> Expert Review is done with main focus to identify issues pertaining to
> design in any product and identify specific areas where these issues occur.
> Expert Review is widely used to evaluate a product when it is to be
> redesigned. The findings are then used to come up with a proposed new
> design. In an expert review, the reviewer brings in their expertise in
> usability and design, domain experience, and also sometimes their personal
> choices/biases. This is also a disadvantage of an expert review. There is a
> possibility that the outcome of this evaluation is also based on the
> evaluator's personal choices, or understanding of the product and its
> domain.
>
> I hope this helps.
>
> Regards,
> Devesh
>
> On Thu, Jun 18, 2009 at 2:07 PM, shrikant ekbote
> <shrikant.ekbote at gmail.com>wrote:
>
> > I don't think that's a correct differentiation between the two methods.
> > Principally both methods are same. Just two names. During heuristics
> > evaluation also an evaluator has to perform tasks and evaluate flows by
> > applying usability principles to find out if the end users is likely face
> > any issues while performing those tasks.
> >
> > Regards,
> > Shrikant
> >
> > On Thu, Jun 18, 2009 at 4:24 AM, Kishor Sonawane <cskishor at gmail.com>
> > wrote:
> >
> > > Thanks Suze for response.
> > >
> > > Had some reading on this and got understanding that in Heuristics,
> > > evaluator does not perform any task or evaluates any flow as such,
> > > rather he just apply usability principles to find out usability
> > > issues, whereas in expert review expert performs task, task
> > > optimization, etc (a cognitive walkthrough).
> > >
> > > So both approaches seems to be different.
> > >
> > > Regards,
> > > Kishor
> > > http://perceivedesign.wordpress.com/
> > >
> > >
> > >
> > >
> > > . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> > > Posted from the new ixda.org
> > > http://www.ixda.org/discuss?post=42932
> > >
> > >
> > > ________________________________________________________________
> > > Welcome to the Interaction Design Association (IxDA)!
> > > To post to this list ....... discuss at ixda.org
> > > Unsubscribe ................ http://www.ixda.org/unsubscribe
> > > List Guidelines ............ http://www.ixda.org/guidelines
> > > List Help .................. http://www.ixda.org/help
> > >
> > ________________________________________________________________
> > Welcome to the Interaction Design Association (IxDA)!
> > To post to this list ....... discuss at ixda.org
> > Unsubscribe ................ http://www.ixda.org/unsubscribe
> > List Guidelines ............ http://www.ixda.org/guidelines
> > List Help .................. http://www.ixda.org/help
> >
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

18 Jun 2009 - 8:35am
Bonnie E. John
2008

Heuristic evaluation has been shown to work with evaluators who
_are_not_experts_. (See Nielsen's old papers.) That's a big difference
between the two.

However, you have to have many more non-expert evaluators to find the
same number of usability problems when the evaluators are not experts in
either the domain of the product or usability as when the evaluators are
expert in both domain and usability ("double experts"). Usability folks
who are not expert in the domain are in-between the non-experts and the
double-experts.

There is also evidence that Heuristic Evaluation finds many "false
alarms", i.e., "problems" that, if effort is put into fixing them, do
not improve users' performance (see Bailey for that work).

Bonnie

Kishor Sonawane wrote:
> Hi all,
>
> In Heuristic evaluation, evaluator evaluates design with reference of
> design principles to find out the usability issues.
> How “Expert Review” is different from this? Is there any major
> difference or it’s the same?
>
> Thanks in advance.
>
>
> Regards,
> Kishor
> http://perceivedesign.wordpress.com/
>
> ------------------------------------------------------------------------
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help

18 Jun 2009 - 9:13am
Tiffany MIchelon
2009

My question from here would be on your definition of "Expert". For
example, if I need an expert review of new accounting software, would
I find an accountant, or a User Experience Architect?

Even Devesh, who seems to make the best case for the distinction
between the 2 review types, still describes basically the same review
process for both types of review.

Is the distinction between the 2 simply in who performs the
evaluation?

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42932

18 Jun 2009 - 11:27am
Kevin Arthur
2009

There was a good panel at UPA last week about this very topic
("Heuristic Evaluation - Uses and Abuses"). On the panel were Jakob
Nielsen, Kyle Soucy, Rolf Molich, and Chauncy Wilson.

The points made at the panel echo what people have already written
above, but I'd encourage you to look at the proceedings when they
come out later this month (though I'm not sure what gets archived
for a panel).

Some of the points I remember:
- Rolf Molich distinguishes the techniques by saying that in expert
review the reviewer incorporates his/her "considerable expertise"
rather than just evaluating against a fixed list of heuristics. So I
think that in practice, ER is basically a broadened, more
flexible/inclusive version of HE, and (obviously) it requires that
the evaluator has expertise.
- Molich also said that he has found that in practice, when people
say they're doing heuristic evaluation they're really doing expert
review. In other words, nobody blindly looks at just the heuristics.
(My take-away from the panel was "don't do heuristic evaluation if
you're a robot.")
- Jakob Nielsen said that heuristic evaluation was never meant to be
an end in itself, just a discount method that's better than doing
nothing. And his 10 heuristics were never meant to be a complete
list, just a starting point.
- Everyone seemed to agree that if you have an evaluator with
expertise then by all means do an expert review instead of a
heuristic review.
- Molich thinks it's important to use the correct terminology, but I
think it's fair to say that to some in the audience (including me)
the difference in terms seems a bit academic. A lot of people seem to
use them interchangeably, and even the UPA's own body-of-knowledge
site seems to confuse the terms.

For more details on how to actually perform ER/HE, I'd check out
usability.gov, Jakob Nielsen's site, Bruce Tognazzini's (TOG) first
principles of interaction design, and STC's usability resources.

Kevin
http://www.touchusability.com

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42932

18 Jun 2009 - 8:04am
Will Hacker
2009

The approaches are very similar. I have been through the Forrester
site review exercise and it is pretty much a heuristic evaluation
with a little more qualitative input from the reviewer. I think of
heuristic evaluations as abbreviated expert reviews.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42932

18 Jun 2009 - 5:46pm
Robert Hoekman, Jr.
2005

>
> In Heuristic evaluation, evaluator evaluates design with reference of
> design principles to find out the usability issues.
> How “Expert Review” is different from this? Is there any major
> difference or it’s the same?
>

Anyone can evaluate a site using heuristics. Expert usability professionals
and designers, however, apply their knowledge, skill, talent, and experience
to the process to evaluate a design more thoroughly than by simply
traversing a
checklist. There may be little difference between the terms "Heuristic
review" and "Expert review", but there can
be a world of difference in their respective outcomes.

-r-

18 Jun 2009 - 7:27pm
Jared M. Spool
2003

On Jun 18, 2009, at 7:46 PM, Robert Hoekman Jr wrote:

>>
>> In Heuristic evaluation, evaluator evaluates design with reference of
>> design principles to find out the usability issues.
>> How “Expert Review” is different from this? Is there any major
>> difference or it’s the same?
>>
>
> Anyone can evaluate a site using heuristics. Expert usability
> professionals
> and designers, however, apply their knowledge, skill, talent, and
> experience
> to the process to evaluate a design more thoroughly than by simply
> traversing a
> checklist. There may be little difference between the terms "Heuristic
> review" and "Expert review", but there can
> be a world of difference in their respective outcomes.

While I'm spending my evening ironing out semantics, my understanding
of "expert reviews" is that they needn't be conducted by "usability
professionals."

An expert in medical diagnostics could perform an expert review of a
medical diagnostic system. Of course, the issues they would likely
find would be more about the domain than about the ergonomics or ease
of use, but it would still be a review based on expertise.

Jared

18 Jun 2009 - 11:55pm
kishor
2009

Jared, I've a question here, if a domain expert is going to perform
Expert Review then how it is possible for him to identify usability
issues? As main goal of expert review/HE is to find out usability
issues.

My second point id, how valid is this - without knowing user, user
goals, primary tasks we evaluate the design? So in an ideal scenario,
for an expert review do we need A Usability Expert, a domain expert
(SME) and access to user?

Regards,
Kishor
http://perceivedesign.wordpress.com/

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42932

19 Jun 2009 - 1:34am
Abhay Rautela
2008

Devesh hits the nail on the head when he differentiates the two
methods. Here's my own bit on helping explain the difference
further.

Both a heuristic evaluation and an expert review have the same goal-
to evaluate the usability of the product. The goal is the same, the
methods are however different.

Heuristic evaluation- is the evaluation of the usability of a product
against a set of heuristics. Issues are found and reported and
recommendations are made explicitly referencing this set of
heuristics.

Expert Review- is the evaluation of the usability of a product by an
expert in the usability domain and preferably in the domain the
product applies to. An expert may or may not directly refer to a set
of heuristics during the evaluation and while reporting issues and
recommendations. Besides this, an expert will evaluate the usability
of the product against what the expert has learnt throughout their
experience of working on usability of products- through data yielded
on their own or accessed from others%u2019 research.

It would be safe to say that higher the expertise of the evaluator
performing a heuristic evaluation or an expert review, the higher are
your chances of yielding useful results. Also, an expert review will
yield better results as compared to a heuristic evaluation when
performed by experts since this incorporates knowledge about the
domain and their own experience which in most cases may go beyond
what a set of heuristics might help one find.

However, in the case a usability evaluation has to be done by a group
of evaluators that do not have much experience in usability or the
domain the product applies to, a heuristic evaluation will have
higher chances of yielding better results than an expert review,
since these evaluators will have a set of heuristics (rules of thumb)
to refer to, as opposed to utilizing their own experience to make
judgments which in this case will be very limited and may yield a
considerable number of 'false alarms'.

I also agree with Molich as Kevins point out- it is all too common to
hear people terming an expert review a heuristic evaluation when in
actuality the evaluators evaluated the usability of the product
referring to their own knowledge of right and wrong rather than
explicitly referencing against a set of heuristics.

Hope this further helps clarify the difference between the two.

-Abhay

-- Cone Trees- User Research & Design
http://www.conetrees.com
http://www.twitter.com/conetrees
http://www.theuxbookmark.com
http://uxbookclub.org/doku.php?id=new_delhi
http://www.slideshare.net/group/web-accessibility

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42932

19 Jun 2009 - 9:03am
Sarah Weise
2009

I also sat in on the UPA session in Portland last week where an
esteemed panel spent a full 90 minutes bantering about the semantic
differences between a heuristic evaluation and an expert review.

Honestly, the difference is so small that does it really matter?

Any review is better than no review - whether you're using
Nielsen's 2,000-something heuristics, a sub-set of those, your own
set of heuristics, or heuristics with your own experience (which is
probably based on heuristics you picked up somewhere along the lines
anyway).

My "expert" recommendation: to stop arguing over semantics and
start actually making websites and applications easier to use.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42932

19 Jun 2009 - 11:32pm
Abhay Rautela
2008

"Honestly, the difference is so small that does it really matter?"

Sarah, honestly, it does. I'll quote Robert from what he said
earlier on in this thread. "There may be little difference between
the terms 'Heuristic review' and 'Expert review', but there can be
a world of difference in their respective outcomes."

The difference in outcome can be very significant, which I've
briefly described in my earlier reply.

-Abhay

-- Cone Trees- User Research & Design
http://www.conetrees.com
http://www.twitter.com/conetrees
http://www.theuxbookmark.com
http://uxbookclub.org/doku.php?id=new_delhi
http://www.slideshare.net/group/web-accessibility

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42932

20 Jun 2009 - 12:54pm
Jared M. Spool
2003

On Jun 17, 2009, at 3:55 PM, Kishor Sonawane wrote:

> In Heuristic evaluation, evaluator evaluates design with reference of
> design principles to find out the usability issues.
> How “Expert Review” is different from this? Is there any major
> difference or it’s the same?

As you can see from the answers, none of these differences are really
true.

"Heuristic Evaluation" and "Expert Review" are just labels that people
put on the activity they are doing, which is usually some sort of
criticism of a design. The actual distinctions are typically
meaningless.

I've seen "heuristic evaluations" that didn't involve any heuristics
at all. I've seen "expert reviews" where no-one in the room was
actually an expert beyond just being a smart person who could write a
report. Yet, everyone believed the labels and, in most cases, believed
the results.

I wouldn't worry about the naming.

That's my opinion -- worth what you paid for it.

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: jspool at uie.com p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks Twitter: @jmspool
UIE Roadshow: Seattle, Denver, DC in June: http://is.gd/gxwe

20 Jun 2009 - 3:07pm
Andrei Herasimchuk
2004

On Jun 20, 2009, at 11:54 AM, Jared Spool wrote:

> I've seen "heuristic evaluations" that didn't involve any heuristics
> at all. I've seen "expert reviews" where no-one in the room was
> actually an expert beyond just being a smart person who could write
> a report. Yet, everyone believed the labels and, in most cases,
> believed the results.

I'm going to quote that. Awesome.

--
Andrei Herasimchuk

Chief Design Officer, Involution Studios
innovating the digital world

e. andrei at involutionstudios.com
c. +1 408 306 6422

25 Nov 2011 - 10:22am
akheela
2011

What is the latest issue in usability evaluation that is being taken up for discussion?

28 Nov 2011 - 6:05am
edson rufino de...
2008

dear akheela,

i think that heuristic evaluation is a type of expert review.  we can apply the knowledge of experts in different methodologies, not neccessarily guided by Nielsen technique.

regards!
--
edson rufino de souza
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
email / gtalk / msn: edson.rufino@gmail.com
tel: +55-21-9805-6563
twitter: @edsonrufino

Em 28/11/2011 06:08, "akheela" <akheela.khanum@gmail.com> escreveu:

What is the latest issue in usability evaluation that is being taken up for discussion?

(
Syndicate content Get the feed