Nice Research on Persona Effectiveness

28 May 2009 - 7:03am
5 years ago
59 replies
5633 reads
Jared M. Spool
2003

I know that members on this list are dubious about using personas in
the design process.

Here's a nice, solid research paper by Frank Long at NCAD in Dublin
that shows how they can improve team dynamics:

> Real or imaginary: The effectiveness of using personas in the
> product design process.
>
> As you know, the use of personas as a method for communicating user
> requirements in collaborative design environments is well
> established. However, very little research has been conducted to
> quantify the benefits of using this technique.
>
> The aim of this study was to investigate the effectiveness of using
> personas. An experiment was conducted using students from NCAD. The
> results showed that, through using personas, designs with superior
> usability characteristics were produced. They also indicate that
> using personas provides a significant advantage during the research
> and conceptualisation stages of the design process (supporting
> previously unfounded claims).
>
> The study also investigated the effects of using different
> presentation methods to present personas and concluded that
> photographs worked better than illustrations, and that visual
> storyboards were more effective in presenting task scenarios than
> text only versions.
>
> The research: http://is.gd/I5Zk

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: jspool at uie.com p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks Twitter: @jmspool
UIE Roadshow: Seattle, Denver, DC in June: http://is.gd/gxwe

Comments

28 May 2009 - 8:28am
Jerome Ryckborst
2007

Jared: NICE. Thanks for sharing that link.

If the research below makes anyone (re)consider user personas, in the past
week I wrote about how I struggled with how many user personas we needed for
a massive software suite (600+ commands). I also happened to throw in some
advice about how to get them reviewed.
http://fivesketches.com/2009/05/how-many-user-personas/

-=- Jerome

-----Original Message-----
From: new-bounces at ixda.org [mailto:new-bounces at ixda.org] On Behalf Of Jared
Spool
Sent: Thu, 28 May, 2009 5:03 AM
To: discuss at ixda.org
Subject: [IxDA Discuss] Nice Research on Persona Effectiveness

I know that members on this list are dubious about using personas in
the design process.

Here's a nice, solid research paper by Frank Long at NCAD in Dublin
that shows how they can improve team dynamics:

> Real or imaginary: The effectiveness of using personas in the
> product design process.
>
> As you know, the use of personas as a method for communicating user
> requirements in collaborative design environments is well
> established. However, very little research has been conducted to
> quantify the benefits of using this technique.
>
> The aim of this study was to investigate the effectiveness of using
> personas. An experiment was conducted using students from NCAD. The
> results showed that, through using personas, designs with superior
> usability characteristics were produced. They also indicate that
> using personas provides a significant advantage during the research
> and conceptualisation stages of the design process (supporting
> previously unfounded claims).
>
> The study also investigated the effects of using different
> presentation methods to present personas and concluded that
> photographs worked better than illustrations, and that visual
> storyboards were more effective in presenting task scenarios than
> text only versions.
>
> The research: http://is.gd/I5Zk

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: jspool at uie.com p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks Twitter: @jmspool
UIE Roadshow: Seattle, Denver, DC in June: http://is.gd/gxwe

________________________________________________________________
Reply to this thread at ixda.org
http://www.ixda.org/discuss?post=42315

________________________________________________________________
Welcome to the Interaction Design Association (IxDA)!
To post to this list ....... discuss at ixda.org
Unsubscribe ................ http://www.ixda.org/unsubscribe
List Guidelines ............ http://www.ixda.org/guidelines
List Help .................. http://www.ixda.org/help

28 May 2009 - 9:10am
Mike Myles
2009

I've personally found personas to be very effective design tools, and
have frankly been surprised to hear so many negative opinions about
personas from designers on forums like these, at conferences, and in
my work over the last couple of years.

It's fantastic to see some research that not only shows personas can
be effective, but identifies the specific characteristics that work
best.

Great report! Thanks for the link.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42315

28 May 2009 - 10:10am
James Page
2008

I think the issue I have with Personas is that they are, as the paper points
out, "Fictional".
The paper compares three groups; one group that is briefed with photos of
personas, one which uses illustrations of the personas and the last group is
briefed to with no personas, and uses aesthetic design.
>From my reading of the paper is that people design better with a user in
mind, rather than having no user in mind. But is there a reason why one can
not use real people rather than "Fictional" people? It does not answer why
using fake people rather than using real people is an advantage. Is there
any reason why the techniques developed for persona can not be used with
real data subjects?

All the best

James

2009/5/28 Mike Myles <mmyles2001 at yahoo.com>

> I've personally found personas to be very effective design tools, and
> have frankly been surprised to hear so many negative opinions about
> personas from designers on forums like these, at conferences, and in
> my work over the last couple of years.
>
> It's fantastic to see some research that not only shows personas can
> be effective, but identifies the specific characteristics that work
> best.
>
> Great report! Thanks for the link.
>
>
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Posted from the new ixda.org
> http://www.ixda.org/discuss?post=42315
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

28 May 2009 - 1:06pm
Jared M. Spool
2003

On May 28, 2009, at 11:10 AM, James Page wrote:

> I think the issue I have with Personas is that they are, as the
> paper points
> out, "Fictional".

The paper was bounded by experimental constraints, like all research
is. Supplying fictional personas that represent the fictional users
for a fictional design project made sense for the study.

There are lots of ways to make personas. As I've discussed here
before, robust personas use a solid data background that eliminates
the absolute fiction from the process.

> But is there a reason why one can
> not use real people rather than "Fictional" people? It does not
> answer why
> using fake people rather than using real people is an advantage. Is
> there
> any reason why the techniques developed for persona can not be used
> with
> real data subjects?

There are good reasons to combine attributes into archetypal
caricatures, but that's not the point of this research. I suggest you
read this interview with Kim Goodwin (http://is.gd/IjFb) where she
states:

> Certainly there are some real people who are very similar to a
> persona the design team may create, but it's a dangerous approach
> because real humans are idiosyncratic. For example, any individual
> user might hate the color blue or have some other random opinions
> that aren’t necessarily representative of a larger population.
>
> One of the strength of personas is that they gloss over those little
> idiosyncratic things and really focus on the essence of what is
> common to this particular type of person. That's one of the reasons
> why we rely on personas instead of real users--they are more
> representative.

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: jspool at uie.com p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks Twitter: @jmspool
UIE Roadshow: Seattle, Denver, DC in June: http://is.gd/gxwe

28 May 2009 - 3:09pm
Will Hacker
2009

Jared raises a valuable point often left out of discussions of
personas as "fiction". Personas should be based on rich research
data. For every descriptive statement in a persona document, you
should be able to go back to the qualitative or quantitative research
to answer the question "where did that come from?".

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42315

28 May 2009 - 4:55pm
Robert Hoekman, Jr.
2005

>
> Here's a nice, solid research paper by Frank Long at NCAD in Dublin that
> shows how they can improve team dynamics:
>

Oh, man. This is hardly solid research. Now you're just begging for a
debate. ;)

1. They tested the effectiveness of personas by performing heuristic
evaluations. That's like testing the happiness of a cat by determining the
ground speed of a duck. Usability and the effectiveness of personas have
little to do with each other. Any decent designer can put something together
that does well in a heuristic evaluation — it doesn't mean the app meets the
needs of its audience. Not even remotely.

2. They say they set up the teams to be of relatively similar strength, but
how was that done, exactly? How do you make sure each team has a relatively
equal level of experience, skill, talent, knowledge, and an ability to
persuade? How do you set up three teams in a way that no one person on any
one team is able to talk the rest of that team into a bad idea based on
faulty knowledge? It may seem like I'm nitpicking on this one, but I think
this is a pretty important point. Human beings are slippery.

3. The paper doesn't indicate what the control group used instead of
personas beyond "image boards" (which are meaningless as research tools).
All three of these teams were comprised of people from the same class —
meaning they all had received education on personas by the same instructor.
What else did the instructor teach these people that they could put to use
to come up with a good design without personas?

The only thing this study shows is that 2 out of the 3 teams created a more
usable design as measured against heuristics (this assumes, of course (and
it's a big assumption), that the evaluators did good evaluations), and that
they happened to be the same groups that used personas in the project. At
the absolute best, this is a loose correlation. It's absolutely not proof of
genuine causation. I could have fared as well as any of them without
personas and without a team.

4. Even if you throw out arguments #2 and #3 above, #1 still makes it all a
moot point.

All that said, I still love you Jared. :)

-r-

28 May 2009 - 4:58pm
Robert Hoekman, Jr.
2005

>
> Jared raises a valuable point often left out of discussions of
> personas as "fiction". Personas should be based on rich research
> data. For every descriptive statement in a persona document, you
> should be able to go back to the qualitative or quantitative research
> to answer the question "where did that come from?".
>

Well, sure, but that doesn't mean persona descriptions aren't fictitious.
Double-negatives aside, "based on a true story" is different than "true
story".

-r-

28 May 2009 - 5:42pm
Robert Reimann
2003

The straw man argument that "personas aren't real" isn't a fair
characterization when proper data gathering and analysis techniques are
appropriately applied. Think of personas as putting a human face on sets of
behavioral variables or dimensions. The behaviors are real, correlated, and
composited for the purpose of separating idiosycratic behaviors of
individuals from common attitudes, actions, and mental models of a
behavioral cohort. Whether this is done via visual inspection of plotted
data (as we frequently did at Cooper when we were developing the methods) ,
or more rigorously (in the case of larger samples) by methods like primary
factor analysis, it is most certainly real data based on real observation,
and not in any measure "fictional". Any "proper" persona can (given the time
and resurces to do so) be validated with a mix of qualitative and
quantitative methods, via the use of psychographic surveys and the like.

That said, some clients and practitioners do have difficulty with the
narrative aspect of personas: giving them a character photo or sketch and
composite backstory. The application of any amount of created persona
content is viewed suspciously (as perhaps it initially should), and equated
with falsifying the evidence.

But the truth this mechanism is designed to point toward is an emotional
truth... an empathetic connection to these otherwise abstract constellations
of observed behaviors. It is a storytelling technique, using omission and
minor embellishment not to alter the facts or pull the wool over anyone's
eyes, but rather to communicate and accentuate the underlying reality and
truth being represented. Unless we believe that novels describing the human
condition contain only falsehoods and deceits, we must concede that
storytelling methods can, when employed prudently, clarify rather than
obscure the truth of human narratives and behaviors.

Robert.

Robert Reimann
IxDA Seattle

Associate Creative Director
frog design
Seattle, WA

On Thu, May 28, 2009 at 2:58 PM, Robert Hoekman Jr <robert at rhjr.net> wrote:

> >
> > Jared raises a valuable point often left out of discussions of
> > personas as "fiction". Personas should be based on rich research
> > data. For every descriptive statement in a persona document, you
> > should be able to go back to the qualitative or quantitative research
> > to answer the question "where did that come from?".
> >
>
> Well, sure, but that doesn't mean persona descriptions aren't fictitious.
> Double-negatives aside, "based on a true story" is different than "true
> story".
>
> -r-
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

28 May 2009 - 6:31pm
Robert Hoekman, Jr.
2005

>
> The straw man argument that "personas aren't real" isn't a fair
> characterization when proper data gathering and analysis techniques are
> appropriately applied.

Well, they don't eat, they don't sleep, and they don't pay taxes, and you
have to do at least two of those things to be a real, living person.
Personas may be thoroughly based on real people, but calling them "real" is
like calling a movie that is thoroughly based on a real series of events a
"documentary".

Look, I'm sure personas are all kinds of useful for people who believe
they're all kinds of useful. I'm not here to debate that. I was just
pointing out that when it comes to actual fact, personas aren't real. My
response was meant as a clarification, not an argument.

(I've got to stop taking the bait on threads like this.)

-r-

28 May 2009 - 7:07pm
Angel Marquez
2008

Their should be a SIMS <http://thesims3.ea.com/> mirror community of the
IxDA community and the personas created by your sim character would be a sim
within a sim...
Personas are for women.

28 May 2009 - 10:59pm
Jared M. Spool
2003

On May 28, 2009, at 8:07 PM, Angel Marquez wrote:

> Personas are for women.

Dude, do you ever make sense?

28 May 2009 - 11:17pm
Angel Marquez
2008

Not to just anyone, dude.

29 May 2009 - 12:53am
Angel Marquez
2008

After a little more thought
[redact]FUCK YOU[/redact]

29 May 2009 - 1:11am
Robert Hoekman, Jr.
2005

That was absolutely uncalled for, and incredibly unproductive.

-r-

Sent from 602.421.5800

On May 28, 2009, at 10:53 PM, Angel Marquez <angel.marquez at gmail.com>
wrote:

> After a little more thought
>
> XXXX XXX

28 May 2009 - 9:34pm
femmebot
2009

Was there an actual link to the paper or did it just include the
excerpt? Jerome mentioned a link but I can't seem to find it.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42315

29 May 2009 - 1:08am
Karen Summers
2009

PC Version: I find I must object to the tone of the recent messages
in this thread.
Teenage Version: OMG! Is this guy fer real?
Honest Version: This is why I never did the "chat" thing.

Personas are a useful tool, like a WYSIWYG editor, but are not for
everyone. You could stab a man in the lung with a screwdriver too,
but I wouldn't recommend it unless you were a tinker.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42315

29 May 2009 - 1:21am
Sachin Ghodke
2008

Angel, it was seriously uncalled for. it does not augur well for IxDA
community, D - U - D - E. Did that DUDE make a point?

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42315

29 May 2009 - 1:48am
Angel Marquez
2008

Send me the link to remove myself and I will click it.

Sent from my iPhone

On May 28, 2009, at 11:21 PM, Sachin Ghodke <sachyn.ghodke at gmail.com>
wrote:

> Angel, it was seriously uncalled for. it does not augur well for IxDA
> community, D - U - D - E. Did that DUDE make a point?
>
>
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Posted from the new ixda.org
> http://www.ixda.org/discuss?post=42315
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help

29 May 2009 - 2:06am
Angel Marquez
2008

The PC people I associate with would have ignored it and myane laughed
about in their secret groups.

The teenagers I hung out with would have bought me a case of beer,
cheap beer but quantity.

Honestly, I thought it was fair and made sense.

I already sent an explanation to the IxDA police. I'd rather not have
to repeat myself... the email I replied to was outside of the thread.
I could forward if you like.

Sent from my iPhone

On May 28, 2009, at 11:08 PM, Karen Summers <kdsummers at gmail.com> wrote:

> PC Version: I find I must object to the tone of the recent messages
> in this thread.
> Teenage Version: OMG! Is this guy fer real?
> Honest Version: This is why I never did the "chat" thing.
>
> Personas are a useful tool, like a WYSIWYG editor, but are not for
> everyone. You could stab a man in the lung with a screwdriver too,
> but I wouldn't recommend it unless you were a tinker.
>
>
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Posted from the new ixda.org
> http://www.ixda.org/discuss?post=42315
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help

29 May 2009 - 4:41am
Frank Long
2009

Oh, man. This is hardly solid research. Now you're just begging for a
debate. ; )

1. They tested the effectiveness of personas by performing heuristic
evaluations. That's like testing the happiness of a cat by
determining the ground speed of a duck. Usability and the
effectiveness of personas have little to do with each other. Any
decent designer can put something together that does well in a
heuristic evaluation %u2014 it doesn't mean the app meets the needs
of its audience. Not even remotely.

2. They say they set up the teams to be of relatively similar
strength, but how was that done, exactly? How do you make sure each
team has a relatively equal level of experience, skill, talent,
knowledge, and an ability to persuade? How do you set up three teams
in a way that no one person on any one team is able to talk the rest
of that team into a bad idea based on faulty knowledge? It may seem
like I'm nitpicking on this one, but I think this is a pretty
important point. Human beings are slippery.

3. The paper doesn't indicate what the control group used instead of
personas beyond "image boards" (which are meaningless as research
tools). All three of these teams were comprised of people from the
same class %u2014 meaning they all had received education on personas
by the same instructor. What else did the instructor teach these
people that they could put to use to come up with a good design
without personas?

The only thing this study shows is that 2 out of the 3 teams created
a more usable design as measured against heuristics (this assumes, of
course (and it's a big assumption), that the evaluators did good
evaluations), and that they happened to be the same groups that used
personas in the project. At the absolute best, this is a loose
correlation. It's absolutely not proof of genuine causation. I could
have fared as well as any of them without personas and without a team.

4. Even if you throw out arguments # 2 and # 3 above, # 1 still makes
it all a moot point.

All that said, I still love you Jared. : )

-r-

Hi Robert,
Thanks for taking the time to read the paper %u2013 In answer to the
3 points that you made regarding the validity of the paper.

Point 1 %u2013 happiness of cats V ground speed of ducks.

Everyone knows that cats like to chase ducks. The happiness of the
domestic feline is therefore inversely proportional to the ground
speed of said duck.
But seriously, the research did not use heuristics to evaluate the
effectiveness of personas. I used heuristics to evaluate the
usability of the resulting designs %u2013 allowing me to compare all
of the diverse design solutions consistently with each other. (user
testing would have been preferable but not feasible). The measure of
effectiveness was based on a number of factors including the
usability of the solution, interviews with students, observations
from their tutors and group discussions after the project. My initial
research question was to see if using personas made any difference
%u2013 and based on the research I found that they did.

Point 2 %u2013 teams of equal strength.

The students had completed 2.5 years of industrial design course when
I conducted the experiment. The course is entirely project based. The
same tutors work with the students day-in, day-out. After 2.5 years
you have a pretty accurate idea of each student%u2019s ability.
The fact that all students were working in close proximity was an
acknowledged weakness of the study. To work around this unavoidable
problem we informed all students of the experimental nature of the
project %u2013 not the specific goals, but the fact that each group
would be using a different design tool to solve the brief. The
importance of secrecy between the groups was stressed and we simply
asked for the student%u2019s cooperation in this regard. At the end
of the research we gathered feedback on the level of
%u2018information leaks%u2019 between groups. While a small degree
of information did pass between groups, it was not significant.

Point 3 %u2013 The control group

All teams received a set of briefing documents outlining the product
specifications, the manufacturing constraints, and a market research
file outlining the target user demographics. The brief stated that
the user-friendliness of the product was of paramount importance
%u2013 especially the set-up task as this was seen as a barrier to
product sales. The market research outlined the user profile and the
personas were created from this same user profile.
The image boards were given to the control group covered 3 areas,
Product environment, lifestyle and brand landscape. The items shown
on the image boards were also mentioned in the personas so that the
information provided to all teams were as uniform as possible.

Point 4 - conclusion
The conclusions of the study find that using personas is an effective
tool and did produce more user-focused solutions. It also acknowledges
that personas are only one of many design tools %u2013 and does not
claim that it is the most effective method. As the saying
goes%u2026there is more than one way to skin a cat, or to make a duck
happy.

:)

Regards
Frank

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42315

29 May 2009 - 5:00am
Frank Long
2009

the link to the research paper

http://www.frontend.com/products-digital-devices/real-or-imaginary-the-effectiveness-of-using-personas-in-product-design.html

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42315

29 May 2009 - 5:12am
James Page
2008

Jared,

I think my issues are more philophosical. How do you separate persona's from
been a Cargo Science?

> In the South Seas there is a cargo cult of people. During the war they saw
> airplanes with lots of good materials, and they want the same thing to
> happen now. So they've arranged to make things like runways, to put fires
> along the sides of the runways, to make a wooden hut for a man to sit in,
> with two wooden pieces on his head to headphones and bars of bamboo sticking
> out like antennas -- he's the controller -- and they wait for the airplanes
> to land. They're doing everything right. The form is perfect. It looks
> exactly the way it looked before. But it doesn't work. No airplanes land.
> "Surely You're Joking, Mr. Feynman!"

My argument is just because you have based your persona on data, does not
make it robust. Is there a way to test your persona without falling into an
Inductive trap? (An inductive trap is for example making a statement that "A
turkey gets fed every day, therefore on Thanksgiving it gets fed.")

Your interview with Kim Goodwin (http://is.gd/IjFb) is interesting, but she
does not deal with the inductive problem. The two issues that she has about
using real people can be dealt with by increasing the number of participants
used, and by limiting the number of variables. I think Feynman said that
with a model that had 5 variables he could prove anything.

Would using the persona as hypothesis solve the inductive issue?

My second issue is coming from Europe is the danger of the Stereotype. It
may be possible in the US to create maybe a couple of persona's that
describe the market, but in Europe we have over 50 countries. Maybe if your
product is aiming at a narrow market it may be homogenised, but for a more
general product, each country population has a different level of
knowledge. Knowledge effects behaviour. For example Credit Card usage,
Internet Banking, Online Commerce varies greatly country to country, and
even compared to two countries like the UK and Germany.

How do you get around the diversity issue?

All the best

James
http://blog.feralabs.com

2009/5/28 Jared Spool <jspool at uie.com>

>
> On May 28, 2009, at 11:10 AM, James Page wrote:
>
> I think the issue I have with Personas is that they are, as the paper
> points
> out, "Fictional".
>
>
> The paper was bounded by experimental constraints, like all research is.
> Supplying fictional personas that represent the fictional users for a
> fictional design project made sense for the study.
>
> There are lots of ways to make personas. As I've discussed here before,
> robust personas use a solid data background that eliminates the absolute
> fiction from the process.
>
> But is there a reason why one can
> not use real people rather than "Fictional" people? It does not answer why
> using fake people rather than using real people is an advantage. Is there
> any reason why the techniques developed for persona can not be used with
> real data subjects?
>
>
> There are good reasons to combine attributes into archetypal caricatures,
> but that's not the point of this research. I suggest you read this interview
> with Kim Goodwin (http://is.gd/IjFb) where she states:
>
> Certainly there are some real people who are very similar to a persona the
> design team may create, but it's a dangerous approach because real humans
> are idiosyncratic. For example, any individual user might hate the color
> blue or have some other random opinions that aren’t necessarily
> representative of a larger population.
>
> One of the strength of personas is that they gloss over those little
> idiosyncratic things and really focus on the essence of what is common to
> this particular type of person. That's one of the reasons why we rely on
> personas instead of real users--they are more representative.
>
>
> Jared
>
> Jared M. Spool
> User Interface Engineering
> 510 Turnpike St., Suite 102, North Andover, MA 01845
> e: jspool at uie.com p: +1 978 327 5561
> http://uie.com Blog: http://uie.com/brainsparks Twitter: @jmspool
> UIE Roadshow: Seattle, Denver, DC in June: http://is.gd/gxwe
>

29 May 2009 - 5:36am
James Page
2008

Frank,

An interesting study.

I think the issue with using heuristic evaluations is the well known issue
with the evaluator effect.

For example see:
http://akira.ruc.dk/~mhz/Research/Publ/IJHCI2001_preprint.pdf<http://akira.ruc.dk/%7Emhz/Research/Publ/IJHCI2001_preprint.pdf>

On to the issue of the brief.
Was the market research given to the teams both Qualitative and
Quantitative?

All the best

James
http://blog.feralabs.com

2009/5/29 frank long <frank at frontend.com>

> Oh, man. This is hardly solid research. Now you're just begging for a
> debate. ; )
>
> 1. They tested the effectiveness of personas by performing heuristic
> evaluations. That's like testing the happiness of a cat by
> determining the ground speed of a duck. Usability and the
> effectiveness of personas have little to do with each other. Any
> decent designer can put something together that does well in a
> heuristic evaluation %u2014 it doesn't mean the app meets the needs
> of its audience. Not even remotely.
>
> 2. They say they set up the teams to be of relatively similar
> strength, but how was that done, exactly? How do you make sure each
> team has a relatively equal level of experience, skill, talent,
> knowledge, and an ability to persuade? How do you set up three teams
> in a way that no one person on any one team is able to talk the rest
> of that team into a bad idea based on faulty knowledge? It may seem
> like I'm nitpicking on this one, but I think this is a pretty
> important point. Human beings are slippery.
>
> 3. The paper doesn't indicate what the control group used instead of
> personas beyond "image boards" (which are meaningless as research
> tools). All three of these teams were comprised of people from the
> same class %u2014 meaning they all had received education on personas
> by the same instructor. What else did the instructor teach these
> people that they could put to use to come up with a good design
> without personas?
>
> The only thing this study shows is that 2 out of the 3 teams created
> a more usable design as measured against heuristics (this assumes, of
> course (and it's a big assumption), that the evaluators did good
> evaluations), and that they happened to be the same groups that used
> personas in the project. At the absolute best, this is a loose
> correlation. It's absolutely not proof of genuine causation. I could
> have fared as well as any of them without personas and without a team.
>
> 4. Even if you throw out arguments # 2 and # 3 above, # 1 still makes
> it all a moot point.
>
> All that said, I still love you Jared. : )
>
> -r-
>
> Hi Robert,
> Thanks for taking the time to read the paper %u2013 In answer to the
> 3 points that you made regarding the validity of the paper.
>
> Point 1 %u2013 happiness of cats V ground speed of ducks.
>
> Everyone knows that cats like to chase ducks. The happiness of the
> domestic feline is therefore inversely proportional to the ground
> speed of said duck.
> But seriously, the research did not use heuristics to evaluate the
> effectiveness of personas. I used heuristics to evaluate the
> usability of the resulting designs %u2013 allowing me to compare all
> of the diverse design solutions consistently with each other. (user
> testing would have been preferable but not feasible). The measure of
> effectiveness was based on a number of factors including the
> usability of the solution, interviews with students, observations
> from their tutors and group discussions after the project. My initial
> research question was to see if using personas made any difference
> %u2013 and based on the research I found that they did.
>
> Point 2 %u2013 teams of equal strength.
>
> The students had completed 2.5 years of industrial design course when
> I conducted the experiment. The course is entirely project based. The
> same tutors work with the students day-in, day-out. After 2.5 years
> you have a pretty accurate idea of each student%u2019s ability.
> The fact that all students were working in close proximity was an
> acknowledged weakness of the study. To work around this unavoidable
> problem we informed all students of the experimental nature of the
> project %u2013 not the specific goals, but the fact that each group
> would be using a different design tool to solve the brief. The
> importance of secrecy between the groups was stressed and we simply
> asked for the student%u2019s cooperation in this regard. At the end
> of the research we gathered feedback on the level of
> %u2018information leaks%u2019 between groups. While a small degree
> of information did pass between groups, it was not significant.
>
> Point 3 %u2013 The control group
>
> All teams received a set of briefing documents outlining the product
> specifications, the manufacturing constraints, and a market research
> file outlining the target user demographics. The brief stated that
> the user-friendliness of the product was of paramount importance
> %u2013 especially the set-up task as this was seen as a barrier to
> product sales. The market research outlined the user profile and the
> personas were created from this same user profile.
> The image boards were given to the control group covered 3 areas,
> Product environment, lifestyle and brand landscape. The items shown
> on the image boards were also mentioned in the personas so that the
> information provided to all teams were as uniform as possible.
>
> Point 4 - conclusion
> The conclusions of the study find that using personas is an effective
> tool and did produce more user-focused solutions. It also acknowledges
> that personas are only one of many design tools %u2013 and does not
> claim that it is the most effective method. As the saying
> goes%u2026there is more than one way to skin a cat, or to make a duck
> happy.
>
> :)
>
> Regards
> Frank
>
>
>
>
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Posted from the new ixda.org
> http://www.ixda.org/discuss?post=42315
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

29 May 2009 - 8:01am
Michael Stiso
2006

Frank,

Good job on trying to empirically investigate this issue -- it's a
thorny one. That said, though, I do have a concern about your
methods.

Am I correct in thinking that you were the sole evaluator of the
final products of the three groups, at least in terms of the
heuristic analysis? And given the constraints of the study, I'm
guessing you also knew -- prior to that analysis -- which groups used
the personas and which didn't?

If so, then the possibility of experimenter bias really hurts your
conclusions. Subjective analyses of performance (e.g., heuristic
analysis) are pliable, and if you knew beforehand which groups were
using personas and which weren't, it would be easy to mold the
results into something that matches your expectations.

On the other hand, if you were blind to that condition, then that
would give your conclusions a little more weight.

Mike

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42315

29 May 2009 - 8:19am
Jared M. Spool
2003

On May 29, 2009, at 6:12 AM, James Page wrote:

> Would using the persona as hypothesis solve the inductive issue?

We've discussed this at length before. Give me an example of *any*
research that, in your mind, solves the "inductive issue" and we can
talk about it further.

> How do you get around the diversity issue?

More thorough research. User research in design, like the design
process itself, doesn't shoot for perfection out of the gate. Instead,
it looks for incremental improvement, bring more information into the
design process on every iteration.

Jared

29 May 2009 - 8:28am
Jared M. Spool
2003

On May 29, 2009, at 6:36 AM, James Page wrote:

> I think the issue with using heuristic evaluations is the well known
> issue
> with the evaluator effect.

Wow, James. You are *so* missing the point here.

Sure, there are evaluator effects in heuristic evaluations. But that
has to do with *different* evaluators inspecting the *same* design.

In this case, Frank had the *same* evaluators inspecting *different*
designs.

One can shoot holes in any research study citing any of the myriads
previous research (which itself could be shot full of holes). That's
not the point.

The point is this is a nicely done study (I've reviewed hundreds of
studies like this in my career -- trust me, this is excellent) *and*
it does do what it set out to do, which is to see if given personas,
design teams behaved differently and in a way that *could* lead to
better design.

Jared

29 May 2009 - 8:42am
Jared M. Spool
2003

On May 28, 2009, at 5:55 PM, Robert Hoekman Jr wrote:

> Oh, man. This is hardly solid research. Now you're just begging for
> a debate. ;)

Trust me. Compared to many of the hundreds of research papers I review
each year, this one is pretty solid.

And maybe, just maybe, I was begging for debate.

Oh, and debate is for women. (Wait, no, I was thinking of something
else. Nevermind,)

> 1. ...
>
> 2. ...
>
> 3. ...

I actually think Frank did a nice job of defending his study, so I
won't do that here.

> The only thing this study shows is that 2 out of the 3 teams created
> a more usable design as measured against heuristics (this assumes,
> of course (and it's a big assumption), that the evaluators did good
> evaluations), and that they happened to be the same groups that used
> personas in the project. At the absolute best, this is a loose
> correlation. It's absolutely not proof of genuine causation. I could
> have fared as well as any of them without personas and without a team.
>
> 4. Even if you throw out arguments #2 and #3 above, #1 still makes
> it all a moot point.

No, no, no. You're looking at this wrong.

Scientific research studies like this are little building blocks. You
disassemble the problem into little problems, evaluate each problem,
then reassemble them to build your case.

While the bigger problem is, "Do teams that employ personas produce
effective designs for their audiences?" that, as you've correctly
pointed out, is hard to prove in a study. So, you break it down.

What this study does (in a very sweet, nice way) tackle one small
aspect of the problem: if you take a group of designers, break them
up, give some personas and others not, do you see different results.
The null hypothesis is, if personas don't make a differences, then the
control group (the folks w/o personas) will not produce
distinguishably different results from those that do.

This study contradicts the null hypothesis, because the teams with
personas produced different results based on the criteria (heuristic
evaluation).

Now, as you rightly point out, we can question the criteria (and
should!). That would be a different study. What criteria would you
like Frank or other researchers to measure against? That's the next
building block.

Also, we could perform the same study with different user research
tools. What tools would you like to see studied?

Of course, this study can't stand alone. Good research, like this,
needs to be duplicated elsewhere before you can really stand behind
it. Other researchers should try to replicate the experiment to see if
they get similar results. Then, and only then, will we empirically
know that the results were great. Frank did a good job of explaining
to another team how to duplicate the experiment.

You have to take this type of research for what it is and not expect a
single study to prove everything. I found it fascinating because it
duplicated what we've seen in our research in a controlled setting:
teams that use personas have a different dynamic than teams that don't
-- a dynamic that, in my opinion, leads to better design.

> All that said, I still love you Jared. :)

Oh, Robert, you certainly know how to make a guy swoon. Hugs & Kisses.

Jared

29 May 2009 - 8:56am
Joshua Porter
2007

On May 29, 2009, at 9:28 AM, Jared Spool wrote:

>
> On May 29, 2009, at 6:36 AM, James Page wrote:
>
>> I think the issue with using heuristic evaluations is the well
>> known issue
>> with the evaluator effect.
>
> Wow, James. You are *so* missing the point here.
>
> Sure, there are evaluator effects in heuristic evaluations. But that
> has to do with *different* evaluators inspecting the *same* design.

If I understand James correctly, he is suggesting that if the
evaluators knew which designers used personas then they would be
biased from judging objectively which designs were more user-centered.
That is, the so-called differences uncovered by the heuristic
evaluation might only be biases introduced by the evaluators.

If this is the case (and it's not clear from the writeup exactly what
was done), then the results of the study are biased.

Frank, could you shed some more light on this? Did the evaluators know
which designs were done using personas and which were not?

Cheers,

Josh

Joshua Porter, Founder
Bokardo Design
Interface design & strategy for social web applications
phone: 508-954-1896
http://bokardo.com
porter at bokardo.com
twitter: bokardo

29 May 2009 - 9:05am
Jared M. Spool
2003

On May 29, 2009, at 1:53 AM, Angel Marquez wrote:

> After a little more thought
>
> XXXX XXX

And that succinctly answers my question.

The defense rests, Your Honor.

29 May 2009 - 9:16am
Leisa Reichelt
2006

> Oh, and debate is for women. (Wait, no, I was thinking of something else.
> Nevermind,)
>

Just on the off chance anyone thinks that continuing this meme would be
funny
It's not.

(and I just got done explaining elsewhere why the first time it came up it
didn't bother me so it's not that I don't have a sense of humour about these
things)
________________________
Leisa Reichelt
Disambiguity.com
Contextual Research, User Centred Design & Social Design

leisa at disambiguity.com
+44 778 071 2129

2009/5/29 Jared Spool <jspool at uie.com>

>
> On May 28, 2009, at 5:55 PM, Robert Hoekman Jr wrote:
>
> Oh, man. This is hardly solid research. Now you're just begging for a
>> debate. ;)
>>
>
> Trust me. Compared to many of the hundreds of research papers I review each
> year, this one is pretty solid.
>
> And maybe, just maybe, I was begging for debate.
>
> Oh, and debate is for women. (Wait, no, I was thinking of something else.
> Nevermind,)
>
> 1. ...
>>
>> 2. ...
>>
>> 3. ...
>>
>
> I actually think Frank did a nice job of defending his study, so I won't do
> that here.
>
> The only thing this study shows is that 2 out of the 3 teams created a
>> more usable design as measured against heuristics (this assumes, of course
>> (and it's a big assumption), that the evaluators did good evaluations), and
>> that they happened to be the same groups that used personas in the project.
>> At the absolute best, this is a loose correlation. It's absolutely not proof
>> of genuine causation. I could have fared as well as any of them without
>> personas and without a team.
>>
>> 4. Even if you throw out arguments #2 and #3 above, #1 still makes it all
>> a moot point.
>>
>
> No, no, no. You're looking at this wrong.
>
> Scientific research studies like this are little building blocks. You
> disassemble the problem into little problems, evaluate each problem, then
> reassemble them to build your case.
>
> While the bigger problem is, "Do teams that employ personas produce
> effective designs for their audiences?" that, as you've correctly pointed
> out, is hard to prove in a study. So, you break it down.
>
> What this study does (in a very sweet, nice way) tackle one small aspect of
> the problem: if you take a group of designers, break them up, give some
> personas and others not, do you see different results. The null hypothesis
> is, if personas don't make a differences, then the control group (the folks
> w/o personas) will not produce distinguishably different results from those
> that do.
>
> This study contradicts the null hypothesis, because the teams with personas
> produced different results based on the criteria (heuristic evaluation).
>
> Now, as you rightly point out, we can question the criteria (and should!).
> That would be a different study. What criteria would you like Frank or other
> researchers to measure against? That's the next building block.
>
> Also, we could perform the same study with different user research tools.
> What tools would you like to see studied?
>
> Of course, this study can't stand alone. Good research, like this, needs to
> be duplicated elsewhere before you can really stand behind it. Other
> researchers should try to replicate the experiment to see if they get
> similar results. Then, and only then, will we empirically know that the
> results were great. Frank did a good job of explaining to another team how
> to duplicate the experiment.
>
> You have to take this type of research for what it is and not expect a
> single study to prove everything. I found it fascinating because it
> duplicated what we've seen in our research in a controlled setting: teams
> that use personas have a different dynamic than teams that don't -- a
> dynamic that, in my opinion, leads to better design.
>
> All that said, I still love you Jared. :)
>>
>
> Oh, Robert, you certainly know how to make a guy swoon. Hugs & Kisses.
>
> Jared
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

29 May 2009 - 9:27am
Jared M. Spool
2003

On May 29, 2009, at 10:16 AM, Leisa Reichelt wrote:

>
> Oh, and debate is for women. (Wait, no, I was thinking of something
> else. Nevermind,)
>
>
> Just on the off chance anyone thinks that continuing this meme would
> be funny
> It's not.
>
> (and I just got done explaining elsewhere why the first time it came
> up it didn't bother me so it's not that I don't have a sense of
> humour about these things

But it was such a fabulous non-sequitor!

Ok. I'll stop now.

Jared

29 May 2009 - 11:31am
Angel Marquez
2008

Whatever makes you feel better.

Sent from my iPhone

On May 29, 2009, at 7:05 AM, Jared Spool <jspool at uie.com> wrote:

>
> On May 29, 2009, at 1:53 AM, Angel Marquez wrote:
>
>> After a little more thought
>>
>> XXXX XXX
>
> And that succinctly answers my question.
>
> The defense rests, Your Honor.
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help

29 May 2009 - 11:40am
Dan Saffer
2003

You're still here?

Seriously, list admin and IxDA Board? We allow this kind of abuse? Of
Jared? From someone whose only contribution to the list has been
pointless noise?

Someone is asleep at the switch.

Dan

On May 29, 2009, at 9:31 AM, Angel wrote:

> Whatever makes you feel better.
>
> Sent from my iPhone
>
> On May 29, 2009, at 7:05 AM, Jared Spool <jspool at uie.com> wrote:
>
>>
>> On May 29, 2009, at 1:53 AM, Angel Marquez wrote:
>>
>>> After a little more thought
>>>
>>> XXXX XXX
>>
>> And that succinctly answers my question.
>>
>> The defense rests, Your Honor.
>> ___

29 May 2009 - 12:27pm
Janna DeVylder
2006

As stated in my earlier post about the discussion list guidelines, it is not
allowed.
We're a volunteer organization with a small group of moderators around the
world.
We are addressing this.

If anyone would like to volunteer to help out, that would also be welcome.

Janna

On Fri, May 29, 2009 at 12:40 PM, Dan Saffer <dan at odannyboy.com> wrote:

> You're still here?
>
> Seriously, list admin and IxDA Board? We allow this kind of abuse? Of
> Jared? From someone whose only contribution to the list has been pointless
> noise?
>
> Someone is asleep at the switch.
>
> Dan
>
>
>

29 May 2009 - 11:05am
Kathi Kaiser
2009

@Frank: Thank you for conducting research to explore this important
question about the role of personas in the design process. I%u2019m
curious about the potential impact of other flavors of personas, such
as those that describe a class rather than an individual. For
example, at Centralis we often summarize user research findings by
profiling types of users based on their core activities (e.g.,
%u201CThe Consultant%u201D v. %u201CThe Clerk%u201D for different
approaches to manufacturing order entry). Another variable in
personas is whether or not demographic or psychographic information
is included along with the user%u2019s interaction with the task
under study. Has your research explored any of these variations?

Beyond the role of personas for the design team, it%u2019s also
important to understand the impact of personas on selling user
research findings and subsequent designs to others in the
organization. Do executives find personas to be useful
personifications of their target customers, or do they dismiss them
as qualitative storytelling? What characteristics of personas would
make them more and less compelling to stakeholders beyond the design
team?

Kathi Kaiser
Centralis
www.centralis.com

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42315

29 May 2009 - 1:03pm
Greg Petroff
2004

I'm sad at the tone of this conversation and the specific demeaning
comments. Demeaning language should not be tolerated and it's not PC
to believe this, it just common decency.

In addition this community has to try to be less "snarky".

IxDA is a place where people CHOOSE to share what they know with each
other. This is the GREAT thing about IxDA. The sharing wont happen if
a few start to behave badly.

We are a global community...Understanding what's well intentioned
sarcasm versus snarkiness in someone's comments is difficult for me.
I imagine it is difficult for others.

So if you not willing to be here in the spirit of sharing what you
know in a respectful way then please leave. If you are here and you
find something that you are in violent disagreement with, make your
counter argument in a respectful and meaningful way.

Leave room for people to build on what you are saying. Limit the
number of posts that you make during contentious discussions to
enable other voices to participate. And if you are on the sidelines
Speak up if you have not spoken before. It's ok. There are some of
us out there you will back you up.

-Greg

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42315

29 May 2009 - 1:31pm
Robert Reimann
2003

I suppose the question, then, is how important that (obvious but misleading)
clarification is to the overall discussion. A model of a person is not a
person. A model of a group is not a group. A map is not the territory. Point
taken, but I can still use a (good) map to help get me to Albuquerque.

Personas suffer (in some circles) from the erroneous perception that they
are based on substantially (or entirely) fictitious data.
This is a straw man argument, and one that you are indirectly referencing
and lending credence to.

Personas help in the interpretation of behavioral data because they help
weed out idiosyncracies and contradictions; they are a qualitative means of
aggregating and categorizing (i.e., modeling) this data, as well as being a
narrative and empathic tool. Are personas a perfect tool, or universally
applicable? No. But they certainly (IMHO) add value when used appropriately.
In the end, the effectiveness of models built in good faith can only be
judged by outcomes of use, which is what makes the research we are
discussing so pertinent.

Robert.

On Thu, May 28, 2009 at 4:31 PM, Robert Hoekman Jr <robert at rhjr.net> wrote:

> The straw man argument that "personas aren't real" isn't a fair
>> characterization when proper data gathering and analysis techniques are
>> appropriately applied.
>
>
> Well, they don't eat, they don't sleep, and they don't pay taxes, and you
> have to do at least two of those things to be a real, living person.
> Personas may be thoroughly based on real people, but calling them "real"
> is like calling a movie that is thoroughly based on a real series of
> events a "documentary".
>
> Look, I'm sure personas are all kinds of useful for people who believe
> they're all kinds of useful. I'm not here to debate that. I was just
> pointing out that when it comes to actual fact, personas aren't real. My
> response was meant as a clarification, not an argument.
>
> (I've got to stop taking the bait on threads like this.)
>
> -r-
>

29 May 2009 - 1:43pm
Robert Reimann
2003

An interesting topic that arises from this is: what are personas better at--
creating accurate behavioral segmentation/models, or inspiring a more
user-centered dynamic within design groups/organizations? Or both?

One of the things that make personas interesting (to me anyway) is their
apparent ability to provide value as both narrative/empathic tools and data
modeling tools.

Robert.

On Fri, May 29, 2009 at 6:42 AM, Jared Spool <jspool at uie.com> wrote:

>
>
> You have to take this type of research for what it is and not expect a
> single study to prove everything. I found it fascinating because it
> duplicated what we've seen in our research in a controlled setting: teams
> that use personas have a different dynamic than teams that don't -- a
> dynamic that, in my opinion, leads to better design.
>
>

29 May 2009 - 2:06pm
James Page
2008

Jared,

I did not mean that all your ancestors had children is a negative. Just that
this is an example that does not have the inductive issue.

It must be start of summer :-)

James
http://blog.feralabs.com

2009/5/29 James Page <jamespage at gmail.com>

> @Jared,
> Don't get me wrong this is great first step at trying to establish if
> Personas work, or not.
>
> Give me an example of *any* research that, in your mind, solves the
>> "inductive issue"
>
> and we can talk about it further.
>
>
> We can prove the negative "Some Turkeys don't get fed every day", or that
> Newton got some
> of his assumptions wrong, or "Not all swans are white" . And we can say
> that all your ancestors had children.
>
> The null hypothesis is, if personas don't make a differences, then the
>> control group
>
> (the folks w/o personas) will not produce distinguishably different results
>> from those that do.
>>
> As you show here by using a null hypothesis. So Frank Long has shown that
> "Persona
> are better than X" The issue is exactly what is X?
>
> [OT I believe one issue people have with Personas is that there is no '
> null hypothesis' ]
>
> Scientific research studies like this are little building blocks. You
>> disassemble the problem into
>
> little problems, evaluate each problem, then reassemble them to build your
>> case.
>
>
> That works if the studies are replicable. If we don't know how to set up X,
> we can not replicate it.
>
> Sure, there are evaluator effects in heuristic evaluations. But that has to
>> do with
>
> *different* evaluators inspecting the *same* design.
>
>
> So if we agree that it becomes very important to build upon Frank's
> research, then "evaluator effect"
> becomes important as the study needs to be replicable. So if we re-ran the
> study would we get the
> same results. Then there is the other issues which Joshua Porter points out
> such as blinding.
>
> But I would agree some research is better than no research and puts a line
> in the sand for further studies.
>
> @frank
> One interesting point from your research is that the effect of the pictures
> have on the design teams.
> The teams with a photo seam to do far better, scoring nearly twice as much
> as the teams with
> an illustration. Was it the persona's that had the impact or the pictures
> of users. I wonder
> if just having random pictures of people handed out to the designers would
> have the same effect?
> Against a sample of photos of real users. Is it the pictures that give the
> designer the empathic tool
> or is it the narrative. What would the effect be of giving
> the designers access to the facebook/myspace pages,
> against using personas.
>
> The control group would just be given a picture of one user - Steve Jobs.
> :-)
>
> James
> http://blog.feralabs.com
>
>
> 2009/5/29 Joshua Porter <porter at bokardo.com>
>
>
>> On May 29, 2009, at 9:28 AM, Jared Spool wrote:
>>
>>
>>> On May 29, 2009, at 6:36 AM, James Page wrote:
>>>
>>> I think the issue with using heuristic evaluations is the well known
>>>> issue
>>>> with the evaluator effect.
>>>>
>>>
>>> Wow, James. You are *so* missing the point here.
>>>
>>> Sure, there are evaluator effects in heuristic evaluations. But that has
>>> to do with *different* evaluators inspecting the *same* design.
>>>
>>
>>
>> If I understand James correctly, he is suggesting that if the evaluators
>> knew which designers used personas then they would be biased from judging
>> objectively which designs were more user-centered. That is, the so-called
>> differences uncovered by the heuristic evaluation might only be biases
>> introduced by the evaluators.
>>
>> If this is the case (and it's not clear from the writeup exactly what was
>> done), then the results of the study are biased.
>>
>> Frank, could you shed some more light on this? Did the evaluators know
>> which designs were done using personas and which were not?
>>
>>
>> Cheers,
>>
>> Josh
>>
>>
>>
>>
>> Joshua Porter, Founder
>> Bokardo Design
>> Interface design & strategy for social web applications
>> phone: 508-954-1896
>> http://bokardo.com
>> porter at bokardo.com
>> twitter: bokardo
>>
>>
>>
>>
>> ________________________________________________________________
>> Welcome to the Interaction Design Association (IxDA)!
>> To post to this list ....... discuss at ixda.org
>> Unsubscribe ................ http://www.ixda.org/unsubscribe
>> List Guidelines ............ http://www.ixda.org/guidelines
>> List Help .................. http://www.ixda.org/help
>>
>
>

29 May 2009 - 2:03pm
James Page
2008

@Jared,
Don't get me wrong this is great first step at trying to establish if
Personas work, or not.

Give me an example of *any* research that, in your mind, solves the
> "inductive issue"

and we can talk about it further.

We can prove the negative "Some Turkeys don't get fed every day", or that
Newton got some
of his assumptions wrong, or "Not all swans are white" . And we can say that
all your ancestors had children.

The null hypothesis is, if personas don't make a differences, then the
> control group

(the folks w/o personas) will not produce distinguishably different results
> from those that do.
>
As you show here by using a null hypothesis. So Frank Long has shown that
"Persona
are better than X" The issue is exactly what is X?

[OT I believe one issue people have with Personas is that there is no 'null
hypothesis' ]

Scientific research studies like this are little building blocks. You
> disassemble the problem into

little problems, evaluate each problem, then reassemble them to build your
> case.

That works if the studies are replicable. If we don't know how to set up X,
we can not replicate it.

Sure, there are evaluator effects in heuristic evaluations. But that has to
> do with

*different* evaluators inspecting the *same* design.

So if we agree that it becomes very important to build upon Frank's
research, then "evaluator effect"
becomes important as the study needs to be replicable. So if we re-ran the
study would we get the
same results. Then there is the other issues which Joshua Porter points out
such as blinding.

But I would agree some research is better than no research and puts a line
in the sand for further studies.

@frank
One interesting point from your research is that the effect of the pictures
have on the design teams.
The teams with a photo seam to do far better, scoring nearly twice as much
as the teams with
an illustration. Was it the persona's that had the impact or the pictures of
users. I wonder
if just having random pictures of people handed out to the designers would
have the same effect?
Against a sample of photos of real users. Is it the pictures that give the
designer the empathic tool
or is it the narrative. What would the effect be of giving
the designers access to the facebook/myspace pages,
against using personas.

The control group would just be given a picture of one user - Steve Jobs.
:-)

James
http://blog.feralabs.com

2009/5/29 Joshua Porter <porter at bokardo.com>

>
> On May 29, 2009, at 9:28 AM, Jared Spool wrote:
>
>
>> On May 29, 2009, at 6:36 AM, James Page wrote:
>>
>> I think the issue with using heuristic evaluations is the well known
>>> issue
>>> with the evaluator effect.
>>>
>>
>> Wow, James. You are *so* missing the point here.
>>
>> Sure, there are evaluator effects in heuristic evaluations. But that has
>> to do with *different* evaluators inspecting the *same* design.
>>
>
>
> If I understand James correctly, he is suggesting that if the evaluators
> knew which designers used personas then they would be biased from judging
> objectively which designs were more user-centered. That is, the so-called
> differences uncovered by the heuristic evaluation might only be biases
> introduced by the evaluators.
>
> If this is the case (and it's not clear from the writeup exactly what was
> done), then the results of the study are biased.
>
> Frank, could you shed some more light on this? Did the evaluators know
> which designs were done using personas and which were not?
>
>
> Cheers,
>
> Josh
>
>
>
>
> Joshua Porter, Founder
> Bokardo Design
> Interface design & strategy for social web applications
> phone: 508-954-1896
> http://bokardo.com
> porter at bokardo.com
> twitter: bokardo
>
>
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

29 May 2009 - 2:25pm
Andrei Herasimchuk
2004

On May 29, 2009, at 11:31 AM, Robert Reimann wrote:

> Personas suffer (in some circles) from the erroneous perception that
> they
> are based on substantially (or entirely) fictitious data.
> This is a straw man argument, and one that you are indirectly
> referencing
> and lending credence to.

I disagree that there's an erroneous perception out there. Personas
suffer from two things in my experience:

1) There are too many people who *do* make up information -- in part
or in whole -- when they write them, then use that document as a
justification for opinion based or highly subjective decisions, which
basically does double damage to the design process and the end
product. I've seen it firsthand too many times to count, and I still
see it happening all of the time.

2) There are too many people use personas as a replacement for being
involved in the research itself. In reading Henry Dreyfuss and
listening to some of the core proponents of people who clarify what
personas are all about on this list, it is clear that the act of being
involved in the research is the point, not the documenting of said
research.

In my personal design processes, I always tend to do what Dreyfuss
writes about in Designing for People. I go into the field and watch
people do things, ask lots of questions, listen a lot, take notes, and
I often do what they do myself to get a sense of the problem without
any filters. I've done this my entire career so far and everywhere
I've been as much as I can. My poor reactions to personas in the past
are because of the two things I mention above, but had I met someone
back in 1992 who actually explained to me that personas are about
hard, functional data and research, and that the collection and
observation of that research is really what personas were all about,
that the document is basically shorthand for remembering the research
activity, I'd have never resisted personas in the first place.

It is because of items #1 and #2 that I still to this day refuse to
engage in personas as a design process voluntarily. I'm fine with my
own data collection and note taking methods, thank you very much.

Can personas be saved? Sure. But there's a lot of damage that has to
repaired and people in this field need to take more responsibility for
the processes they unleash into the hands of people who aren't trained
or simply don't know what they are doing.

--
Andrei Herasimchuk

Chief Design Officer, Involution Studios
innovating the digital world

e. andrei at involutionstudios.com
c. +1 408 306 6422

29 May 2009 - 3:06pm
Robert Reimann
2003

On Fri, May 29, 2009 at 12:25 PM, Andrei Herasimchuk <
aherasimchuk at involutionstudios.com> wrote:

>
> On May 29, 2009, at 11:31 AM, Robert Reimann wrote:
>
> Personas suffer (in some circles) from the erroneous perception that they
>> are based on substantially (or entirely) fictitious data.
>> This is a straw man argument, and one that you are indirectly referencing
>> and lending credence to.
>>
>
> I disagree that there's an erroneous perception out there. Personas suffer
> from two things in my experience:
>
> 1) There are too many people who *do* make up information -- in part or in
> whole -- when they write them, then use that document as a justification for
> opinion based or highly subjective decisions, which basically does double
> damage to the design process and the end product. I've seen it firsthand too
> many times to count, and I still see it happening all of the time.

I don't disagree with this observation. I just wouldn't call those made-up
profiles personas. So there is also an erroneous set of practices
surrounding persona creation that requires education. Hopefully Kim
Goodwin's new book will help remedy some of the damage caused by sloppy
practice.

> 2) There are too many people use personas as a replacement for being
> involved in the research itself. In reading Henry Dreyfuss and listening to
> some of the core proponents of people who clarify what personas are all
> about on this list, it is clear that the act of being involved in the
> research is the point, not the documenting of said research.
>
>
I also agree with the issue raised, if not completely with the conclusion
drawn. Personas are a way of pre-processing the design problem by
experiencing and analyzing the behaviors and mental models of potential
users. Participation in the data gathering is critical to this, as you
suggest. I wouldn't however go so far as to say that the models generated
from the research are not the point; they are powerful mnemonic and
communication tools. If you are not working with a team, or your clients
never question your design decisions, then perhaps the communication aspect
of personas is superfluous, but otherwise I wouldn't underestimate it for
the average practioner.

Robert.

29 May 2009 - 4:13pm
Jared M. Spool
2003

On May 29, 2009, at 3:25 PM, Andrei Herasimchuk wrote:

> Personas suffer from two things in my experience:
>
> 1) There are too many people who *do* make up information -- in part
> or in whole -- when they write them, then use that document as a
> justification for opinion based or highly subjective decisions,
> which basically does double damage to the design process and the end
> product. I've seen it firsthand too many times to count, and I still
> see it happening all of the time.
>
> 2) There are too many people use personas as a replacement for being
> involved in the research itself. In reading Henry Dreyfuss and
> listening to some of the core proponents of people who clarify what
> personas are all about on this list, it is clear that the act of
> being involved in the research is the point, not the documenting of
> said research.

Agreed.

However...

What you're talking about are poorly executed personas. Sure, too many
persona projects produce sucky results, but if we look beyond that at
the benefits produced by the few well-executed projects, can we agree
that, when done well, this tool has merit?

> It is because of items #1 and #2 that I still to this day refuse to
> engage in personas as a design process voluntarily. I'm fine with my
> own data collection and note taking methods, thank you very much.

But, Andrei, isn't it the case that there's a lot of poorly-executed
design projects out there? Should *I* refuse to engage in design
because too many design projects end up crappy?

Can't you really make that argument about anything? (Once again, we
see Sturgeon's Law raising it's ugly head.)

I'm just thinking that we need to start focusing among the few really
well done things in our practice and stop throwing out all the babies
with our bath water.

Jared

29 May 2009 - 5:18pm
Andrei Herasimchuk
2004

On May 29, 2009, at 2:13 PM, Jared Spool wrote:

> What you're talking about are poorly executed personas. Sure, too
> many persona projects produce sucky results, but if we look beyond
> that at the benefits produced by the few well-executed projects, can
> we agree that, when done well, this tool has merit?

Is that a trick question? Because I would presume the answer to
question is the same regardless of topic. IOW, anything well-executed
has merit. So are you trying to trick me here? I'm about to open a keg
of beer for a Friday bash, so I'm not susceptible... yet.

>> It is because of items #1 and #2 that I still to this day refuse to
>> engage in personas as a design process voluntarily. I'm fine with
>> my own data collection and note taking methods, thank you very much.
>
> But, Andrei, isn't it the case that there's a lot of poorly-executed
> design projects out there? Should *I* refuse to engage in design
> because too many design projects end up crappy?

I don't think of it as mixing the results of something with the
process of making it or the tools used to make it. Personas are a tool
and generate a process around their existence. The results have
nothing to do with the larger issue. As a tool, personas in their
current state aren't very well designed nor very well understood. As
for the process, personas don't add a lot of value due to the nature
of the tool not being well defined nor well understood. (Obviously, in
my experience.) We're finishing a project right now where the client's
product team kept bring up a persona they created and used the
"character" to pretty much justify whatever they felt like. It got to
the point that the persona was basically every user, and its entire
value became meaningless from my perspective. I see that happen over
and over and have now for 10+ years.

My experience is that 9 out of 10 times, people make personas
improperly, and worse, use them to make or justify whatever design
decision suits their fancy that day. But nearly every single person
I've seen use them *thinks* they are doing it right.

At least with software, often times the product prompts you with an
annoying alert that usually says something like, "You're doing it
wrong. Please RTFM."

> Can't you really make that argument about anything? (Once again, we
> see Sturgeon's Law raising it's ugly head.)

Ah... it was a trick question.

I probably wasn't being clear enough in my disagreement with Robert's
post.

I am of the opinion it is incumbent on the people who create a tool or
process to follow through on how it is used and executed in the field,
especially when the tool or process is being offered as a standard
practice. I do not believe it is incumbent on the people in the field
to do all the legwork themselves to understand what the tool is
supposed to be. They have to be met halfway. In that regard, my
disagreement with Robert is that I felt his comment:

"Personas suffer (in some circles) from the erroneous perception that
they are based on substantially (or entirely) fictitious data."

There's no erroneous perception going on here at all. People actually
*DO* base personas on substantially or entirely fictitious data, and
even when they are based on real data, they're sometimes injected into
the design process in entirely inappropriate ways. So while I get that
Robert claims that those poorly written character studies aren't
"personas" because they basically works of fiction, that doesn't
change the fact that people are doing it and *calling* them personas.

> I'm just thinking that we need to start focusing among the few
> really well done things in our practice and stop throwing out all
> the babies with our bath water.

In the current form and as practiced, personas are massively broken. I
have no dog in the hunt since I do my own thing which works for me,
except when I have to constantly deal with people who use them in ways
that breaks the design process. However, if the people who care about
such things finally take it upon themselves to fix personas (as both a
tool and a process) and all that entails in this profession, I
guarantee I'll be the first in line to make sure it's implemented with
all the clients and design teams I work with.

--
Andrei Herasimchuk

Chief Design Officer, Involution Studios
innovating the digital world

e. andrei at involutionstudios.com
c. +1 408 306 6422

29 May 2009 - 5:36pm
usabilitycounts
2008

> My experience is that 9 out of 10 times, people make personas
> improperly, and worse, use them to make or justify whatever design
> decision suits their fancy that day. But nearly every single person
> I've seen use them *thinks* they are doing it right.

Their using the hammer wrong. Don't blame the hammer. ;)

About Personas (and everything else):

I think of the mistakes we make generally in the UX community is that
we think we're inventing everything for the first time. We aren't, and
we shouldn't.

There are a lot of tools (like Market Research, used by advertising
agencies) that have similar roles in software development. http://www.allbusiness.com/marketing/market-research/1287-1.html
, for example. Nothing against Alan Cooper, but a lot of the
information in a typical persona is the same as customer profiling. I
saw a lot of this during direct mail political campaign work I did
before the web, in crafting the message, and that was well before the
web.

You know, if we had standardized processes for a lot of this, it
wouldn't really be an issue. You don't know you are doing personas
wrong unless there are guidelines for doing them. The same goes for
most interaction design. Unless this happens, will get into these Mac
vs. PCs, Personas vs. No Personas arguments until the end of time.
Patrick

twitter: usabilitycounts | uxlosangeles | cooltechjobs

email: pat at usabilitycounts.com | blog: http://www.usabilitycounts.com

cell: (562) 508-1750 | office: (562) 612-3346

The last UX books you'll ever need.

29 May 2009 - 6:24pm
Andrei Herasimchuk
2004

On May 29, 2009, at 3:36 PM, Patrick wrote:

> I think of the mistakes we make generally in the UX community is
> that we think we're inventing everything for the first time. We
> aren't, and we shouldn't.

This I absolutely agree with. It drives me insane actually.

--
Andrei Herasimchuk

Chief Design Officer, Involution Studios
innovating the digital world

e. andrei at involutionstudios.com
c. +1 408 306 6422

29 May 2009 - 10:06pm
Robert Hoekman, Jr.
2005

Thanks for your thorough response. Responses below ...

But seriously, the research did not use heuristics to evaluate the
> effectiveness of personas. I used heuristics to evaluate the
> usability of the resulting designs %u2013 allowing me to compare all
> of the diverse design solutions consistently with each other.

But that's exactly my point. You used a heuristic evaluation, which is a
measurement for usability. Personas have nothing to do with designing
usable applications.

The brief stated that
> the user-friendliness of the product was of paramount importance
> %u2013 especially the set-up task as this was seen as a barrier to
> product sales.

How is this something that can be helped through the use of personas? This
is a usability issue, plain and simple.

> The market research outlined the user profile and the
> personas were created from this same user profile.
> The image boards were given to the control group covered 3 areas,
> Product environment, lifestyle and brand landscape.

None of these things have an affect on designing a usable setup process,
which is what you measured with your heuristic evaluation. Image boards
can't tell you anything about how people might perform the tasks supported
in the application.

I really don't get this. You're evaluating apples by looking at oranges. If
I'm just missing your point, please elaborate.

The conclusions of the study find that using personas is an effective
> tool and did produce more user-focused solutions.

Personas are a strategic tool. Heuristics are a usability measurement tool.
Even the most masterful persona-driven strategy can end with a design with a
low level of usability. And clearly, applications can have an incredibly
high degree of usability when no personas are used whatsoever. Please tell
me you see that.

This study's results show that the two groups that used personas also
happened to create more usable applications. The results do not explicitly
show that personas caused increase usability, or even that it's possible for
personas to do so.

And to address Jared's argument — yes, I understand that this is but a small
sliver of a complete argument that proves the value of persona usage, but
the way it the paper is written, it seems to attempt to claim proof of quite
a bit.

It also acknowledges
> that personas are only one of many design tools %u2013 and does not
> claim that it is the most effective method. As the saying
> goes%u2026there is more than one way to skin a cat, or to make a duck
> happy.
>

But in the study, no research materials were provided to the control group
that could reasonably be considered comparable to personas. Personas are the
result of researching the people who will use an application. The image
boards, as you said yourself, were the result of looking at the "product
environment, lifestyle and brand landscape". Those things aren't people. The
control group's materials don't compare.

I really don't mean to knock your work here — seriously, kudos for the
effort to prove out something so hotly disputed — but there are just too
many issues here to draw any valuable conclusions.

-r-

29 May 2009 - 10:16pm
Robert Hoekman, Jr.
2005

>
> Trust me. Compared to many of the hundreds of research papers I review each
> year, this one is pretty solid.
>

Oh, I completely believe that. But — and I've probably said this a million
times — "better by comparison" does not equal "good".

While the bigger problem is, "Do teams that employ personas produce
> effective designs for their audiences?" that, as you've correctly pointed
> out, is hard to prove in a study.
>

But unless I'm mistaken, that's exactly what this paper claims to have
shown.

Of course, this study can't stand alone. Good research, like this, needs to
> be duplicated elsewhere before you can really stand behind it. Other
> researchers should try to replicate the experiment to see if they get
> similar results. Then, and only then, will we empirically know that the
> results were great. Frank did a good job of explaining to another team how
> to duplicate the experiment.
>

I'm so glad you said that.

I kinda wish I hadn't taken the bait on this thread. It's been in the back
of my head all day today.

Studies are for women!

-r-

29 May 2009 - 10:25pm
Robert Hoekman, Jr.
2005

>
> I suppose the question, then, is how important that (obvious but
> misleading)
> clarification is to the overall discussion. A model of a person is not a
> person. A model of a group is not a group. A map is not the territory.
> Point
> taken, but I can still use a (good) map to help get me to Albuquerque.
>

Oh, definitely. I think the main issue is that designers dispute that
they're fictitious, when they are, in fact, fictitious. Why bother arguing
that red is blue?

Instead of arguing that they're real, designers should try embracing the
idea that personas are the "maps" in your analogy. "Maps can help you get to
Albuquerque without actually being road signs. Personas can help you design
good stuff without actually being living, breathing people. They are based
on real people and are designed to reflect and represent real people, much
in the same way that maps reflect the route to Albuquerque. As such, they
(personas and maps) are a great way to make sure we all stay on the same
path."

-r-

30 May 2009 - 5:24am
Steve Baty
2009

I don't like the characterisation of personas as fictitious, since that
conveys a perjorative sense of "being made up". Personas, like all audience
segmentation and user modelling techniques, are an articulation of a
collection of shared attributes. The lines between segments can be fuzzy,
and the narrative provided to capture the flavour or essence of the segment
can feel like story-telling, but the attributes (or dimensions) and the
values of those attributes are very real.

I don't believe that this is the same as mistaking the map for the
landscape; nor is it mistaking a map for a sign-post. If I describe the
Australian desert landscape using stories and memories from a hundred
different locations, does it matter that no single place shares all of those
qualities, or is it more important that you'd now be able to design a
product much better suited to that landscape?

Steve

2009/5/30 Robert Hoekman Jr <robert at rhjr.net>

> Oh, definitely. I think the main issue is that designers dispute that
> they're fictitious, when they are, in fact, fictitious. Why bother arguing
> that red is blue?
>
> Instead of arguing that they're real, designers should try embracing the
> idea that personas are the "maps" in your analogy. "Maps can help you get
> to
> Albuquerque without actually being road signs. Personas can help you design
> good stuff without actually being living, breathing people. They are based
> on real people and are designed to reflect and represent real people, much
> in the same way that maps reflect the route to Albuquerque. As such, they
> (personas and maps) are a great way to make sure we all stay on the same
> path."
>

Syndicate content Get the feed