Do these companies do user research or utesting?

16 Nov 2006 - 6:04pm
7 years ago
12 replies
833 reads
Donna Maurer
2003

I'm wondering if any of you know whether these web 2.0 poster children
do user research or usability testing (or, if they do it now, whether
they did before they became famous or bought):

- Flickr
- Del.icio.us
- Threadless
- Youtube
- Digg
- Upcoming
- Lastfm
- Etsy

I know I could approach each of them, but I'm guessing some of you know
the answers...

Why do I want to know? I'm writing a keynote at the moment and exploring
the idea that plenty of terrific stuff gets done without either of these...

Ta

Donna

--
Donna Maurer
Maadmob Interaction Design
e: donna at maadmob.net
web: http://maadmob.net/maadmob_id/
book: http://rosenfeldmedia.com/books/cardsorting/

Comments

16 Nov 2006 - 6:23pm
Hayden Vink
2004

I don't know about any of those companies, but I agree with Chris Fahey that
there's a lot to be said for "user research smoke and mirrors":
http://www.graphpaper.com/2006/07-10_user-research-smoke-mirrors-part-1-design-vs-science

Hayden

On 11/17/06, Donna Maurer <donnam at maadmob.net> wrote:
>
> I'm wondering if any of you know whether these web 2.0 poster children
> do user research or usability testing (or, if they do it now, whether
> they did before they became famous or bought):
>
> - Flickr
> - Del.icio.us
> - Threadless
> - Youtube
> - Digg
> - Upcoming
> - Lastfm
> - Etsy
>
> I know I could approach each of them, but I'm guessing some of you know
> the answers...
>
> Why do I want to know? I'm writing a keynote at the moment and exploring
> the idea that plenty of terrific stuff gets done without either of
> these...
>
> Ta
>
> Donna
>
>
> --
> Donna Maurer
> Maadmob Interaction Design
> e: donna at maadmob.net
> web: http://maadmob.net/maadmob_id/
> book: http://rosenfeldmedia.com/books/cardsorting/
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> List Guidelines ............ http://listguide.ixda.org/
> List Help .................. http://listhelp.ixda.org/
> (Un)Subscription Options ... http://subscription-options.ixda.org/
> Announcements List ......... http://subscribe-announce.ixda.org/
> Questions .................. lists at ixda.org
> Home ....................... http://ixda.org/
> Resource Library ........... http://resources.ixda.org
>

--

Hayden Vink

HANSEL » Research, Strategy & Interaction Design

16 Nov 2006 - 6:20pm
Steve Baty
2009

Donna,

Out of interest, is your argument centred on the quality of the concept or
the execution?

Steve

On 17/11/06, Donna Maurer <donnam at maadmob.net> wrote:
>
>
> Why do I want to know? I'm writing a keynote at the moment and exploring
> the idea that plenty of terrific stuff gets done without either of
> these...
>
>
--
----------------------------------------------
Steve 'Doc' Baty B.Sc (Maths), M.EC, MBA
Director, User Experience Strategy
Red Square
P: +612 8289 4930
M: +61 417 061 292

Member, UPA - www.upassoc.org
Member, IxDA - www.ixda.org
Member, Web Standards Group - www.webstandardsgroup.org

16 Nov 2006 - 6:32pm
Robert Hoekman, Jr.
2005

Kudos on the topic. I'd back you up. In fact, I'll be part of a SxSW panel
called "Why we should ignore users" that will approach this subject from a
slightly different angle.

Anyway, Yahoo owns Delicious now, and they have quite a large UED team, so
I'd guess they do some homework on that one. Others I don't know about.

Where will your keynote be?

-r-

On 11/16/06, Donna Maurer <donnam at maadmob.net> wrote:
>
> I'm wondering if any of you know whether these web 2.0 poster children
> do user research or usability testing (or, if they do it now, whether
> they did before they became famous or bought):
>
> - Flickr
> - Del.icio.us
> - Threadless
> - Youtube
> - Digg
> - Upcoming
> - Lastfm
> - Etsy
>
> I know I could approach each of them, but I'm guessing some of you know
> the answers...
>
> Why do I want to know? I'm writing a keynote at the moment and exploring
> the idea that plenty of terrific stuff gets done without either of
> these...
>
> Ta
>
> Donna
>
>
> --
> Donna Maurer
> Maadmob Interaction Design
> e: donna at maadmob.net
> web: http://maadmob.net/maadmob_id/
> book: http://rosenfeldmedia.com/books/cardsorting/
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> List Guidelines ............ http://listguide.ixda.org/
> List Help .................. http://listhelp.ixda.org/
> (Un)Subscription Options ... http://subscription-options.ixda.org/
> Announcements List ......... http://subscribe-announce.ixda.org/
> Questions .................. lists at ixda.org
> Home ....................... http://ixda.org/
> Resource Library ........... http://resources.ixda.org
>

16 Nov 2006 - 6:41pm
Christian Crumlish
2006

Formal testing? I wonder. I gather that at least Flickr, especially
during its seemingly neverending beta period, was updated constantly
based on user behaviors and feature requests. Could that be considered
"real-world testing"?

Christian Crumlish
Director of Strategic Services, Extractable
Director, IT/Web Projects, IA Institute

> -----Original Message-----
> From: discuss-bounces at lists.interactiondesigners.com
> [mailto:discuss-bounces at lists.interactiondesigners.com] On
> Behalf Of Hayden Vink
> Sent: Thursday, November 16, 2006 3:24 PM
> To: Donna Maurer
> Cc: discuss at ixda.org
> Subject: Re: [IxDA Discuss] Do these companies do user
> research or utesting?
>
> I don't know about any of those companies, but I agree with
> Chris Fahey that there's a lot to be said for "user research
> smoke and mirrors":
> http://www.graphpaper.com/2006/07-10_user-research-smoke-mirro
rs-part-1-design-vs-science
>
> Hayden
>
> On 11/17/06, Donna Maurer <donnam at maadmob.net> wrote:
> >
> > I'm wondering if any of you know whether these web 2.0
> poster children
> > do user research or usability testing (or, if they do it
> now, whether
> > they did before they became famous or bought):
> >
> > - Flickr
> > - Del.icio.us
> > - Threadless
> > - Youtube
> > - Digg
> > - Upcoming
> > - Lastfm
> > - Etsy
> >
> > I know I could approach each of them, but I'm guessing some of you
> > know the answers...
> >
> > Why do I want to know? I'm writing a keynote at the moment and
> > exploring the idea that plenty of terrific stuff gets done without
> > either of these...
> >
> > Ta
> >
> > Donna
> >
> >
> > --
> > Donna Maurer
> > Maadmob Interaction Design
> > e: donna at maadmob.net
> > web: http://maadmob.net/maadmob_id/
> > book: http://rosenfeldmedia.com/books/cardsorting/
> > ________________________________________________________________
> > Welcome to the Interaction Design Association (IxDA)!
> > To post to this list ....... discuss at ixda.org List Guidelines
> > ............ http://listguide.ixda.org/ List Help
> ..................
> > http://listhelp.ixda.org/ (Un)Subscription Options ...
> > http://subscription-options.ixda.org/
> > Announcements List ......... http://subscribe-announce.ixda.org/
> > Questions .................. lists at ixda.org Home
> > ....................... http://ixda.org/ Resource Library
> ...........
> > http://resources.ixda.org
> >
>
>
>
> --
>
> Hayden Vink
>
> HANSEL > Research, Strategy & Interaction Design
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org List Guidelines
> ............ http://listguide.ixda.org/ List Help
> .................. http://listhelp.ixda.org/ (Un)Subscription
> Options ... http://subscription-options.ixda.org/
> Announcements List ......... http://subscribe-announce.ixda.org/
> Questions .................. lists at ixda.org Home
> ....................... http://ixda.org/ Resource Library
> ........... http://resources.ixda.org
>

16 Nov 2006 - 7:20pm
Jared M. Spool
2003

At 06:41 PM 11/16/2006, Christian Crumlish wrote:
>Formal testing? I wonder. I gather that at least Flickr, especially
>during its seemingly neverending beta period, was updated constantly
>based on user behaviors and feature requests. Could that be considered
>"real-world testing"?

How is that different from any other product or service that has more than
one release?

Jared

Jared M. Spool, Founding Principal, User Interface Engineering
510 Turnpike Street, Suite 102, North Andover, MA 01845
978 327-5561 jspool at uie.com http://www.uie.com
Blog: http://www.uie.com/brainsparks

16 Nov 2006 - 7:34pm
Steve Baty
2009

Jared,

I think the distinction comes down to whether the company takes any notice
of the feedback (either implicit or explicit) it receives from its users.
Steve

On 17/11/06, Jared M. Spool <jspool at uie.com> wrote:

>
> How is that different from any other product or service that has more than
> one release?
>
> Jared
>
>
----------------------------------------------
Steve 'Doc' Baty B.Sc (Maths), M.EC, MBA
Director, User Experience Strategy
Red Square
P: +612 8289 4930
M: +61 417 061 292

Member, UPA - www.upassoc.org
Member, IxDA - www.ixda.org
Member, Web Standards Group - www.webstandardsgroup.org

16 Nov 2006 - 9:36pm
Christian Crumlish
2006

Not sure it is different, unless update are driven, say, entirely by an internal business requirements roadmap and not reality-checked against actual user behavior.

-----Original Message-----
From: Jared M. Spool [mailto:jspool at uie.com]
Sent: Thu 11/16/2006 4:20 PM
To: Christian Crumlish
Cc: discuss at ixda.org
Subject: Re: [IxDA Discuss] Do these companies do user research or utesting?

At 06:41 PM 11/16/2006, Christian Crumlish wrote:
>Formal testing? I wonder. I gather that at least Flickr, especially
>during its seemingly neverending beta period, was updated constantly
>based on user behaviors and feature requests. Could that be considered
>"real-world testing"?

How is that different from any other product or service that has more than
one release?

Jared

Jared M. Spool, Founding Principal, User Interface Engineering
510 Turnpike Street, Suite 102, North Andover, MA 01845
978 327-5561 jspool at uie.com http://www.uie.com
Blog: http://www.uie.com/brainsparks

16 Nov 2006 - 9:47pm
Josh
2006

I agree that it's very likely Flickr is are doing usability testing because
of the Y! influence, and I'm assuming you're referring to "bring in 8 people
one at a time, give them some tasks, and ask them to think outloud" standard
qualitative usability testing.

I'd definitely be interested in any discussion re: building without
usability testing. I'm not against qualitative usability testing by any
means, because the quotes from users provide can provide a lot of leverage
when you're stuck with a team that can't make up it's mind. I've just gotten
more and more convinced that sites interested in increasing conversion
should be looking at building internal services similar to those provided by
companies like Optimost that randomly rotate and test content against live
users. The idea being that the live users will tell you what converts better
than any designer or usability pro.

Listening and paying attention to users is key to running an online
business, but why not build sites that change and optimize themselves? Back
to the topic at hand, isn't that what many of the "2.0" sites are doing with
reliance on user provided content and tags?

- Josh Viney

17 Nov 2006 - 12:44am
Ashim
2006

I did some net research into the poster companies of web 2.0 some time back
and if they did any intital user studies or testing. Some facts were quite
revealing. Like flickr didnt do any initial usability studies or testing.

http://notelab.infogami.com/startupschool2006

Speaker: Caterina Fake (Flickr)

Q: "What did you do for usability testing?"

* Almost none, got it out early as Paul also suggested (Paul Graham - he
also spoke at the conference)
* Flickr Alpha was the usability testing- "put it out with its zits and
blemishes"
* Heavy monitoring of/posting on their support forums
- In response to a change in the product: "I had a really
difficult childhood, and this new change you've just made has pushed me over
the edge!"
* "Users don't have a problem teling you that something really sucks..."

Q: "How did you first start getting some good press? PR firm? How did you
start getting users?"

* Necessity (Being underfunded) is the mother of invention- blogging /
viral
* 80% of initial users came due to blogs
* Alpha-geek reporters started writing articles without talking to them
* Eventually hired a PR firm- not to generate publicity, but to handle
it
* The UI evolved organically.
* Joining a community is like arriving at a party without knowing
anybody- someone should take your coat, serve you a drunk, and rub your feet
( y-combinator take notice :P )

I dont know if this non user testing approach can apply to each new startup
service. Flickr spent almost two years evolving from a game to
photo-sharing. And I think the "release early, release often" mantra loses a
lot of early userbase, who get frustrated by the non-user-friendliness of
the service.

Even del.icio.us didnt do any inital usability study or tests at first. They
hired creativegood <http://www.creativegood.com/> to redesign their
website<http://www.creativegood.com/casestudies/delicious.html>only
after the site became popular.

Do you think this is this a trend thats here to stay ?

Ashim

17 Nov 2006 - 3:58am
Leisa Reichelt
2006

yes, I can confirm that Flickr do no formal user testing (such that
would be recognised in a formal UCD methodology). I interviewed
George Oates one of two designers at Flickr, and who has been with
them since Ludicorp days, who confirmed this.

You can hear that interview here: http://tinyurl.com/yy4gtl

On 17/11/06, Ashim <ashim.s at gmail.com> wrote:
> I did some net research into the poster companies of web 2.0 some time back
> and if they did any intital user studies or testing. Some facts were quite
> revealing. Like flickr didnt do any initial usability studies or testing.
>
>
> http://notelab.infogami.com/startupschool2006
>
>
> Speaker: Caterina Fake (Flickr)
>
> Q: "What did you do for usability testing?"
>
> * Almost none, got it out early as Paul also suggested (Paul Graham - he
> also spoke at the conference)
> * Flickr Alpha was the usability testing- "put it out with its zits and
> blemishes"
> * Heavy monitoring of/posting on their support forums
> - In response to a change in the product: "I had a really
> difficult childhood, and this new change you've just made has pushed me over
> the edge!"
> * "Users don't have a problem teling you that something really sucks..."
>
>
>
> Q: "How did you first start getting some good press? PR firm? How did you
> start getting users?"
>
> * Necessity (Being underfunded) is the mother of invention- blogging /
> viral
> * 80% of initial users came due to blogs
> * Alpha-geek reporters started writing articles without talking to them
> * Eventually hired a PR firm- not to generate publicity, but to handle
> it
> * The UI evolved organically.
> * Joining a community is like arriving at a party without knowing
> anybody- someone should take your coat, serve you a drunk, and rub your feet
> ( y-combinator take notice :P )
>
>
> I dont know if this non user testing approach can apply to each new startup
> service. Flickr spent almost two years evolving from a game to
> photo-sharing. And I think the "release early, release often" mantra loses a
> lot of early userbase, who get frustrated by the non-user-friendliness of
> the service.
>
> Even del.icio.us didnt do any inital usability study or tests at first. They
> hired creativegood <http://www.creativegood.com/> to redesign their
> website<http://www.creativegood.com/casestudies/delicious.html>only
> after the site became popular.
>
> Do you think this is this a trend thats here to stay ?
>
> Ashim
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> List Guidelines ............ http://listguide.ixda.org/
> List Help .................. http://listhelp.ixda.org/
> (Un)Subscription Options ... http://subscription-options.ixda.org/
> Announcements List ......... http://subscribe-announce.ixda.org/
> Questions .................. lists at ixda.org
> Home ....................... http://ixda.org/
> Resource Library ........... http://resources.ixda.org
>

--
________________________
Leisa Reichelt
User Experience Consultant

leisa.reichelt at gmail.com
www.disambiguity.com

16 Nov 2006 - 9:51pm
Adler
2006

you might want to read the article from Norman "Human-Centered Design
Considered Harmful" (http://www.jnd.org/dn.mss/human-centered.html)

taking the article argument, I guess these companies were successful
because they focused in Activity-Centered Design, were good in doing
it and constantly improved their product. So when they became famous
they had committed all major mistakes and solved any big usability
problem.

Adler

On 11/17/06, Donna Maurer <donnam at maadmob.net> wrote:
> I'm wondering if any of you know whether these web 2.0 poster children
> do user research or usability testing (or, if they do it now, whether
> they did before they became famous or bought):
>
> - Flickr
> - Del.icio.us
> - Threadless
> - Youtube
> - Digg
> - Upcoming
> - Lastfm
> - Etsy
>
> I know I could approach each of them, but I'm guessing some of you know
> the answers...
>
> Why do I want to know? I'm writing a keynote at the moment and exploring
> the idea that plenty of terrific stuff gets done without either of these...

--
http://www.linkedin.com/in/adler

20 Nov 2006 - 12:47am
Christine Boese
2006

Question: what if the shoe is on the wrong foot, in terms of why these
companies didn't use "formal" user-centered research and testing?

On 11/17/06, Donna Maurer <donnam at maadmob.net> wrote:
> I'm wondering if any of you know whether these web 2.0 poster children
> do user research or usability testing (or, if they do it now, whether
> they did before they became famous or bought):
>
> - Flickr
> - Del.icio.us <http://del.icio.us/>
> - Threadless
> - Youtube
> - Digg
> - Upcoming
> - Lastfm
> - Etsy

>From reading this list, I have come to understand that there is often a
basic assumption here that USERS are constructed as autonomous agents. We
have personas, all kinds of models that show them acting alone, acting
autonomously, interacting with interfaces, with communications, processes,
but rarely do those models profile community/communal activity, group
activity, social activity without a single solitary "task" in mind (that
would be hard to do, and exponentially difficult to predict). It would be
social activity for its own sake, perhaps the entire reason why social media
suddenly appeared on the scene while business types were myopically busy
selling widgets.

Like the Field of Dreams, the social media folks built it, and people came,
for the social features, and not the widgets. That has to be damned
frustrating for widget-sellers focused on individuals, or rather, on
"conversions," which basically means having a meaningful relationship with
an individual's wallet.

What did these companies know that did an end-run around that idea?

What if it is that SOCIAL GROUPS and COMMUNITIES should be the center, and
not an artificial construct of an autonomous individual? Could that be where
the greater interface power lies? Greater motivation, greater participation,
greater grassroots-style success?

How can user-centered design bottle a formula for a kind of site that
REQUIRES social group-centered design to be effective?

And would it not necessarily follow that social groups behave in a
significantly different manner than autonomous actors, and require a major
revision of the idea of personas and use case models?

So these companies didn't apply our formula. What if the formula has a
massive blind spot, and they intuitively chose to fill the need they sensed
from the blind spot instead of blindly following a disciplinary formula that
is itself is an evolving social construct?

just a thought.

Chris

On 11/16/06, Pedro-Jorge Adler <pedro.adler at acm.org> wrote:
>
> you might want to read the article from Norman "Human-Centered Design
> Considered Harmful" (http://www.jnd.org/dn.mss/human-centered.html)
>
> taking the article argument, I guess these companies were successful
> because they focused in Activity-Centered Design, were good in doing
> it and constantly improved their product. So when they became famous
> they had committed all major mistakes and solved any big usability
> problem.
>
> Adler
>
> On 11/17/06, Donna Maurer <donnam at maadmob.net> wrote:
> > I'm wondering if any of you know whether these web 2.0 poster children
> > do user research or usability testing (or, if they do it now, whether
> > they did before they became famous or bought):
> >
> > - Flickr
> > - Del.icio.us
> > - Threadless
> > - Youtube
> > - Digg
> > - Upcoming
> > - Lastfm
> > - Etsy
> >
> > I know I could approach each of them, but I'm guessing some of you know
> > the answers...
> >
> > Why do I want to know? I'm writing a keynote at the moment and exploring
> > the idea that plenty of terrific stuff gets done without either of
> these...
>
>
> --
> http://www.linkedin.com/in/adler
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> List Guidelines ............ http://listguide.ixda.org/
> List Help .................. http://listhelp.ixda.org/
> (Un)Subscription Options ... http://subscription-options.ixda.org/
> Announcements List ......... http://subscribe-announce.ixda.org/
> Questions .................. lists at ixda.org
> Home ....................... http://ixda.org/
> Resource Library ........... http://resources.ixda.org
>

--
christine boese
www.serendipit-e.com

Syndicate content Get the feed