Traditional user testing vs micro design evolution (was: The death of web usability testing as we know it?)

2 Jan 2008 - 6:49pm
6 years ago
3 replies
1760 reads
Nicholas Iozzo
2007

A great discussion appears to be happening under the original topic. I wanted to further focus it into something I have been thinking a lot about recently. So I have hijacked it and created this new thread. My apologies to Oleh.

I find all of the applications coming out to do quick real time testing of alternate design solutions a great tool to have in our toolbox. But it is just another tool, not a replacement. I see this as a complement (not a supplement) to traditional user research and testing.

If you where to just use these tools to iterate and improve your design, I am sure you will be able to get the best design possible considering the point you started from. Maybe, however, if you started with a very different design your end iteration would perform even better. So, if the only tool you used was a micro-design evolution approach, then you may end up not with the best design possible, but with the best design considering the point you started from. (In mathematical terms, a local maximum http://en.wikipedia.org/wiki/Local_maximum).

So how do you get the very best design possible?
If you had all the time any money you wanted, you would build the best designs you could dream of that would cover as much of the design space as possible. You would send small, but significant traffic to each and keep tweaking the design until it was the best it could be. Eventually designs would begin to drop-out because other designs would be performing better. Finally, when you are left with a single design performing better then the rest, then you know you have the best design for that design space.

Of course we have real world issue of making bad first impressions with users (not to mention the unlimited money part). So we can not follow a model like this.

Instead, what we can do, is to develop prototypes of the best design possibilities we can dream up. We run small samples of users through these prototypes and we see which performs the best and which appears to have the best potential upside and most design flexibility. It may not be the design all the users said they liked the best. (What a user says and what they actually think are very different things).

My opinion (In summary)
These analytical tools that are coming out to do micro evaluations of design options are great. They will help you climb that hill to get your design to the best it can be! But, they will not help you pick the tallest hill to climb. It is too expensive to try and climb them all so you need a low-cost method to explore and evaluate the potential of each... a method like low-fidelity prototypes with usability test evaluations.

We worked with one our client's, Orbitz, to do a presentation on this subject at Gartner's web innovation summit. http://agendabuilder.gartner.com/WEB1/WebPages/SessionDetail.aspx?EventSessionId=833. Its spin was more focused on the business topics of the issue and less on the design and design process side of things.

Nick Iozzo
Principal User Experience Architect

tandemseven

847.452.7442 mobile

niozzo at tandemseven.com
http://www.tandemseven.com/

Comments

2 Jan 2008 - 7:42pm
Bruce Esrig
2006

I'll take a chance that the following thought hasn't already been expressed
on these threads.

A lot of work has been done in somewhat-numerical realms to avoid getting
stuck in local maxima.

For example, you could throw a whole bunch of trial balloons into a room
with a bumpy ceiling, see which ones end up highest, and artificially
smooth out the cavities that the other balloons get stuck in. That way,
gradually the ceiling gets smoother, and the highest trial balloons end up
higher on later trials.

Simulated annealing runs in a slightly different way. (
http://en.wikipedia.org/wiki/Simulated_annealing ). You throw a bunch of
trial balloons in, and let them bounce around on the ceiling, so that they
bobble out of the local maxima and have a tendency to bounce up and into
higher and higher local maxima, hopefully including the global maximum. To
make sure that they don't bounce out of the global maximum, you gradually
decrease the bounciness of the balloons.

There have been attempts to find good designs through genetic algorithms
based on simulated annealing. They all depend on being able to reward
better designs, and you also have to have some idea of how to write down a
specification for a design.

What we're missing in the kind of design we want to do is the specification
language. What would you say in order to ensure that the right design
elements are being included?

In the initial discovery phase, that's what we need to find out from the
users that we interview. What is the problem space in which we will be
doing design? What concepts matter? What concepts are implied? What are the
most significant relationships among the concepts, namely the ones that the
users will want to follow? (I've got this, I want that, so I'm going to use
this relationship to get there.)

This is why I feel strongly that a good conceptual framework is essential
to defining the space in which to do design.

The complementary method works also, but I'm just finding out about it. You
put up a candidate design, find out what concepts and relationships are
missing, and revise the design. Because you're changing the framework as
you go along, I don't think that traditional optimization ideas transfer
very well to this sort of design.

A second point about this complementary method. Revising the design
requires two steps: adding the concepts and relationships, and figuring out
how to build a new design that incorporates them. The new concepts and
relationships may be disruptive to the existing design, so you hope in the
ideal case that you added the most fundamental concepts first. If you
didn't, you may have a radical re-design on your hands. Even if you did,
some of the things you inferred in your first design may need to be un-done
so that the new concepts and relationships can be incorporated as
thoroughly as if they were considered from the start.

Best wishes,

Bruce Esrig

At 06:49 PM 1/2/2008, Nick Iozzo wrote:
>A great discussion appears to be happening under the original topic. I
>wanted to further focus it into something I have been thinking a lot about
>recently. So I have hijacked it and created this new thread. My apologies
>to Oleh.
>
>I find all of the applications coming out to do quick real time testing of
>alternate design solutions a great tool to have in our toolbox. But it is
>just another tool, not a replacement. I see this as a complement (not a
>supplement) to traditional user research and testing.
>
>If you where to just use these tools to iterate and improve your design, I
>am sure you will be able to get the best design possible considering the
>point you started from. Maybe, however, if you started with a very
>different design your end iteration would perform even better. So, if the
>only tool you used was a micro-design evolution approach, then you may end
>up not with the best design possible, but with the best design considering
>the point you started from. (In mathematical terms, a local maximum
>http://en.wikipedia.org/wiki/Local_maximum).
>
>So how do you get the very best design possible?
>If you had all the time any money you wanted, you would build the best
>designs you could dream of that would cover as much of the design space as
>possible. You would send small, but significant traffic to each and keep
>tweaking the design until it was the best it could be. Eventually designs
>would begin to drop-out because other designs would be performing better.
>Finally, when you are left with a single design performing better then the
>rest, then you know you have the best design for that design space.
>
>Of course we have real world issue of making bad first impressions with
>users (not to mention the unlimited money part). So we can not follow a
>model like this.
>
>Instead, what we can do, is to develop prototypes of the best design
>possibilities we can dream up. We run small samples of users through these
>prototypes and we see which performs the best and which appears to have
>the best potential upside and most design flexibility. It may not be the
>design all the users said they liked the best. (What a user says and what
>they actually think are very different things).
>
>My opinion (In summary)
>These analytical tools that are coming out to do micro evaluations of
>design options are great. They will help you climb that hill to get your
>design to the best it can be! But, they will not help you pick the tallest
>hill to climb. It is too expensive to try and climb them all so you need a
>low-cost method to explore and evaluate the potential of each... a method
>like low-fidelity prototypes with usability test evaluations.
>
>We worked with one our client's, Orbitz, to do a presentation on this
>subject at Gartner's web innovation summit.
>http://agendabuilder.gartner.com/WEB1/WebPages/SessionDetail.aspx?EventSessionId=833.
>Its spin was more focused on the business topics of the issue and less on
>the design and design process side of things.
>
>Nick Iozzo
>Principal User Experience Architect
>
>tandemseven
>
>847.452.7442 mobile
>
>niozzo at tandemseven.com
>http://www.tandemseven.com/
>________________________________________________________________
>*Come to IxDA Interaction08 | Savannah*
>February 8-10, 2008 in Savannah, GA, USA
>Register today: http://interaction08.ixda.org/
>
>________________________________________________________________
>Welcome to the Interaction Design Association (IxDA)!
>To post to this list ....... discuss at ixda.org
>Unsubscribe ................ http://www.ixda.org/unsubscribe
>List Guidelines ............ http://www.ixda.org/guidelines
>List Help .................. http://www.ixda.org/help

3 Jan 2008 - 9:16am
Nicholas Iozzo
2007

Thanks for the info to simulated Annealing; first I heard of it.
But my concern is the cost in building production quality "trial balloons". I feel that it will be cheaper to use traditional usability testing techniques when you explore the design space early on in the process. Once you feel you have a candidate design that has a lot of great potential, then you can use the design micro-evolution techniques.

Nick Iozzo
Principal User Experience Architect

tandemseven

847.452.7442 mobile

niozzo at tandemseven.com
http://www.tandemseven.com/

From: Bruce Esrig
Sent: Wed 1/2/2008 6:42 PM
To: Nick Iozzo; discuss at ixda.org
Subject: Re: [IxDA Discuss] Traditional user testing vs micro design evolution (was: The death of web usability testing as we know it?)

I'll take a chance that the following thought hasn't already been expressed
on these threads.

A lot of work has been done in somewhat-numerical realms to avoid getting
stuck in local maxima.

For example, you could throw a whole bunch of trial balloons into a room
with a bumpy ceiling, see which ones end up highest, and artificially
smooth out the cavities that the other balloons get stuck in. That way,
gradually the ceiling gets smoother, and the highest trial balloons end up
higher on later trials.

Simulated annealing runs in a slightly different way. (
http://en.wikipedia.org/wiki/Simulated_annealing ). You throw a bunch of
trial balloons in, and let them bounce around on the ceiling, so that they
bobble out of the local maxima and have a tendency to bounce up and into
higher and higher local maxima, hopefully including the global maximum. To
make sure that they don't bounce out of the global maximum, you gradually
decrease the bounciness of the balloons.

There have been attempts to find good designs through genetic algorithms
based on simulated annealing. They all depend on being able to reward
better designs, and you also have to have some idea of how to write down a
specification for a design.

What we're missing in the kind of design we want to do is the specification
language. What would you say in order to ensure that the right design
elements are being included?

In the initial discovery phase, that's what we need to find out from the
users that we interview. What is the problem space in which we will be
doing design? What concepts matter? What concepts are implied? What are the
most significant relationships among the concepts, namely the ones that the
users will want to follow? (I've got this, I want that, so I'm going to use
this relationship to get there.)

This is why I feel strongly that a good conceptual framework is essential
to defining the space in which to do design.

The complementary method works also, but I'm just finding out about it. You
put up a candidate design, find out what concepts and relationships are
missing, and revise the design. Because you're changing the framework as
you go along, I don't think that traditional optimization ideas transfer
very well to this sort of design.

A second point about this complementary method. Revising the design
requires two steps: adding the concepts and relationships, and figuring out
how to build a new design that incorporates them. The new concepts and
relationships may be disruptive to the existing design, so you hope in the
ideal case that you added the most fundamental concepts first. If you
didn't, you may have a radical re-design on your hands. Even if you did,
some of the things you inferred in your first design may need to be un-done
so that the new concepts and relationships can be incorporated as
thoroughly as if they were considered from the start.

Best wishes,

Bruce Esrig

At 06:49 PM 1/2/2008, Nick Iozzo wrote:
>A great discussion appears to be happening under the original topic. I
>wanted to further focus it into something I have been thinking a lot about
>recently. So I have hijacked it and created this new thread. My apologies
>to Oleh.
>
>I find all of the applications coming out to do quick real time testing of
>alternate design solutions a great tool to have in our toolbox. But it is
>just another tool, not a replacement. I see this as a complement (not a
>supplement) to traditional user research and testing.
>
>If you where to just use these tools to iterate and improve your design, I
>am sure you will be able to get the best design possible considering the
>point you started from. Maybe, however, if you started with a very
>different design your end iteration would perform even better. So, if the
>only tool you used was a micro-design evolution approach, then you may end
>up not with the best design possible, but with the best design considering
>the point you started from. (In mathematical terms, a local maximum
>http://en.wikipedia.org/wiki/Local_maximum).
>
>So how do you get the very best design possible?
>If you had all the time any money you wanted, you would build the best
>designs you could dream of that would cover as much of the design space as
>possible. You would send small, but significant traffic to each and keep
>tweaking the design until it was the best it could be. Eventually designs
>would begin to drop-out because other designs would be performing better.
>Finally, when you are left with a single design performing better then the
>rest, then you know you have the best design for that design space.
>
>Of course we have real world issue of making bad first impressions with
>users (not to mention the unlimited money part). So we can not follow a
>model like this.
>
>Instead, what we can do, is to develop prototypes of the best design
>possibilities we can dream up. We run small samples of users through these
>prototypes and we see which performs the best and which appears to have
>the best potential upside and most design flexibility. It may not be the
>design all the users said they liked the best. (What a user says and what
>they actually think are very different things).
>
>My opinion (In summary)
>These analytical tools that are coming out to do micro evaluations of
>design options are great. They will help you climb that hill to get your
>design to the best it can be! But, they will not help you pick the tallest
>hill to climb. It is too expensive to try and climb them all so you need a
>low-cost method to explore and evaluate the potential of each... a method
>like low-fidelity prototypes with usability test evaluations.
>
>We worked with one our client's, Orbitz, to do a presentation on this
>subject at Gartner's web innovation summit.
>http://agendabuilder.gartner.com/WEB1/WebPages/SessionDetail.aspx?EventSessionId=833.
>Its spin was more focused on the business topics of the issue and less on
>the design and design process side of things.
>
>Nick Iozzo
>Principal User Experience Architect
>
>tandemseven
>
>847.452.7442 mobile
>
>niozzo at tandemseven.com
>http://www.tandemseven.com/
>________________________________________________________________
>*Come to IxDA Interaction08 | Savannah*
>February 8-10, 2008 in Savannah, GA, USA
>Register today: http://interaction08.ixda.org/
>
>________________________________________________________________
>Welcome to the Interaction Design Association (IxDA)!
>To post to this list ....... discuss at ixda.org
>Unsubscribe ................ http://www.ixda.org/unsubscribe
>List Guidelines ............ http://www.ixda.org/guidelines
>List Help .................. http://www.ixda.org/help

3 Jan 2008 - 9:56am
Barbara Ballard
2005

I actually took a swing at this (genetic algorithms to do control
panel layout) for my aborted dissertation. I chose control panel
layout because it was a limited design domain: knobs, dials, and so
forth, that couldn't particularly morph.

The problems I ran into were:

1. representing the design constraints
2. representing the candidate designs (each control had a type, size,
location, and some other variables)
3. adding genes during the evolution
4. determining "goodness"

For #4, I took a set of four heuristics from human factors: frequent
controls/displays should be larger and in the middle, important
controls should be larger and in the middle, controls should be
ordered by sequence of use, and controls with adverse effects should
be difficult to use (avoid accidental activation). Even when these
heuristics were developed, they were known to be simple. I could have
expanded into Fitts' law and several other evaluative rules, but what
I had was plenty complex. (too complex: this was three dissertations
not one)

The problem with these approaches in "real time", using live products
or usability testing, is the number of generations necessary.

If you want to have some fun seeing what these algorithms can do to
design, check out this genetic algorithm geek's response to
"intelligent design", and see that evolution can design clocks:
http://www.youtube.com/watch?v=mcAq9bmCeR0

On 1/2/08, Bruce Esrig <esrig-ia at esrig.com> wrote:
> Simulated annealing runs in a slightly different way. (
> http://en.wikipedia.org/wiki/Simulated_annealing ). You throw a bunch of
> trial balloons in, and let them bounce around on the ceiling, so that they
> bobble out of the local maxima and have a tendency to bounce up and into
> higher and higher local maxima, hopefully including the global maximum. To
> make sure that they don't bounce out of the global maximum, you gradually
> decrease the bounciness of the balloons.
>
> There have been attempts to find good designs through genetic algorithms
> based on simulated annealing. They all depend on being able to reward
> better designs, and you also have to have some idea of how to write down a
> specification for a design.
>
> What we're missing in the kind of design we want to do is the specification
> language. What would you say in order to ensure that the right design
> elements are being included?
>
> In the initial discovery phase, that's what we need to find out from the
> users that we interview. What is the problem space in which we will be
> doing design? What concepts matter? What concepts are implied? What are the
> most significant relationships among the concepts, namely the ones that the
> users will want to follow? (I've got this, I want that, so I'm going to use
> this relationship to get there.)
>

--
Barbara Ballard
barbara at littlespringsdesign.com 1-785-838-3003

Syndicate content Get the feed