ROI on Interaction Design - financial analysis

27 May 2005 - 2:40pm
9 years ago
13 replies
907 reads
Wendy Fischer
2004

I am waiting on my new copy of Cost Justifying Usability - An Internet Update. I am also reading a sample chapter of it about online surveys.

However, something I am particularly interested in is there any examples of financial analysis/modeling formulas, spreadsheets, etc, out there that are used to compute or model the exact ROI in a dollar amount?

I read the online survey chapter, but while it cited examples, I am looking for actual formulas/financial analysis tools I can hand my financial modeler and he can plug in numbers and predict the ROI on a given activity.

-Wendy

Comments

27 May 2005 - 8:47pm
Peter Merholz
2004

> However, something I am particularly interested in is there any
> examples of financial analysis/modeling formulas, spreadsheets, etc,
> out there that are used to compute or model the exact ROI in a dollar
> amount?

No.

Because it can't be done.

ROI is a red herring.

Well, ROI is a helpful model for thinking about the value of the work
we do.

But calculating specific numbers for "interaction design" is pretty
much impossible, because it's pretty much impossible to isolate the
influence of interaction design.

At Adaptive Path, we researched how companies value user experience,
and it lead to this report:
http://www.adaptivepath.com/publications/reports/businessvalue/

We went in, thinking we could learn how companies "calculate the ROI of
user experience." They don't. Even the savvy ones don't.

As close as we got was realizing that the work we do affects behavior,
and behavior can sometimes be linked to financial metrics. So, you look
for the metric which is of interest, and you determine how you need to
affect behavior to get that metric to trend the way you want it to.

The thing to know about ROI is that ROI is best used as a guesstimation
tool. Finance departments use ROI when they have a number of projects
side by side that they are planning to invest in. It's not an after the
fact tool. It also often requires, well, best guesses.

--peter

27 May 2005 - 8:50pm
dszuc
2005

May provide some insights:

http://www.forrester.com/ER/Research/Report/Summary/0,1338,11180,00.html

Rgds,

Daniel Szuc
Principal Usability Consultant
Apogee Usability Asia Ltd
www.apogeehk.com
'Usability in Asia'

-----Original Message-----
From:
discuss-interactiondesigners.com-bounces at lists.interactiondesigners.com
[mailto:discuss-interactiondesigners.com-bounces at lists.interactiondesigners.
com] On Behalf Of Peter Merholz
Sent: Saturday, May 28, 2005 9:47 AM
To: discuss at interactiondesigners.com
Subject: Re: [ID Discuss] ROI on Interaction Design - financial analysis

[Please voluntarily trim replies to include only relevant quoted material.]

> However, something I am particularly interested in is there any
> examples of financial analysis/modeling formulas, spreadsheets, etc,
> out there that are used to compute or model the exact ROI in a dollar
> amount?

No.

Because it can't be done.

ROI is a red herring.

Well, ROI is a helpful model for thinking about the value of the work
we do.

But calculating specific numbers for "interaction design" is pretty
much impossible, because it's pretty much impossible to isolate the
influence of interaction design.

At Adaptive Path, we researched how companies value user experience,
and it lead to this report:
http://www.adaptivepath.com/publications/reports/businessvalue/

We went in, thinking we could learn how companies "calculate the ROI of
user experience." They don't. Even the savvy ones don't.

As close as we got was realizing that the work we do affects behavior,
and behavior can sometimes be linked to financial metrics. So, you look
for the metric which is of interest, and you determine how you need to
affect behavior to get that metric to trend the way you want it to.

The thing to know about ROI is that ROI is best used as a guesstimation
tool. Finance departments use ROI when they have a number of projects
side by side that they are planning to invest in. It's not an after the
fact tool. It also often requires, well, best guesses.

--peter

_______________________________________________
Welcome to the Interaction Design Group!
To post to this list ....... discuss at ixdg.org
(Un)Subscription Options ... http://discuss.ixdg.org/ Announcements List
......... http://subscribe-announce.ixdg.org/
Questions .................. lists at ixdg.org
Home ....................... http://ixdg.org/

28 May 2005 - 9:04am
Jared M. Spool
2003

At 09:47 PM 5/27/2005, Peter Merholz wrote:
>However, something I am particularly interested in is there any examples
>of financial analysis/modeling formulas, spreadsheets, etc, out there that
>are used to compute or model the exact ROI in a dollar amount?
>
>No.
>
>Because it can't be done.

Actually, it can.

But it's very complicated and expensive. Need a solid background in
econometric modelling and derivatives analysis.

It's probably not worth doing, because when all the analysis is said and
done, it's only as good as the myriad of assumptions you have to put into
it. If you do the research required to validate the assumptions, you've
actually done enough research to tell you exactly what to fix and no longer
need to justify the work.

Inotherwords, the cost to cost-justify isn't justified.

Maybe this will help instead? (It's cheaper than the Forrester Report that
Daniel suggested and says basically the same thing.)

Identifying the Business Value of What We Do -- 04/15/2005

Resources in our organization are usually tightly constrained -- not
enough time,
money, or people to accomplish everything we want to improve. Knowing
how to
identify and communicate the business value of a project will
substantially help
it get approved and supported by the organization. Jared talks about
the key five
business value areas and how to relate design improvements into the
overall
success of the organization.

http://www.uie.com/articles/business_value/

Jared

Jared M. Spool
User Interface Engineering
http://www.uie.com jspool at uie.com

UI10 Spotlight Presenter: Flow author Mihaly Csikszentmihalyi
See details at http://www.uiconf.com

28 May 2005 - 11:35am
Nick Ragouzis
2004

This (below) is just so much high-priest hooey. From both camps.

It is not high economics. It is simple, straight-forward real options
analysis. Do you know it (not just the math, but also the thinking)?
Probably not, but that's a HUGE GAP in your practice and knowledge.

Once you do know it, and practice the thinking that goes along
with it, you will find it useful from the very early faint
traces of the project, when you have almost no "firm" knowledge,
through to many decisions during the project, and on to the next
project. In other words, rather than imposing an unchanged huge
data gathering and analytical burden, it scales in parallel with
the changes in your knowledge, tolerance for types of risks,
nature and levels of investments, and expectations among various
stakeholders for returns.

Those methods are also crucial to basic design decisions,
including competitive analysis and innovation processes, as well
as scaling research, and testing.

Significantly, these analyses are the only credible way to judge
intangible returns ... just the types of things we like to deliver.
Traditional High/medium/low sensitivity analysis, looking oh-so
scientific and balanced, is entirely discredited for this. It is
a fraud perpetrated on clients.

VERIFIABLE IMPROVEMENTS IN USER PERCEIVED VALUE, the crux of what
counts in this practice and what delivers leveragable value to
clients, cannot be dealt with that way, and the proxies are not
resilient or robust (i.e. they are easily disturbed or undermined
by many other common, probable, activities).

But, happily or unfortunately according to POV, most client's
design decision teams also don't know this. That is, many clients
still believe the only ROI worth looking is the "hard" stuff, cash
they can keep in pocket or watch being moved between accounts.
This the stuff of flat-earth thinking. Many have played to that
gallery, a codependency supported by claiming to just be trying
to get some tiny bit of something useful out to real users, and
feeding the designer-as-god phenomenon.

You can get some idea of how tied up this is in what interaction
designers do by checking out the chi-web thread on Six Sigma in
Dec 2003, where I also reference my presentation from a CHI2001
panel on measuring interaction design. These will lead you to
other related work. Even in the ACM DL (not the least of which,
and still barely realized, is the 1998 Lukose and Huberman work,
if you have a taste for the more complex aspects, and a thirst
to see how this methodology underlies everything you are doing).

But, really, the best thing is to just get started. Trying just
a little bit of it, learning as you go, will give you some really
great insights. IOW, it will give *you* a strong, early, ROI.

--Nick

> Jared Spool wrote:
>
> At 09:47 PM 5/27/2005, Peter Merholz wrote:
> >However, something I am particularly interested in is there
> any examples of financial analysis/modeling formulas,
> spreadsheets, etc, out there that are used to compute or model
> the exact ROI in a dollar amount?
> >
> >No.
> >
> >Because it can't be done.
>
> Actually, it can.
>
> But it's very complicated and expensive. Need a solid
> background in econometric modelling and derivatives analysis.
>
> It's probably not worth doing, because when all the analysis
> is said and done, it's only as good as the myriad of assumptions
> you have to put into it. If you do the research required to
> validate the assumptions, you've actually done enough research
> to tell you exactly what to fix and no longer need to justify
> the work.
>
> Inotherwords, the cost to cost-justify isn't justified.
>

28 May 2005 - 12:43pm
Jared M. Spool
2003

I'm sorry, Nick. I don't understand. Are you saying *you have* spreadsheet
models for Wendy to use?

Jared "Hooey at No Charge" Spool

At 12:35 PM 5/28/2005, Nick Ragouzis wrote:

>This (below) is just so much high-priest hooey. From both camps.
>
>It is not high economics. It is simple, straight-forward real options
>analysis. Do you know it (not just the math, but also the thinking)?
>Probably not, but that's a HUGE GAP in your practice and knowledge.

...

>But, really, the best thing is to just get started. Trying just
>a little bit of it, learning as you go, will give you some really
>great insights. IOW, it will give *you* a strong, early, ROI.
>
>--Nick
>
> > Jared Spool wrote:
> >
> > At 09:47 PM 5/27/2005, Peter Merholz wrote:
> > >However, something I am particularly interested in is there
> > any examples of financial analysis/modeling formulas,
> > spreadsheets, etc, out there that are used to compute or model
> > the exact ROI in a dollar amount?
> > >
> > >No.
> > >
> > >Because it can't be done.
> >
> > Actually, it can.
> >
> > But it's very complicated and expensive. Need a solid
> > background in econometric modelling and derivatives analysis.
> >
> > It's probably not worth doing, because when all the analysis
> > is said and done, it's only as good as the myriad of assumptions
> > you have to put into it. If you do the research required to
> > validate the assumptions, you've actually done enough research
> > to tell you exactly what to fix and no longer need to justify
> > the work.
> >
> > Inotherwords, the cost to cost-justify isn't justified.
> >

Jared M. Spool, Founding Principal
User Interface Engineering
4 Lookout Lane, Unit 4d
Middleton, MA 01949
978 777-9123
jspool at uie.com
http://www.uie.com

28 May 2005 - 5:48pm
Nick Ragouzis
2004

Jared,

Well, I don't know which to address first. Let's try
this order:

Jared wrote:
> Are you saying *you have* spreadsheet models

Yes, of course I do. In Excel, in Mathematica, in Matlab
supported by GraphViz, in Python/OpenGL. Complete with graphs.
For analytic approaches and binomial tree approaches, plus
some modeling of the way things might change, among others.
Also running the simplex modeling that can tease apart the
relationships. All sounding much fancier than it is (sometimes
it looks sooo hacky and simple I feel compelled to graphjunk
it ... to increase its credibility :-) ).

And anybody can have these on quick order. The web has many
such calculators available for simple starters. There's even
a simple and really easy to teach, yet very useful, tool:
"Insight", that you can get bundled with Sam Savage's Decision
Making with Insight book.

Binomial trees is something you can do with an ordinary
calculator. The thinking is much more important than the math.
And it is simple math. If you're more inclined towards your
own programming, you can make great progress fast with Jackson
and Staunton's Advanced Modelling [sic] In Finance using Excel
and VBA -- the CD's got it all, including the more complex (yet
merely undergraduate) maths. But a less serious-looking version
is in Benninga's Financial Modeling -- it has more pictures and
spreadsheets (and a CD).

Just how simple it all is, and how much it depends on thinking
about the problem, not running "complex econometrics," is brought
clearly to the surface with Timothy Luehrman's work, for one.
Spend the $ to get the bundle of his reprints from HBR. Start
with the APR paper ... but you won't get the depth of the ideas
until you dig deeper.

One way to glimpse that deeper result is Lenos Trigeorgis'
Real Options -- read this not for the math, but the models for
thinking ... from Chapter 1 written around Table 1.1, to the
entirety of chapter 4, as well as chapters 8 and 9. By that point
you should already have a charged understanding that the key is
thinking about your problem, and the results you desire. And that
these methods let you talk to ROI-heads about *that*, not some
very remote, fragile proxy.

Exhale.

> for Wendy to use?

Uh? You aren't suggesting that as a working professional I give
away the high value work I do for a living, are you? Not only
would that counter ethical conduct advised by the AIGA (no longer
belong) and ACM (still belong), among others, it would be
irresponsible. Providing, gratis, significant expertise for a
specific good cause, as I do, constantly, is one thing. It's an
important thing to do. I know you do that too.

But Wendy has lots of choices for sources. A teacher, a researcher,
or a publishing author (even among us, for those whose practice is
less directly in the area, or who feel differently). She might
even take it up herself and then tell her tale of exploration.

Inhale.

> I'm sorry, Nick. I don't understand.

I'm guessing that you do understand. I haven't read UIE's paper,
or Adaptive Path's, so I'm not saying folks won't get results from
either.

I *am* saying: it's not complicated, remote, mysterious, arduous,
taxing, or fruitless.

I *am* saying: if an interaction designer doesn't know this way of
thinking, and can't work through a simple model, that designer isn't
qualified to be thinking about "a seat at the table;" try another
branch of design, ally closely with a parther that does understand.

I *am* saying: if you just flip a coin on the iteration/design
question in front of you, and spend the saved few hours learning
about this ... well, then, you will have already demonstrated for
yourself the value of real options to interaction design. Later,
looking back, you will understand why that decision methodology
had as much creditability as the one you thought you were about
to undertake.

--Nick

> -----Original Message-----
> From: Jared M. Spool [mailto:jspool at uie.com]
> Sent: Saturday, May 28, 2005 10:43 AM
> To: Nick Ragouzis
> Cc: 'Peter Merholz'; discuss at interactiondesigners.com
> Subject: RE: [ID Discuss] ROI on Interaction Design -
> financial analysis
>
>
> I'm sorry, Nick. I don't understand. Are you saying *you
> have* spreadsheet models for Wendy to use?
>
> Jared "Hooey at No Charge" Spool
>
> At 12:35 PM 5/28/2005, Nick Ragouzis wrote:
>
> >This (below) is just so much high-priest hooey. From both camps.
> >
> >It is not high economics. It is simple, straight-forward real options
> >analysis. Do you know it (not just the math, but also the thinking)?
> >Probably not, but that's a HUGE GAP in your practice and knowledge.
>
> ...
>
> >But, really, the best thing is to just get started. Trying just
> >a little bit of it, learning as you go, will give you some really
> >great insights. IOW, it will give *you* a strong, early, ROI.
> >
> >--Nick
> >
> > > Jared Spool wrote:
> > >
> > > At 09:47 PM 5/27/2005, Peter Merholz wrote:
> > > >However, something I am particularly interested in is there
> > > any examples of financial analysis/modeling formulas,
> > > spreadsheets, etc, out there that are used to compute or model
> > > the exact ROI in a dollar amount?
> > > >
> > > >No.
> > > >
> > > >Because it can't be done.
> > >
> > > Actually, it can.
> > >
> > > But it's very complicated and expensive. Need a solid
> > > background in econometric modelling and derivatives analysis.
> > >
> > > It's probably not worth doing, because when all the analysis
> > > is said and done, it's only as good as the myriad of assumptions
> > > you have to put into it. If you do the research required to
> > > validate the assumptions, you've actually done enough research
> > > to tell you exactly what to fix and no longer need to justify
> > > the work.
> > >
> > > Inotherwords, the cost to cost-justify isn't justified.
> > >
>
> Jared M. Spool, Founding Principal
> User Interface Engineering
> 4 Lookout Lane, Unit 4d
> Middleton, MA 01949
> 978 777-9123
> jspool at uie.com
> http://www.uie.com
>
>

28 May 2005 - 7:32pm
Nick Ragouzis
2004

By the way, I hasten to underscore one aspect in this dialog
... Jared "Hooey at No Charge" Spool is right, IMO, about his
intermediate claim:

> > > Actually, it can [be done].
...
> > > when all the analysis is said and done, it's only
> > > as good as the myriad of assumptions you have to
> > > put into it. If you do the research required to
> > > validate the assumptions, you've actually done
> > > enough research to tell you exactly what to fix
...

It's his apparent consequent, the claim

> > > the cost to cost-justify isn't justified

with which I took issue. Mainly because it's really a corollary
to the actual central claim, implicitly made, that the there's a
"said and done" here, and that it's too difficult to get there.

The key is shifting to an understanding that the research
*is* the design, and v.v., and that there is a set of tools
that provide guidance along the way. Real options has a
significant role, scaling flexibly with what's known and
the certainty desired. And so on, as addressed earlier.

. . . .

On prompting I'll add two additional notes to the prior message.

One is that, as far as thinking in this mode goes, I have
published, in 1993, a use case[1]. It will only be interesting
(if at all!) to those thinking from the consulting end -- don't
go there otherwise, really. (the locus is not interaction design,
but office systems support). The ONLY possibly relevant aspects
are the sections "Organizational Transformation" and "Economics
of Information." There's no math.

Another is that, I'm reminded, the CHI2001 presentation (much
more relevant than the above) does have two items with direct
applicability (perhaps) on this topic. One is the 2-slide list
of "Wealth of Opportunities." More on that in a minute. The
other is at the end, showing the effect of the Inverse Gaussian
-- which is something you can program on any decent calculator
(that's a shot of HP49G) or have primed on your laptop and use
in any meeting where traffic discussions come up, to stop dead
the usual foolishness. There's a reference on the slide to the
Science article. But whoowho, it's now posted on the W3C site[2].

. . .

Back to that comment about "Wealth of Opportunities." It relates
to "Economics of Information," and together the two can suggest
many routes forward to evaluating a project. Most are obvious;
I'll suggest two benefits that are often overlooked.

First, once you understand the approach, and have helped your
client understand it too, you can use the approach to divert
nonsensical arguments about design details or approaches.
It helps you place a value on the information you might get,
against the time and other 'soft' factors. Notice, you aren't
left simply with arguments about cash and deadlines (which
seem 'hard' but are actually relative). You now have a basis to
argue for starting before all information is known, you can even
use the method to find the best places to start and levels to
run at.

Second, you can use the method to price and structure your services.
I have somewhere, probably on one of my abandoned Macs (in favor
of MSOS/CommercialCOTS/OpenSource, and LinuxOS/OpenSource), a draft
of a response to an earlier dialog on this, probably on the AIGA
Advanced list. But there are very direct and flexible, and risk
mitigating, ways to structure pricing of design work on performance.
Real options are an excellent scaffolding for the dialog and
proposing the relationship frameworks. The eventual model might not
be built on a derivatives formula, but the route taken to get there
will have amplified the true values for the parties.

--Nick

[1] <http://www.enosis.com/resources/trilevel.pdf>
(It's really not all that interesting.)

[2] <http://www.w3.org/Protocols/HTTP-NG/1998/02/1998-02-surfing-final.pdf>

29 May 2005 - 8:23pm
Wendy Fischer
2004

Heh, I am only really looking for a point to start from. I think I have enough info to start from the literature and I have a bunch of spreadsheet happy Harvard MBA types that love calculating the ROI of my existence. I wanted to give them a start so they could swoosh through the excel spreadsheets, crunch numbers and think happy thoughts.....from my experience these guys can cost justify my need for a parking spot and tie it to increase in revenue.

-Wendy

Nick Ragouzis <nickr at radicalmode.com> wrote:
[Please voluntarily trim replies to include only relevant quoted material.]

Jared,

Well, I don't know which to address first. Let's try
this order:

Jared wrote:
> Are you saying *you have* spreadsheet models

Yes, of course I do. In Excel, in Mathematica, in Matlab
supported by GraphViz, in Python/OpenGL. Complete with graphs.
For analytic approaches and binomial tree approaches, plus
some modeling of the way things might change, among others.
Also running the simplex modeling that can tease apart the
relationships. All sounding much fancier than it is (sometimes
it looks sooo hacky and simple I feel compelled to graphjunk
it ... to increase its credibility :-) ).

And anybody can have these on quick order. The web has many
such calculators available for simple starters. There's even
a simple and really easy to teach, yet very useful, tool:
"Insight", that you can get bundled with Sam Savage's Decision
Making with Insight book.

Binomial trees is something you can do with an ordinary
calculator. The thinking is much more important than the math.
And it is simple math. If you're more inclined towards your
own programming, you can make great progress fast with Jackson
and Staunton's Advanced Modelling [sic] In Finance using Excel
and VBA -- the CD's got it all, including the more complex (yet
merely undergraduate) maths. But a less serious-looking version
is in Benninga's Financial Modeling -- it has more pictures and
spreadsheets (and a CD).

Just how simple it all is, and how much it depends on thinking
about the problem, not running "complex econometrics," is brought
clearly to the surface with Timothy Luehrman's work, for one.
Spend the $ to get the bundle of his reprints from HBR. Start
with the APR paper ... but you won't get the depth of the ideas
until you dig deeper.

One way to glimpse that deeper result is Lenos Trigeorgis'
Real Options -- read this not for the math, but the models for
thinking ... from Chapter 1 written around Table 1.1, to the
entirety of chapter 4, as well as chapters 8 and 9. By that point
you should already have a charged understanding that the key is
thinking about your problem, and the results you desire. And that
these methods let you talk to ROI-heads about *that*, not some
very remote, fragile proxy.

Exhale.

> for Wendy to use?

Uh? You aren't suggesting that as a working professional I give
away the high value work I do for a living, are you? Not only
would that counter ethical conduct advised by the AIGA (no longer
belong) and ACM (still belong), among others, it would be
irresponsible. Providing, gratis, significant expertise for a
specific good cause, as I do, constantly, is one thing. It's an
important thing to do. I know you do that too.

But Wendy has lots of choices for sources. A teacher, a researcher,
or a publishing author (even among us, for those whose practice is
less directly in the area, or who feel differently). She might
even take it up herself and then tell her tale of exploration.

Inhale.

> I'm sorry, Nick. I don't understand.

I'm guessing that you do understand. I haven't read UIE's paper,
or Adaptive Path's, so I'm not saying folks won't get results from
either.

I *am* saying: it's not complicated, remote, mysterious, arduous,
taxing, or fruitless.

I *am* saying: if an interaction designer doesn't know this way of
thinking, and can't work through a simple model, that designer isn't
qualified to be thinking about "a seat at the table;" try another
branch of design, ally closely with a parther that does understand.

I *am* saying: if you just flip a coin on the iteration/design
question in front of you, and spend the saved few hours learning
about this ... well, then, you will have already demonstrated for
yourself the value of real options to interaction design. Later,
looking back, you will understand why that decision methodology
had as much creditability as the one you thought you were about
to undertake.

--Nick

> -----Original Message-----
> From: Jared M. Spool [mailto:jspool at uie.com]
> Sent: Saturday, May 28, 2005 10:43 AM
> To: Nick Ragouzis
> Cc: 'Peter Merholz'; discuss at interactiondesigners.com
> Subject: RE: [ID Discuss] ROI on Interaction Design -
> financial analysis
>
>
> I'm sorry, Nick. I don't understand. Are you saying *you
> have* spreadsheet models for Wendy to use?
>
> Jared "Hooey at No Charge" Spool
>
> At 12:35 PM 5/28/2005, Nick Ragouzis wrote:
>
> >This (below) is just so much high-priest hooey. From both camps.
> >
> >It is not high economics. It is simple, straight-forward real options
> >analysis. Do you know it (not just the math, but also the thinking)?
> >Probably not, but that's a HUGE GAP in your practice and knowledge.
>
> ...
>
> >But, really, the best thing is to just get started. Trying just
> >a little bit of it, learning as you go, will give you some really
> >great insights. IOW, it will give *you* a strong, early, ROI.
> >
> >--Nick
> >
> > > Jared Spool wrote:
> > >
> > > At 09:47 PM 5/27/2005, Peter Merholz wrote:
> > > >However, something I am particularly interested in is there
> > > any examples of financial analysis/modeling formulas,
> > > spreadsheets, etc, out there that are used to compute or model
> > > the exact ROI in a dollar amount?
> > > >
> > > >No.
> > > >
> > > >Because it can't be done.
> > >
> > > Actually, it can.
> > >
> > > But it's very complicated and expensive. Need a solid
> > > background in econometric modelling and derivatives analysis.
> > >
> > > It's probably not worth doing, because when all the analysis
> > > is said and done, it's only as good as the myriad of assumptions
> > > you have to put into it. If you do the research required to
> > > validate the assumptions, you've actually done enough research
> > > to tell you exactly what to fix and no longer need to justify
> > > the work.
> > >
> > > Inotherwords, the cost to cost-justify isn't justified.
> > >
>
> Jared M. Spool, Founding Principal
> User Interface Engineering
> 4 Lookout Lane, Unit 4d
> Middleton, MA 01949
> 978 777-9123
> jspool at uie.com
> http://www.uie.com
>
>

_______________________________________________
Welcome to the Interaction Design Group!
To post to this list ....... discuss at ixdg.org
(Un)Subscription Options ... http://discuss.ixdg.org/
Announcements List ......... http://subscribe-announce.ixdg.org/
Questions .................. lists at ixdg.org
Home ....................... http://ixdg.org/

29 May 2005 - 8:51pm
Wendy Fischer
2004

One comment comes to mind, a quote from Peter Merholz's website.

"Nick is scary-smart. Listen up."

-Wendy

Nick Ragouzis <nickr at radicalmode.com> wrote:
[Please voluntarily trim replies to include only relevant quoted material.]

By the way, I hasten to underscore one aspect in this dialog
... Jared "Hooey at No Charge" Spool is right, IMO, about his
intermediate claim:

> > > Actually, it can [be done].
...
> > > when all the analysis is said and done, it's only
> > > as good as the myriad of assumptions you have to
> > > put into it. If you do the research required to
> > > validate the assumptions, you've actually done
> > > enough research to tell you exactly what to fix
...

It's his apparent consequent, the claim

> > > the cost to cost-justify isn't justified

with which I took issue. Mainly because it's really a corollary
to the actual central claim, implicitly made, that the there's a
"said and done" here, and that it's too difficult to get there.

The key is shifting to an understanding that the research
*is* the design, and v.v., and that there is a set of tools
that provide guidance along the way. Real options has a
significant role, scaling flexibly with what's known and
the certainty desired. And so on, as addressed earlier.

. . . .

On prompting I'll add two additional notes to the prior message.

One is that, as far as thinking in this mode goes, I have
published, in 1993, a use case[1]. It will only be interesting
(if at all!) to those thinking from the consulting end -- don't
go there otherwise, really. (the locus is not interaction design,
but office systems support). The ONLY possibly relevant aspects
are the sections "Organizational Transformation" and "Economics
of Information." There's no math.

Another is that, I'm reminded, the CHI2001 presentation (much
more relevant than the above) does have two items with direct
applicability (perhaps) on this topic. One is the 2-slide list
of "Wealth of Opportunities." More on that in a minute. The
other is at the end, showing the effect of the Inverse Gaussian
-- which is something you can program on any decent calculator
(that's a shot of HP49G) or have primed on your laptop and use
in any meeting where traffic discussions come up, to stop dead
the usual foolishness. There's a reference on the slide to the
Science article. But whoowho, it's now posted on the W3C site[2].

. . .

Back to that comment about "Wealth of Opportunities." It relates
to "Economics of Information," and together the two can suggest
many routes forward to evaluating a project. Most are obvious;
I'll suggest two benefits that are often overlooked.

First, once you understand the approach, and have helped your
client understand it too, you can use the approach to divert
nonsensical arguments about design details or approaches.
It helps you place a value on the information you might get,
against the time and other 'soft' factors. Notice, you aren't
left simply with arguments about cash and deadlines (which
seem 'hard' but are actually relative). You now have a basis to
argue for starting before all information is known, you can even
use the method to find the best places to start and levels to
run at.

Second, you can use the method to price and structure your services.
I have somewhere, probably on one of my abandoned Macs (in favor
of MSOS/CommercialCOTS/OpenSource, and LinuxOS/OpenSource), a draft
of a response to an earlier dialog on this, probably on the AIGA
Advanced list. But there are very direct and flexible, and risk
mitigating, ways to structure pricing of design work on performance.
Real options are an excellent scaffolding for the dialog and
proposing the relationship frameworks. The eventual model might not
be built on a derivatives formula, but the route taken to get there
will have amplified the true values for the parties.

--Nick

[1]
(It's really not all that interesting.)

[2]

_______________________________________________
Welcome to the Interaction Design Group!
To post to this list ....... discuss at ixdg.org
(Un)Subscription Options ... http://discuss.ixdg.org/
Announcements List ......... http://subscribe-announce.ixdg.org/
Questions .................. lists at ixdg.org
Home ....................... http://ixdg.org/

29 May 2005 - 11:05pm
Peter Merholz
2004

> The key is shifting to an understanding that the research
> *is* the design, and v.v., and that there is a set of tools
> that provide guidance along the way...

For the sake of clarity (and possibly beating a horse long dead), I
didn't mean to suggest that you shouldn't try to measure the value of
your interaction design work. We're big big fans of accountability and
metrics, and are frustrated that designers don't do more to step up and
take accountability (and the good and the bad that goes with it).

However, we don't believe that "ROI," as it's traditionally practiced,
is all that relevant to interaction design. And I think (if I can parse
Nick's initial response), we agree with Nick on that. We think that
thinking about returns on investment are important, but calculating ROI
with some fancy equation isn't really the point.

The point is:
- measure the NOW - and something with some financial ramification
- set a benchmark (at this point, a hypothesis)
- engage in your design intervention (whatever that is)
- measure again
- revise your benchmark based on this new data
- lather, rinse, repeat

Whatever you do, DON'T bother with Nielsen/Norman Group's Usability
ROI. Scott Hirsch and I wrote a review of that claptrap here:

http://www.boxesandarrows.com/archives/
report_review_nielsennorman_groups_usability_return_on_investment.php

And I'm still a fan of Aaron Marcus' literature review:

http://www.amanda.com/resources/ROI/AMA_ROIWhitePaper_28Feb02.pdf

--peter

30 May 2005 - 12:48pm
Nick Ragouzis
2004

Peter:

Apart from that most scurrilous calumny quoted by Wendy
(C&D letter on its way to you, Peter :-), I think all the
commentators on this topic *are* mostly in agreement in
the sense that designers better get better at measurement.

But I think we may still not be aligned on the methods
and purpose.

Peter Merholz wrote, on 29May05:
> We're big big fans of accountability and metrics, and are
> frustrated that designers don't do more to step up and take
> accountability (and the good and the bad that goes with it).

I agree. But most folks also agree, in a certain way.

Usually: of accountability and metrics of the now and the
designers responsible (in that: "How could they have been
so blind to do/not do thus and such? Now things really need
fixin'." sort of way.). Plus, accountability and metrics of
the future on the promise (in that: "You agree you'll love it,
right? Just initial our plan right here." sort of way.).

> However, we don't believe that "ROI," as it's traditionally
> practiced, is all that relevant to interaction design. And
> I think (if I can parse Nick's initial response), we agree
> with Nick on that.

I agree, sort of. The traditional practice is not all that
relevant in many respects to design decisions (particularly
those involving information interaction, the most relevant
way to cast interaction design for this discussion, bringing
in high relief the ways in which cash-flow ROI fails us).
But if incorporated in a certain way, many parts of that
same practice gains much importance. Beyond that point,
there's a, perhaps surprising, divergence.

> We think that thinking about returns on investment are
> important, but calculating ROI with some fancy equation
> isn't really the point.

Since 'fancy' is relative, it's difficult to know if we agree
here. What's the threshold? The notion that there was a time
value to money was once considered fancy; now it is so ordinary
that it's irresponsible not to account for time in estimates
of future returns.

> The point is:
> - measure the NOW - and something with some financial
> ramification
> - set a benchmark (at this point, a hypothesis)
> - engage in your design intervention (whatever that is)
> - measure again
> - revise your benchmark based on this new data
> - lather, rinse, repeat

Below I'll point out one of the crucially important missing
ingredients in the above (traditional) recipe. Nothing fancy,
but rather something immediately obvious once someone begins
to think about design decisions in terms of real options.

Before getting to that I'll also mention this: A design team
will face very little controversy in taking your approach. It
seems so sensible, so familiar. Everybody, absolutely everybody
knows it's right. Well, it's probably *not* right.

Just as an experienced chef can coax the sublime from a recipe
where a tyro delivers crust, the ordinary design team following
this recipe is likely to focus on the structure, with rote
execution, and not on the refinements crucial to the outcome.
IOW, your team may be producing haute cuisine from this, but
it's to your team's credit, not the recipe.

In an interesting parallel to the chef, it's actually easier
to do it in a way that's likely to produce better results for
the design decisions any team is facing at any moment.

I mentioned above that there's at least one crucially important
missing ingredient. Let me introduce it in this way, which
some may find surprising.

Lada Gorlenko, just yesterday, on the "Design education" thread,
contributed a trenchant critique of an aspect of Michael Beirut's
comments to the Design Council. Surprisingly, perhaps, both
points of view align on something crucially important to this
topic: an organization, and a design team, and a designer, very
rarely rises above themselves. Further, you need extraordinary
interventions and efforts to change that course, and (I'll add)
one should be extremely skeptical of any such claims (IOW, use
aggressive discounting).

The immediate result (immediate once you know the method) is
that your first step is wrong, in an important way. Measuring
the now (meaning the costs and the returns being claimed as
associated with the current design -- and I'm ignoring entirely
the crucial question of tangible vs intangible) is almost
inconsequential for decisions with characteristics like those
of design decisions.

Oh, sure, you need some sense of where you are ... but for that
to be of *any* credible use you'll also need some sense of the
likelihood that you're wrong about that. Wrong in the scope you
choose, wrong in the actual claim of relevance, wrong in the
numbers ... by large amounts. But I digress.

At that stage, the outset of the measurement cycle, the most
important measure is of the past. Not merely the measures
of investments and returns (reporting those, themselves, will
help very little), but (here it is):

... of how those rates have changed over time.
You need to understand the volatility of the
returns on design decisions and efforts.

IOW, with this understanding and a few other bits of information,
and a properly-structured definition of the effort being
considered (perhaps the most challenging, and fruitful, aspect,
including just which investments and returns are relevant and
to what scale ... and that holds a surprising aspect for all
that work through it), in a few hour's work a design team can
give a reasonable assessment (reasonable given where they stand,
and relative to other methods) of the likely returns to their
design decision.

The suggested recipe is missing other, crucial, ingredients too.
Such as a parallel incorporation of information about the
environment, the form of the benchmarks, the way these measures
should inform the design intervention, the later measurements
and where and how they are derived, and what is actually revised
and assessed in iterations. These all have to be converted to
an alternate form.

All of which, I claim, is crucial and highly relevant to a
skilled practice of design ... and to an organization, a team,
or an individual improving their results from design. It *is*
[integral to] design, not something abstract or distant.

--Nick

30 May 2005 - 2:29pm
Lada Gorlenko
2004

Peter Merholtz wrote:
>> The point is:
>> - measure the NOW - and something with some financial
>> ramification
>> - set a benchmark (at this point, a hypothesis)
>> - engage in your design intervention (whatever that is)
>> - measure again
>> - revise your benchmark based on this new data
>> - lather, rinse, repeat

Nick Ragouzis commented:
NR> The suggested recipe is missing other, crucial, ingredients too.
NR> Such as a parallel incorporation of information about the
NR> environment, the form of the benchmarks, the way these measures
NR> should inform the design intervention, the later measurements
NR> and where and how they are derived, and what is actually revised
NR> and assessed in iterations. These all have to be converted to
NR> an alternate form.

There are [at least] two sides to ROI in Interaction Design.

One is ROI of an *IxD artefact* where 'design' means 'product'.
This is where Peter's recipe is straightforward, concise and fits
in nicely for me. Set a benchmark, set a target, build a bridge,
see if it closes the gap, make adjustments, evaluate ROI of your
design-product.

Another side is ROI of an *IxD practice* where 'design' means
'process'. It is about how design is done (provided we've found
a place for it in our org and project charts), about the nitty-gritty
of planning, running, maintaining and measuring the effectiveness of
design-process. It is very much along the lines of the recent "Why
hire an IxD instead of a software developer" discussion and further.
And this is my understanding of what Nick describes as missing in
Peter's recipe. All these things (incorporation of info about
environment, how measuremens are derived, etc.) are missing - but
missing from a recipe for running a kitchen, not from a recipe for
boiling an egg. The two are different recipes.

So, I guess if we make a distinction between design-artefacts and
design-processes, we shall know better what return on what investment
we are calculating. I wish though I also had a recipe for living
happily ever after [ROI reports] - any suggestions? :-)

Lada

31 May 2005 - 1:28pm
Dave Cronin
2005

I certainly appreciate the sentiment here-- especially with regard to
accountability and metrics, and it sounds like Nick has a handle on the
models (even if it's a bit opaque to me despite the fact I have a degree
in mathematics ;), but something that I'm worried about not being
adequately represented is the saved costs that would have been incurred
by heading down an incorrect or poorly articulated product
design/definition path (resulting from not having IxD as part of the
team).

Another way of saying it, I guess, is that what Peter describes below
sounds like a really good way of estimating value as realized through
revenue, but I'm unclear as to whether it captures the efficiencies
created by having IxD.

And again, I'm not terribly knowledgeable about the econometric models
behind ROI calculations, so this post should be read more as a question
than a refutation.

-dave

> Peter Merholz wrote:
>
> For the sake of clarity (and possibly beating a horse long
> dead), I didn't mean to suggest that you shouldn't try to
> measure the value of your interaction design work. We're big
> big fans of accountability and metrics, and are frustrated
> that designers don't do more to step up and take
> accountability (and the good and the bad that goes with it).
>
> However, we don't believe that "ROI," as it's traditionally
> practiced, is all that relevant to interaction design. And I
> think (if I can parse Nick's initial response), we agree with
> Nick on that. We think that thinking about returns on
> investment are important, but calculating ROI with some fancy
> equation isn't really the point.
>
> The point is:
> - measure the NOW - and something with some financial ramification
> - set a benchmark (at this point, a hypothesis)
> - engage in your design intervention (whatever that is)
> - measure again
> - revise your benchmark based on this new data
> - lather, rinse, repeat

Syndicate content Get the feed