Usability Budgets

25 Oct 2004 - 3:50am
9 years ago
29 replies
880 reads
Sunit
2004

Hello there,

I need some stats on the budgets allocated to the interaction design/
usability process in a typical project (Online application/ Software).
What percentage do the big guys spend on usability/Interaction Design/ in
comparison to the cost of developing the entire product.

Links to any such studies/ statistics would be really helpful.

Warm Regards,
Sunit

=====
Sunit Singh
+91 9820148113

_______________________________
Do you Yahoo!?
Declare Yourself - Register online to vote today!
http://vote.yahoo.com

Comments

25 Oct 2004 - 12:16pm
dmitryn
2004

Jakob Nielsen has advocated spending 10% of a project's budget on usability:

http://www.useit.com/alertbox/20030825.html

It's not clear to me, however, whether this heuristic is meant to
refer to UX as a whole or just usability testing (my guess is the
latter). Also, I don't know of any evidence that it is being followed
in practice. :)

Cheers,

Dmitry

--
Dmitry Nekrasovski
http://www.cs.ubc.ca/~dmitry

25 Oct 2004 - 2:59pm
Listera
2004

Dmitry Nekrasovski:

> It's not clear to me, however, whether this heuristic is meant to
> refer to UX as a whole or just usability testing (my guess is the
> latter).

"Current best practices call for spending about 10% of a design project's
budget on usability."

Clearly, I missed the top-secret memo where these numbers were revealed. I
should really consult more often with those higher authorities who decide on
these 'best practices'.

For most users, the Interface (IA/UI/UX) *is* the app. If what drives the
design of that most important component should (by edict of those who decide
on allotments) get only 10%, God save us.

Sure sounds like a nice round figure to ask for if you're pimping seminars,
training days, etc., though.

Ziya
Nullius in Verba

25 Oct 2004 - 3:10pm
Coryndon Luxmoore
2004

Jakob pronounces:
"Current best practices call for spending about 10% of a design project's
budget on usability."

Ziya says:
"For most users, the Interface (IA/UI/UX) *is* the app. If what drives the
design of that most important component should (by edict of those who decide
on allotments) get only 10%, God save us."

I had always interpreted that number as being specific to the actual testing
effort rather than the design effort dedicated to the development of a
usable UI. But it is vague as to what he includes with the word "usability".

--Coryndon

25 Oct 2004 - 3:13pm
Listera
2004

Coryndon Luxmoore:

> I had always interpreted that number as being specific to the actual testing
> effort rather than the design effort dedicated to the development of a
> usable UI. But it is vague as to what he includes with the word "usability".

Well, OK, I'm allergic to the implied notion of 'design by testing.'

Ziya
Nullius in Verba

25 Oct 2004 - 3:14pm
dmitryn
2004

So do you know of a more plausible number (or a way to determine one)?

Dmitry

On Mon, 25 Oct 2004 15:59:57 -0400, Listera <listera at rcn.com> wrote:
> [Please voluntarily trim replies to include only relevant quoted material.]
>
> Dmitry Nekrasovski:
>
> > It's not clear to me, however, whether this heuristic is meant to
> > refer to UX as a whole or just usability testing (my guess is the
> > latter).
>
> "Current best practices call for spending about 10% of a design project's
> budget on usability."
>
> Clearly, I missed the top-secret memo where these numbers were revealed. I
> should really consult more often with those higher authorities who decide on
> these 'best practices'.
>
> For most users, the Interface (IA/UI/UX) *is* the app. If what drives the
> design of that most important component should (by edict of those who decide
> on allotments) get only 10%, God save us.
>
> Sure sounds like a nice round figure to ask for if you're pimping seminars,
> training days, etc., though.
>
> Ziya
> Nullius in Verba
>
>
>
>
> _______________________________________________
> Interaction Design Discussion List
> discuss at ixdg.org
> --
> to change your options (unsubscribe or set digest): http://discuss.ixdg.org/
> --
> Questions: lists at ixdg.org
> --
> Announcement Online List (discussion list members get announcements already)
> http://subscribe-announce.ixdg.org/
> --
> http://ixdg.org/
>

--
Dmitry Nekrasovski
http://www.cs.ubc.ca/~dmitry

25 Oct 2004 - 3:26pm
Listera
2004

Dmitry Nekrasovski:

> So do you know of a more plausible number (or a way to determine one)?

THERE'S NO SUCH NUMBER. (Sorry, I'm not shouting at you.) Each
project/problem/challenge may deserve a different one. That determines what
the number is not some arbitrary number everybody tries to conform to.

Ziya
Nullius in Verba

25 Oct 2004 - 4:01pm
dmitryn
2004

OK, I realize that the Nielsen number is either a guesstimate or
wishful thinking or both. The question still remains: if I am a
decision-maker and want to know _roughly_ how much of a project's
budget to spend on the UX component, how do I determine this?

And yes, I also realize that in the perfect world the whole project
involves user/client/customer interaction and therefore the UX
component is not just about usability testing. Still, there are
certain cost items that will be directly attributable to UX, hence the
question.

Dmitry

On Mon, 25 Oct 2004 16:26:20 -0400, Listera <listera at rcn.com> wrote:
> [Please voluntarily trim replies to include only relevant quoted material.]
>
> Dmitry Nekrasovski:
>
> > So do you know of a more plausible number (or a way to determine one)?
>
> THERE'S NO SUCH NUMBER. (Sorry, I'm not shouting at you.) Each
> project/problem/challenge may deserve a different one. That determines what
> the number is not some arbitrary number everybody tries to conform to.
>
>
>
> Ziya
> Nullius in Verba
>
> _______________________________________________
> Interaction Design Discussion List
> discuss at ixdg.org
> --
> to change your options (unsubscribe or set digest): http://discuss.ixdg.org/
> --
> Questions: lists at ixdg.org
> --
> Announcement Online List (discussion list members get announcements already)
> http://subscribe-announce.ixdg.org/
> --
> http://ixdg.org/
>

--
Dmitry Nekrasovski
http://www.cs.ubc.ca/~dmitry

25 Oct 2004 - 4:56pm
Michael Quibuyen
2003

I think any successful (attempt at) budget estimation can only be done
*after* you gauge scope/details. Which mix of UX tasks will you require?
Perhaps, you might require tons of field study hours? Frequent or less
frequent user testing or review? Who knows? A team's make-up (how many
developers? designers? what's their wage? etc.) will also limit how/where
you allocate resources/time, so in the real world, I think it makes no sense
to determine budget allocations unless *some* kind of analysis is done.

Mike

----- Original Message -----
From: "Dmitry Nekrasovski" <mail.dmitry at gmail.com>
Cc: "IxD" <discuss-interactiondesigners.com at lists.interactiondesigners.com>
Sent: Monday, October 25, 2004 2:01 PM
Subject: Re: [ID Discuss] Usability Budgets

> [Please voluntarily trim replies to include only relevant quoted
material.]
>
> OK, I realize that the Nielsen number is either a guesstimate or
> wishful thinking or both. The question still remains: if I am a
> decision-maker and want to know _roughly_ how much of a project's
> budget to spend on the UX component, how do I determine this?
>
> And yes, I also realize that in the perfect world the whole project
> involves user/client/customer interaction and therefore the UX
> component is not just about usability testing. Still, there are
> certain cost items that will be directly attributable to UX, hence the
> question.
>
> Dmitry

25 Oct 2004 - 5:16pm
Peter Bagnall
2003

On 25 Oct 2004, at 21:13, Listera wrote:
>> I had always interpreted that number as being specific to the actual
>> testing
>> effort rather than the design effort dedicated to the development of a
>> usable UI. But it is vague as to what he includes with the word
>> "usability".
>
> Well, OK, I'm allergic to the implied notion of 'design by testing.'

I'm similarly allergic, but testing does play a valuable role - it
should happen after design though! Back to elderly users... It's very
hard, when all your design experience is with younger users to be sure
how elders will respond. If you don't test then you're just going with
your own opinion, which hopefully isn't too far out, but how do you
know? In my case testing revealed that the elders we worked with had a
harder time understanding the interfaces (which were very simple by the
standards of desktop software) than we expected. As a consequence I've
learnt more about how to design for elders, which I will wrap into the
next piece of design I do.

One of the problems with design as I see it is that we don't close the
loop enough. That is we don't go back when something is deployed and
see how the design we came up with actually works in reality. If we
don't do this enough then how can we learn? We'll just repeat our
mistaken assumptions on the next design. Doing testing is not
sufficient in itself to achieve this, but it is a step in the right
direction. Clearly this is much more of a problem for consultants than
for in house designers, because the in house designers do get to hear
the problems that their design caused, but even then, I suspect deeper
analysis would be useful, rather than just tweaking based on
complaints. I assume some in house design groups already do this, but I
think it's still far from common.

In projects where you know the user population very well I'd suggest
that you can probably get away with less testing. In projects, such as
the one I'm working on where you are less familiar with the user
population more testing is needed. The more experienced you are, again,
you can probably get away with less testing.

With the elders I should add that I'd already done user interviews, and
created personae, but the problems we encountered weren't with the
goals of the users, but in the fine interaction detail. In this case
they were problems with paging controls. At such a low level, with a
more challenging user group, testing was probably the only way we'd
find those problems. Having said that we didn't seem to need many
people to refine the design to the point where it was working better.
These refinements weren't "add a ...", but rethinking the low level
design concept, having discovered the concepts that were found to be
challenging to the participants of the tests.

--Pete

----------------------------------------------------------
As long as people believe in absurdities they will continue
to commit atrocities.
Francois Marie Arouet (Voltaire), 1694 - 1778

Peter Bagnall - http://people.surfaceeffect.com/pete/

25 Oct 2004 - 5:17pm
Listera
2004

Dmitry Nekrasovski:

> OK, I realize that the Nielsen number is either a guesstimate or
> wishful thinking or both.

This is a terrific start. I mean that sincerely. Because I'll bet a million
dollars that sooner than later somebody will tell *you* across the
conference table that you all should be spending 10%. Why? Because (somebody
like) Nielsen said so. Now you're going to have to spend time arguing why
that's a bogus number unrelated to what you're *specifically* working on.
Waste of my time.

Counter question: how much should any *specific* person allocate to stocks
or bonds in financial planning? Silly question? I agree. :-)

> The question still remains: if I am a decision-maker and want to know
> _roughly_ how much of a project's budget to spend on the UX component, how do
> I determine this?

Fair question. But who are "you" in this case? This is absolutely critical.
This is why I go off about designers driving the design process. If the
"decision-maker" knows little about IA/UI/UX, he can rely on Nielsen to tell
him it's 10% and you'll have to meet that. Or he can ask his developers and
most likely get an even lower number. Or he can ask his designers. What a
concept! Actually asking people who have likely done similar work before and
are in a position to look at the *specific* requirements and come up with a
realistic number: it may be less or it may be more, but it sure will be heck
of a lot more *relevant*.

> Still, there are certain cost items that will be directly attributable to UX,
> hence the question.

It isn't complicated, Dmitry. Go through the processes/steps you'll need.
Then give them to the production manager; usually they'll be the ones to
cost out the days/personnel/dollars numbers. If there's no PM, then Excel or
a project management app should suffice.

Ziya
Nullius in Verba

25 Oct 2004 - 5:42pm
Listera
2004

Peter Bagnall:

> If you don't test then you're just going with your own opinion, which
> hopefully isn't too far out, but how do you know?

No, no, I don't mean at all that you shouldn't test while you're designing.
I was pejoratively referring to a school of design (or line of business, I
should really say) that throws up the design in the air (or at users) and
then pays for endless tests to clean up the mess. I get to do a lot of
IA/UI/UX surgery, so I see such nonsense too often.

The canonical example of this is design of security aspects in Microsoft
apps: release software > wait for customer complaints > release patch >
rinse & repeat. As opposed to forethought and architectural design to begin
with. That's 'design by testing,' in this case, testing on paying customers.

Incidentally, I think the best testing is done at high-volume sites where
you can do A/B tests on actual users, doing actual stuff under actual
conditions. A site like Amazon can figure out what works with any specific
segment of their userbase in a manner of hours, simply by sampling with
different designs.

Ziya
Nullius in Verba

25 Oct 2004 - 5:44pm
dmitryn
2004

> This is a terrific start. I mean that sincerely. Because I'll bet a million
> dollars that sooner than later somebody will tell *you* across the
> conference table that you all should be spending 10%.

This is precisely why I asked the original question. I am not the
"decision-maker" in the question - I am a relative newcomer to the UX
profession, and I asked the question to anticipate the day when a
client or boss will ask it of me. Still, if it is a newbie question, I
apologize for any waste of time that occurred in the process of people
replying to it. :)

> Counter question: how much should any *specific* person allocate to stocks
> or bonds in financial planning? Silly question? I agree. :-)

Touche.

> It isn't complicated, Dmitry. Go through the processes/steps you'll need.
> Then give them to the production manager; usually they'll be the ones to
> cost out the days/personnel/dollars numbers. If there's no PM, then Excel or
> a project management app should suffice.

Great answer. Thanks!

Dmitry

--
Dmitry Nekrasovski
http://www.cs.ubc.ca/~dmitry

25 Oct 2004 - 5:56pm
Listera
2004

Dmitry Nekrasovski:

> Still, if it is a newbie question, I apologize for any waste of time that
> occurred in the process of people replying to it. :)

Sorry, what exasperates me is not your question at all, it's these
self-serving, irresponsible and irrelevant numbers, benchmarks, best
practices, standards, and other...well, OK, I'm done venting for today.:-)

Ziya
Nullius in Verba

25 Oct 2004 - 6:06pm
Peter Bagnall
2003

On 25 Oct 2004, at 23:42, Listera wrote:
> Peter Bagnall:
>
>> If you don't test then you're just going with your own opinion, which
>> hopefully isn't too far out, but how do you know?
>
> No, no, I don't mean at all that you shouldn't test while you're
> designing.
> I was pejoratively referring to a school of design (or line of
> business, I
> should really say) that throws up the design in the air (or at users)
> and
> then pays for endless tests to clean up the mess. I get to do a lot of
> IA/UI/UX surgery, so I see such nonsense too often.

In that case we're entirely in agreement! You can't test your way to
good design. It should be applied as polish later, or as a way of
detecting problems (sometimes in very specific detail), but not as a
way of finding solutions. That remains the job of the designer. Every
design change should be defensible on the basis of the tests, and other
experience/theory, not just "lets try...".

--Pete

----------------------------------------------------------
Manners are a sensitive awareness of the feelings of others.
If you have that awareness, you have good manners, no matter
what fork you use.
- Emily Post

Peter Bagnall - http://people.surfaceeffect.com/pete/

26 Oct 2004 - 6:24am
Kevin Cheng
2004

It's not secret that I like to poke fun at Nielsen's number that
apparently fly out from some unknown dark region. This 10% figure
stank of the same. I have accused Nielsen of publishing numbers
without the methodology behind how he arrived at them. In the case of
Dmitry's question, I think Dmitry linked the follow-up article rather
than this original one:

http://www.useit.com/alertbox/20030107.html

In it, he states a range of the % and does admit to the number varying
based on size of project.

Now we know there are a ton more variables than size of project. I
agree completely with Ziya about how to give an estimate as well as
the fact that different situations call for different allocation.
Sometimes, you have to choose how much to do even if the optimal is
more.

For example, I can't always do on site testing if time and money limit
that in which case, I may opt for phone interviews. The budget is now
less. As professionals, we advise on what we feel are steps we should
take but we also work within real life constraints. Given that, it
becomes a valid question to say "well what's the optimal point?"

So IF I was in Nielsen's shoes, I might go through a ton of projects,
look at budget allocation and measure the difference on a number of
before/after metrics and see what comes up.

I don't know if Nielsen did that because often, his methodologies
baffle the logical but he has a 110 page paper that I haven't looked
at but he's linked. Just giving a slight benefit of the doubt here.

Kevin Cheng (KC)
OK/Cancel: Interface Your Fears
kc at ok-cancel.com
www.ok-cancel.com

-----Original Message-----
From:
discuss-interactiondesigners.com-bounces at lists.interactiondesigners.co
m
[mailto:discuss-interactiondesigners.com-bounces at lists.interactiondesi
gners.com] On Behalf Of Listera
Sent: Monday, October 25, 2004 11:56 PM
To: IxD
Subject: Re: [ID Discuss] Usability Budgets

[Please voluntarily trim replies to include only relevant quoted
material.]

Dmitry Nekrasovski:

> Still, if it is a newbie question, I apologize for any waste of time
that
> occurred in the process of people replying to it. :)

Sorry, what exasperates me is not your question at all, it's these
self-serving, irresponsible and irrelevant numbers, benchmarks, best
practices, standards, and other...well, OK, I'm done venting for
today.:-)

Ziya
Nullius in Verba

_______________________________________________
Interaction Design Discussion List
discuss at ixdg.org
--
to change your options (unsubscribe or set digest):
http://discuss.ixdg.org/
--
Questions: lists at ixdg.org
--
Announcement Online List (discussion list members get announcements
already)
http://subscribe-announce.ixdg.org/
--
http://ixdg.org/

---
Incoming mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.775 / Virus Database: 522 - Release Date: 10/8/2004

26 Oct 2004 - 8:12am
Petteri Hiisilä
2004

Sunit wrote:
> [Please voluntarily trim replies to include only relevant quoted material.]
>
> Hello there,
>
> I need some stats on the budgets allocated to the interaction design/
> usability process in a typical project (Online application/ Software).
> What percentage do the big guys spend on usability/Interaction Design/ in
> comparison to the cost of developing the entire product.
>
> Links to any such studies/ statistics would be really helpful.

Good question.

I don't have any statistical answers, but I have to take a look at some
old projects at some point. I have a hunch that the whole architect's
job takes about 25 percent of the budget and 50 percent of the time.

What you get for that money is blueprints, (flash/html) prototype and
requirements that leave no guesswork for the engineers and programmers.
There's only feasible, interesting technical problems left. When the
engineering plans are ready for programmers, there's only construction
and technical testing to be done.

If you add the engineering part too, it might take something like 50
percent of the budget and and 75 percent of the time. The actual
construction is actually very small, if you have great form & behavior
spec and well-engineered plans. In a half-year project it just takes one
and a half month to actually build it.

An example: I had a pretty huge webmail project. The version jumped from
2.0 to 3.0. The interaction framework was done until 5.0, the design
until 4.0 and the engineering and construction until 3.0. Us interaction
designers really keep the engineering an programming guys (and our
company) busy for the next two years or so :)

The stakeholders get a good view two years ahead. The engineers see,
what's to come next and can plan accordingly. Programmers don't have to
waste their effort, no code gets lost. Not bad?

The 50/75 percent contains EVERYTHING, including meetings, project
management etc. Only construction and technical testing is left.

But this is only a ballpark hunch, based on the webmail and some other
bigger projects. I haven't checked the facts.

Best,
Petteri

--
Petteri Hiisilä
Palveluarkkitehti / Interaction Designer /
Alma Media Interactive Oy / NWS /
+358505050123 / petteri.hiisila at almamedia.fi

"The unbroken spirit
Obscured and disquiet
Finds clearness this trial demands"
- Dream Theater

26 Oct 2004 - 12:14pm
Adlin, Tamara
2004

By the way, the new edition of Bias & Mayhew's book "Cost-justifying usability, 2nd Edition: An update for the information age" is on its way from Morgan Kaufmann (not sure when it's going to hit the shelves, but I know it's coming). This should have some really good info in it regarding process, arguments, and budgets for usability. The first edition is dated but quite informative.

Tamara

26 Oct 2004 - 3:21pm
Marko Hurst
2004

Sorry, so many of the emails seem to keep saying the same thing. I had to just sum it up…

1) Nielson - 10%. Why & how did he choose that number, it’s nice and round? Depending on the time, budget, client, methodology, etc. the number changes. That may be a nice high level to start w/ but until strategy, scope, & a timeline have been defined you can't give a true estimate. But the reality of many projects is at a minimum for forecast planning you need a number for budget estimates, so why not use 10%. If Nielson uses it must be right. Right????

2) There is no one price shopping for user testing. Different tests call for different pricing and skills, as well as the costs of participants. Employees (generally) cost less than outside users. Internal testing labs & test moderators cost less than outsourced ones, etc.

3) When in the development cycle you perform user testing helps determine what type of testing to perform as different tests are more appropriate or better suited at different times during development. I.e. one-on-one user testing as a final sanity check 4 weeks before launch is far more appropriate than conducting surveys or focus groups, (generally).

4) And of course anything I've written or someone else has written, you may have heard, etc. is all subject to change or become null & void due to budget, team skill-set, client's whim, or simply put... the reality of that particular project.

Nothing is definite in interactive development, so we can keep trying to educate others and continue to advocate our work.

What was the original question???? :)

Ciou,

Marko

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

26 Oct 2004 - 3:48pm
Cristian Cheran
2004

My perception about the 10% budget pronouncement is that it's a
heuristic / shot-from-the-hip figure than has a rather qualitative
semantic (rather than being a strict quantitative guideline) which
mainly says: "allocate a big amount of resources for usability
testing".

Couple this hint with his original tenet: "test often discounted (5
users) instead only once thoroughly (with 20 users)", and you'll see
that this becomes an easy-to-pass-around reinforcement of this mindset
(good meme design?). It's just like the other famous feat called
"Flash 99% Bad" - does anyone literally believe the "99%" part?

And, yes, my belief is that this "10% usability" talks only about
usability (testing, heuristic evaluations, etc) rather than the other
user experience activities like IxD or IA.

Cristian Cheran
----------------------
Interaction Designer
Cheran Software
www.cheranware.com

26 Oct 2004 - 3:53pm
Lyle_Kantrovich...
2004

Marko, thanks for summarizing.

Kevin, thanks for pointing to the original articles.

Here's the essence of Jakob's assertion of "best practice" for usability
budgets:

"To assess the total cost of usability (as opposed to the price of a
single test), we collected data from 863 design projects that included
usability activities. Depending on how we estimated it, usability costs
were between 8% and 13% of the projects' budget.

"Based on this finding and findings from other surveys, we conclude that
current best practices call for devoting about 10% of a project's budget
to usability.

"Our full survey data reveals a slightly complicated mathematical model
that relates project size to recommended usability spending. In essence,
the cost of usability doesn't increase linearly with project size, since
many usability activities cost about the same, regardless of how big the
project is. A project that's ten times bigger, for example, typically
requires only four times more usability spending."

I want to make a couple of points:

1. As a sharp consultant recently pointed out, there are no "best
practices", rather only "proven practices." This encompasses the "your
mileage may vary" aspect of real life. (He also said there are no real
"subject matter experts," but that's another topic.)

2. I think of Jakob's 10% number as a way to educate and raise
awareness. To me it's an average figure for what gets spent on
"usability" (evaluation) where it's being done today. It doesn't prove
that that's enough or the "right" amount in any way. So, this # can be
cited to those unenlightened who aren't doing usability evaluation at
all and to help them get comfortable with spending a realistic amount on
usability evaluation. I've run into project managers and product
managers, who had no sense of what "doing usability" would cost
them...they might come in thinking they could get by with 1%...and get
design advice along with evaluation...the 10% number is an external
figure that makes the start thinking in the "right ballpark."

3. The 10% number doesn't include design, at least not from what I can
tell. Of course, he's not real clear on what exactly they counted as
"usability activities." If you consider all possible UCD activities,
not just usability evaluation, then the percentage is clearly way off.

4. Good consultants (and analysts, project managers, etc.) don't just go
with some external "average" or benchmark. Rather, they ask good
questions and listen to what the project/product/business need, and then
craft a realistic approach.

5. Once business people get a sense of what UCD and usability can do for
them, then they really don't care about some external 10% number...they
then rely on a trusted partner (consultant, vendor, whatever) to
recommend the right activities.

Jakob is often trying to raise awareness of the "unwashed" who may
currently not even have usability on their radar. Once you're a skilled
practitioner, you're expected to understand the nuances and complexities
that Jakob can't get muddy his usability promotion pieces with. (Notice
that he mentions a "slightly complicated mathematical model" in his
article. Keep in mind, models aren't the real world either.)

Jakob helps raise people's awareness of UCD and that's good, but I'll
never rationalize a project estimate entirely based on some number in an
Alertbox article. Anyone who does is clearly not very experienced in
the field.

Note: Jakob and his Nielsen Norman Group are not unique from a "research
group" perspective. Forrester, Gartner, AMR and others all offer
similar proscriptions for IT and business all the time. They offer
benchmarks and "best practice" advice that should be used as an input to
one's decision making process, but that also shouldn't be taken as
"gospel."

Regards,

Lyle

----
Lyle Kantrovich
User Experience Architect
Cargill
http://www.cargill.com/
Voice: 952-984-5330

Croc O' Lyle - Personal Commentary on usability, information
architecture and design.
http://crocolyle.blogspot.com/

"Simplicity is the ultimate sophistication."
- Leonardo da Vinci

26 Oct 2004 - 6:05pm
Listera
2004

Marko Hurst:

> so why not use 10%

Because it's lower than 20% and may be far from reality?

Ziya
Nullius in Verba

26 Oct 2004 - 6:57pm
Listera
2004

> "Our full survey data reveals a slightly complicated mathematical model
> that relates project size to recommended usability spending. In essence,
> the cost of usability doesn't increase linearly with project size, since
> many usability activities cost about the same, regardless of how big the
> project is. A project that's ten times bigger, for example, typically
> requires only four times more usability spending."

Is it possible to design an highly effective app without engaging in
'usability activities'? Yes.

Is it possible to conduct 'usability activities' and still have a failing
design? Yes.

Is there an inviolable correlation between 'usability activities' and
effective design? No.

What the heck is 'usability activities'?

Walks like 'design by testing' and talks like 'design by testing' and sounds
like sweet melody to the testing/usability industry.

Usability shouldn't be seen as a 10% tax on post-design, it should be an
*integral* part of design from minute one.

----
Ziya

Cui bono?

26 Oct 2004 - 8:27pm
Marko Hurst
2004

Ziya wrote...

Because it's lower than 20% and may be far from reality?
********

Perhaps the 4th point "will vary by client, budget, team, etc." eluded others as well?

In the current Walgreens campaign commercials there is a land called "Perfect", perhaps there we can find the 'perfect' solution, but as they also say "we don't live anywhere near Perfect".

The chicken or the egg? Which came first? Knowing the cost breakout of each task or estimating the budget for each? Which comes first? At a projects inception before definition, scope, or even before a project plan has been started companies generally require "estimates".

The reality of a project (assuming you're not your own client), is that there are many other factors such as timeline, budget, profits, etc that must all be considered and will also be managed by others. So until the specifics emerge your best estimate is as good as it gets and that is where experience comes into play. Of course, if someone has been holding out on the "magic" formula or the "4-step process" that works every time, I have a time-share in Miami, maybe we could work something out.

Ciou,

Marko

---------------------------------
Do you Yahoo!?
Take Yahoo! Mail with you! Get it on your mobile phone.

26 Oct 2004 - 8:54pm
Listera
2004

Marko Hurst:

> So until the specifics emerge your best estimate is as good as it gets and
> that is where experience comes into play.

That ( experience ) is precisely the point.

You can throw up numbers in the air like the 10% and seek comfort in the
perfectness of your fantasy. Or you can get a designer who has experience in
doing the work projected and a production manager who has experience in
costing out projections. This ain't rocket science by a long shot.

You say you don't have an experienced designer and/or project manager?
Believe me, the 10% variance on 'usability activities' will be the least of
your potential problems on that project!

> At a projects inception before definition, scope, or even before a project
> plan has been started companies generally require "estimates".

Counter question: Can you "estimate" how much it costs to buy a house?

I'm sorry, but I have no sympathy for those who insist on an estimate
"before definition, scope, or even before a project plan has been started."
That doesn't define 'reality' it defines formulaic, idiotic, amateurish
business practices that we should learn not to tolerate.

Ziya
Nullius in Verba

27 Oct 2004 - 1:07am
Lyle_Kantrovich...
2004

Ziya asked:
> Is it possible to design an highly effective app without engaging in
> 'usability activities'? Yes.
>
> Is it possible to conduct 'usability activities' and still have a
failing
> design? Yes.
>
> Is there an inviolable correlation between 'usability activities' and
> effective design? No.
>
> What the heck is 'usability activities'?

First off, I'm confused about how you can answer questions one through
three without knowing what 'usability activities' are. You try to use
logic, but then show that the logic is built on a lack of understanding
the basic concepts being discussed. I'd guess you're making some kind
of assumptions about what 'usability activities' are in your earlier
questions, but you don't make that clear.

Secondly, let's try some parallel questions/logic based on your logic
above:

Is it possible to build a highly effective or bug-free app without
engaging in unit, system or load testing of the code? Yes.

Is it possible to conduct unit, system and load testing of the app and
still have a failing product? Yes.

Is there an inviolable correlation between system testing and effective
design? No.

Important considerations:
1. "Possible" and "likely" are two different things. It's possible
there's some designer out there who's infallible, omniscient, and
possesses other god-like qualities -- I just haven't met her yet.
Testing has value.
2. Flip it around. Ask instead questions like "Is the
system/product/design likely to be better if it's usability tested/load
tested?"
3. To my knowledge, no one has ever said that running a usability test
was going to guarantee success. I can tell you that many apps have
failed for reasons that might have been discovered if usability tests
were run.

Ziya said:
> Walks like 'design by testing' and talks like 'design by testing' and
>
> sounds like sweet melody to the testing/usability industry.
>
> Usability shouldn't be seen as a 10% tax on post-design, it should be
an
> *integral* part of design from minute one.

Not sure what lines you're reading between, but you're not reading the
message accurately. I've been to a number of CHI and UPA conferences,
and found that usability folks who know what they are talking about will
quickly tell you testing *informs* design (as do user research and other
UCD activities). Testing also *evaluates* design (as do heuristic evals
and cognitive walkthroughs, etc.) -- Testing is NOT Design.

I'd agree that testing shouldn't be a 10% add-on after the fact, but
integral to a process towards good design. Good usability practitioners
always want users to be involved earlier in the design process...hence
"integral" is a fitting word.

Too many people think "I don't need to test. I'm a *good* designer" ...
proving themselves wrong in the process.

Regards,

Lyle

----
Lyle Kantrovich
User Experience Architect
Cargill
http://www.cargill.com/
Voice: 952-984-5330

Croc O' Lyle - Personal Commentary on usability, information
architecture and design.
http://crocolyle.blogspot.com/

"Simplicity is the ultimate sophistication."
- Leonardo da Vinci

27 Oct 2004 - 1:09am
Lyle_Kantrovich...
2004

Ziya asked:
> Counter question: Can you "estimate" how much it costs to buy a house?

Yes. For most of the US, I'd be pretty safe estimating somewhere
between $100,000 and $1,000,000 (USD). Estimates are just
that...estimates. Now if you tell me you want to know *exactly* what it
will cost, we need to talk a lot more. If you aren't sure what kind of
house you want or need, I might help you by telling you what an average
house costs in the state you're looking to buy in. If I give you an
average number and tell you that's exactly what your house will cost,
then I'm just an idiot...and you'd be an idiot if you believed me.

There are different types of estimates with different expectations and
uses: high-level budgeting estimates are one example, "most likely
scenario" estimates are another, and a fixed bid "estimate" is an
attempt to tell you exactly what it will cost. We *must* be real clear
with people about what kind of information we need to deliver the kind
of estimate they need or are seeking.

Ziya said:
> I'm sorry, but I have no sympathy for those who insist on an estimate
> "before definition, scope, or even before a project plan has been
> started." That doesn't define 'reality' it defines formulaic, idiotic,

> amateurish business practices that we should learn not to tolerate.

Evidently you think the real world I live in doesn't involve 'reality'
then. My customers ask me, around December of each year, questions like
"how much should I budget for your user experience services next fiscal
year?" Telling them I need to know the exact scope and projects isn't
realistic. If I help them arrive at a budget number, they will likely
have budget for some of my services. If I don't, they won't. Do you
think helping them arrive at a number to put into their budget is a
"formulaic, idiotic, amateurish business practice"?

Definition, scope and project plans are all best guesses at the
beginning...and most decent sized projects have some sort of scope
change as they progress. Of course to create *any* type of estimate
there needs to be at least some level of definition, or assumptions
made.

Ever try starting your own business? If you go to the bank for a loan,
they'll ask you questions like "how much money will your new company
make next year?" I wouldn't characterize that as "amateurish" -- it's a
realistic question that doesn't come with easy answers.

Ambiguity and risks are realities in business. One must learn to
tolerate them.

Regards,

Lyle

----
Lyle Kantrovich
User Experience Architect
Cargill
http://www.cargill.com/

Croc O' Lyle - Personal Commentary on usability, information
architecture and design.
http://crocolyle.blogspot.com/

"Simplicity is the ultimate sophistication."
- Leonardo da Vinci

27 Oct 2004 - 2:14am
Listera
2004

Lyle_Kantrovich at cargill.com:

> I'd agree that testing shouldn't be a 10% add-on after the fact, but
> integral to a process towards good design. Good usability practitioners
> always want users to be involved earlier in the design process...hence
> "integral" is a fitting word.

Again, thanks for your agreement with my central argument.

> Too many people think "I don't need to test. I'm a *good* designer" ...
> proving themselves wrong in the process.

I'm not sure how many times I need to repeat this:

<http://listserver.dreamhost.com/pipermail/discuss-interactiondesigners.com/
2004-October/003199.html>

But please don't try to link me to silly straw man stuff: I'M NOT AGAINST
TESTING. I'm against 'design by testing'.

Ziya
Nullius in Verba

27 Oct 2004 - 2:17am
Listera
2004

Lyle_Kantrovich at cargill.com:

> Ziya asked:
>> Counter question: Can you "estimate" how much it costs to buy a house?
> Yes.

Really?

> For most of the US, I'd be pretty safe estimating somewhere
> between $100,000 and $1,000,000 (USD). Estimates are just
> that...estimates.

Thanks for making my point: that the 10% without context, bereft of a
relation to specific project(s). is meaningless.

If your boss asks you for an "estimate" and you say $100K-$1M he'll laugh at
you, because he can't work with a 10X spread. If you then pick a number, any
number between $100K-$1M because you have no basis in fact ("before
definition, scope, or even before a project plan has been started") I'm not
sure who you're deceiving, him or yourself. Why don't you just throw a dart?

> There are different types of estimates with different expectations and
> uses: high-level budgeting estimates are one example, "most likely
> scenario" estimates are another, and a fixed bid "estimate" is an
> attempt to tell you exactly what it will cost. We *must* be real clear
> with people about what kind of information we need to deliver the kind
> of estimate they need or are seeking.

And you're telling me that throwing up the number 10% accomplishes this?

> If I help them arrive at a budget number,

Let's parse what you're actually saying: you'll "arrive at a budget number"
how? You have no specifics, requirements, plans, actual/potential problems.
You have nothing at all and you'll "arrive at a budget number"? Yes, this is
absolutely "formulaic, idiotic, amateurish business practice."

It's formulaic because you're using something like the 10% or some other
imaginary number, a formula, that in all likelihood doesn't reflect the
reality of *your* business.

It's idiotic because you'll have to constrain your design activities to an
imaginary number regardless of the actual circumstances, your clients will
thus get design services incommensurate with their actual needs and users
will suffer to boot.

It's amateurish because, in 2004, there are better business practices than
the unbelievably bureaucratic, Soviet-era type of dumb asset allocation.

> Of course to create *any* type of estimate there needs to be at least some
> level of definition, or assumptions made.

Thank you. You're going to have to start to figure out on whose side you're
arguing.:-)

> Ever try starting your own business?

Yes, I have risked several hundred thousand dollars of my own and partners'
money, founded a design technology company, ran it for a number of years
(profitably each year) and sold it to a British company, before the dotcom
era. I have started businesses for others too. Have you?

> If you go to the bank for a loan, they'll ask you questions like "how much
> money will your new company make next year?" I wouldn't characterize that as
> "amateurish" -- it's a realistic question that doesn't come with easy answers.

And if you say to your loan officer, "You know, I have no idea what my
company will make next year, but this Nielsen guy says that companies make
$XXX in this industry, so would you take that as a generic answer?" He'll
laugh at you.

> Ambiguity and risks are realities in business. One must learn to
> tolerate them.

You manage risks not by parroting some arbitrary figure you read on a web
site, but by actually consulting professional designers and PMs who have
experience in scoping and costing out projects, which is precisely what I
advocated.

Ziya
Nullius in Verba

27 Oct 2004 - 8:18am
Lyle_Kantrovich...
2004

Ziya said:
> Thanks for making my point: that the 10% without context, bereft of a
> relation to specific project(s). is meaningless.

No, not meaningless, just a data point...if you understand what the 10%
represents. If I told you that surveys show the average American
household spends 30% of their income on housing costs, would that number
be "meaningless"? I don't think so. The number isn't meaningless.
What you do with that number can be rational or irrational.

> If your boss asks you for an "estimate" and you say $100K-$1M he'll
laugh > at you, because he can't work with a 10X spread.

That's what you think. Sometimes they can work with those numbers.
Just the fact that I give him a 10X spread initially tells him we'd need
to know more things to get the range tighter (if he needs it tighter).
On the other hand, if I told him it was between $101,000 and $101,550
then he'll understand there's much more predictability around this
particular activity's cost. If he needs a more exact number, then we
need to get better information into the estimating process.

> I'm not sure who you're deceiving, him or yourself.

That's part of my point: we shouldn't deceive people when providing an
estimate. If you give any number, or a range of numbers, you have to
explain that number or the range, and people need to understand it. Too
often, I've seen people try to create an exact estimate, when all the
client wanted was a range to put into their budgeting process.

Clearly, you are taking the 10% number from Nielsen out of context. Or
at least implying that people will use it out of context. Who's
deceiving people? You, or Nielsen?

> You have nothing at all and you'll "arrive at a budget number"?

No...of course I have to have something to base my high level estimate
on, but it might not be much. In my experience, if a client tells you
"I don't know", you have to start making assumptions in order to give
any kind of estimate. Those assumptions that are then baked into the
estimate have to be explained to the client, or they will misuse the
number.

> It's formulaic because you're using something like the 10% or some
other
> imaginary number, a formula, that in all likelihood doesn't reflect
the
> reality of *your* business.
>
> It's idiotic because you'll have to constrain your design activities
to an
> imaginary number regardless of the actual circumstances, your clients
will
> thus get design services incommensurate with their actual needs and
users
> will suffer to boot.

A) I don't personally use the 10% figure to come up with my estimate.
*I* use it to help customers understand *the average* spent by other
companies *in general* on *these kinds of activities*. That's all that
10% figure is good for.

B) If they don't have much other knowledge to work with, junior UX folks
*might* use the 10% figure to create estimates for their first efforts
(hopefully on lower-risk projects). Experience will tell them over time
how well that figure maps to their business situation. More savvy folks
would also pull in other data points to check their work. For example,
they might review their UX approach and cost estimates with a more
senior practitioner.

C) If I *were to* tell a client that budgeting 10% of their budget for
usability was a good way to go, I feel comfortable that I could do
*something* for them that would add value. 15% might allow me to add
more value, but 10% is better than 5% for starters.

D) Budget numbers are put together many months in advance. Once a
project actually materializes, a project plan and estimate are created.
The client will then have to determine if they have enough budget. They
may need to seek additional funding or shift budget dollars to meet
their priorities. It's important to understand that their budget number
isn't my project estimate. Clients that value my services proactively
seek my input when *requesting* budgetary allocations from the company -
the least I can do is help them come up with inputs to their budget,
since it will impact me later. (Note, I work in a company with 95
businesses, which may work very differently from yours budgetarily.)

> I have started businesses for others too. Have you?

Yep, started my own, and have helped start businesses for others.

> And if you say to your loan officer, "You know, I have no idea what my
> company will make next year, but this Nielsen guy says that companies
make
> $XXX in this industry, so would you take that as a generic answer?"
He'll
> laugh at you.

You're right. BUT, if you say here's how we came up with *our number*,
and, just to show you our number is realistic, here's a benchmark. "A
recent study shows that the average company spends 10% on this type of
activity." By showing him this average figure, he'll have more
confidence in your number, which is at best a "guestimate" when you're
talking pro forma financials.

I think we're actually more in agreement than disagreement on how to
estimate. The big difference is, you claim Nielsen's number is
worthless. I say it's useful *if* you use it responsibly. It's only a
single data point, and it shouldn't be taken out of context.

Regards,

Lyle

----
Lyle Kantrovich
User Experience Architect
Cargill
http://www.cargill.com/
Voice: 952-984-5330

Croc O' Lyle - Personal Commentary on usability, information
architecture and design.
http://crocolyle.blogspot.com/

"Simplicity is the ultimate sophistication."
- Leonardo da Vinci

Syndicate content Get the feed