Incentives for UI testing

30 Sep 2009 - 2:40am
4 years ago
9 replies
1075 reads
Angela Arnold
2009

I am currently looking into different types of incentives for
participants who take part in our UI testing sessions, and would like
to know about other UX types' experiences and recommendations...

Does the type of incentive affect the quality and/or reliability of
the participants?

interested in your thoughts on this one...

Comments

30 Sep 2009 - 3:05am
Harry Brignull
2004

I usually just pay cash, between £35-100 for 60 minutes depending on the
user type (£100 only for special cases).
I suspect that anything other than cold hard cash is going to involve a lot
of no-shows ... Unless you are dealing with dedicated community members who
are looking forward to the opportunity of getting their voices heard.

Harry

30 Sep 2009 - 3:41am
dszuc
2005

We generally pay "cash" as it seems to work best in our market and
for the domains we have worked in.

Also, again depending on who you are recruiting, cash is a nice
substitute for a time in the participant's day that you may be
taking away from their family, friends, business etc

Not really seen the incentive impact the quality of the
participant's feedback rather if people are passionate and
interested in the domain it usually results in useful insights.

rgds,
Dan

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46204

30 Sep 2009 - 5:15am
Brian Hoadley
2009

Cash is the best attractor in my experience. I only find things like
gift certificates, etc work when you are testing on location
someplace and grabbing people and asking them to test for you on the
spot.

Apart from attracting people the other issue is whether they show up
or not. It's easy to shrug off £20 or £30 because you're busy at
work, tired, etc. It's more difficult to shrug off £50. In my
experience £50 is about right for a standard, easy to find user. For
the more specific and complex user, e.g. Petroleum engineer looking
for work in the North Sea, you will pay up to £100-£150 per user.

Hope that helps.
Brian

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46204

30 Sep 2009 - 9:46am
Dana Chisnell
2008

Angela,

You should always give participants *something* to thank them for
helping you evaluate a design. There are a few of ways to look at this
in recruiting:

Incentives -- something to motivate a person to take part in your study
Compensation -- a payment in exchange for time and/or services
Honoraria -- payment in recognition for acts or services for which a
price typically isn't set

Whatever, the point is to be as generous as you possibly can be,
because the people whose time you're taking are doing you a favor. You
can't get good data about how users perform with a design without the
users.

What the incentive/compensation/ honoraria is *does* influence whether
people answer your call for participants. It does not affect whether
they show up or how good the data is that they give you when they're
in the usability study (or other research). Data is good if the
participants are appropriate.

What determines whether participants show up are two things: 1) Are
they motivated to do what you want to observe in the study -- is it a
typical behavior that they already engage in? and 2) How much of a
relationship you've established with them already. If you treat
participants like lab rats or data points, they're not invested in
showing up. If you treat them like important partners, real people who
you're interested in learning about, they will show up.

As for what the incentive should be:

Start with a simple "thank you." A hand-written thank-you note goes a
long way.

Generally, cash works great, for nearly every level of study or
participant. If it's a small test, gift cards for coffee or a local
lunch place can work well. Larger gift certificates for shopping
opportunities like Amazon work really well in situations when you're
asking more of participants.

High school and college students were completely turned off by gift
cards to a specific book store.

I'll let Jared Spool talk about paying people in compelled shopping
usability tests.

If you have participants inside your company, you can give away schwag
like sweatshirts or mugs or calendars, flashlights, pens, etc. This
can work with some types of customers, too.

You can offer instead things like subscriptions or versions of your
software or web site that would normally cost something substantial.

Otherwise, typically government employees are restricted from taking
gifts like incentives. There may be restrictions on compensation for
people in some roles that are heavily regulated or have intense
ethical scrutiny. Wealthy people don't care about cash; likewise some
people in executive management or high-level professionals don't want
the money. In those cases, we offer to make a donation to a favorite
charity.

In short: No, the type of incentive does not affect the quality or
reliability of the participants. What affects the quality and
reliability of the participants is the interviewing/selection process.

Good luck!
Dana

:: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: ::
Dana Chisnell
415.519.1148

dana AT usabilityworks DOT net

www.usabilityworks.net
http://usabilitytestinghowto.blogspot.com/

30 Sep 2009 - 12:53pm
Paul Bryan
2008

We usually pay about $100 per hour for e-commerce user research. It is
often for a retailer, and in that case we use gift cards about 50% of
the time and cash 50% of the time. For employee research, most
companies opt to not provide any incentives, although we try to give
some kind of swag.

My experience has been that the monetary amount of incentives does
impact behavior. This effect has been written about in the medical
literature in terms of conflict of interest in clinical trials.

When we have offered higher incentives, it seemed as though a larger
proportion of the participants tried to "earn" the incentive by
giving us the kinds of answers they thought we wanted. We also seemed
to get a higher percentage of professional research participants who
go from test to test. This obviously impacts the validity and
reliability of the data.

When we've assigned homework, such as a diary, to go along with the
interview, specifying an amount for completing the homework seemed to
result in higher completion rates and more content per entry.

When we've tried lower incentives, we've predictably experienced
more last minute cancellations, especially around rush hour in larger
cities or in bad weather.

Paul Bryan
Usography (http://www.usography.com)
Linked In: http://www.linkedin.com/in/uxexperts

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46204

1 Oct 2009 - 10:56am
Will Hacker
2009

I pay cash, usually $100 for an hour of testing. On the recording
permission slip the participant signs before the test starts it
explicitly says their compensation is unrelated to their comments.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46204

1 Oct 2009 - 2:31pm
Angel Anderson
2010

In the past we've given cards to Amazon, Starbucks, iTunes, etc. in
the amount of anything between 5 to 50 dollars depending on the time
commitment. Steven has a good point about high amounts being off-putting,
but I believe this is particularly true with field intercepts and blind
recruiting. When going through a firm with a registered network of available
participants, those folks usually expect higher compensation.
Kind regards,

Angel Anderson
Senior Interaction Designer
HUGE
----------------------------------
IxDA Los Angeles
----------------------------------
Email: angel.j.anderson at gmail.com
Twitter: AngelAnderson
Skype: AngelJAnderson

On Wed, Sep 30, 2009 at 2:19 AM, Steven Diebold <steven at stevendiebold.com>wrote:

> there is an interesting correlation to the amount of money required to
> engage participants and when they think its too much and it puts them
> off.
>
> They did a test recently for people filling out surveys and found
> $5-$10 gift cards to particular places were the most motivating to
> get people to respond. When they moved it up to $50-$250 people were
> less likely to respond because they thought it was too good to be
> true.
>
> If you are recruiting you should be careful about the incentive and
> test them. When they ran these tests they found the promotion that
> offered $250 actually received lower response rates than the $5-$10
> gift card to Lowes hardware store. They actually got triple the
> response rates to lower amount of gift and it cost them less.
>
> Always test your incentives and don't ask people in forums what
> works for them unless you want them to take your test. Everyone is
> different and is motivated by different things. This is why its
> important to test incentives and not to rely on cash.
>
> If you want proof of this test visit Marketing Sherpa B2B marketing
> summit. Design reviews are surveys and incentives to review anything
> requires testing.
>
> stevendiebold.com
>
>
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Posted from the new ixda.org
> http://www.ixda.org/discuss?post=46204
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

--

1 Oct 2009 - 2:50pm
Todd Warfel
2003

Depending on the audience, we've used:
* $50 Amex gift cards for remote attendees
* $100-150 Amex gift cards/cash for in-person attendees
* iPod nanos
* Free copies of software

Cheers!

Todd Zaki Warfel
Principal Design Researcher
Messagefirst | Designing Information. Beautifully.
----------------------------------
Contact Info
Voice: (215) 825-7423
Email: todd at messagefirst.com
AIM: twarfel at mac.com
Blog: http://toddwarfel.com
Twitter: zakiwarfel
----------------------------------
In theory, theory and practice are the same.
In practice, they are not.

1 Oct 2009 - 4:15pm
Steven Diebold
2009

I totally agree. My reference was exactly related to blind recruiting
and field intercepts with low level involvement surveys. The longer
the survey and more in depth time required then people do expect more
money for sure.

thanks for clarifying that point. :)

thanks
steven

On Oct 1, 2009, at 12:31 PM, Angel Anderson wrote:

> In the past we've given cards to Amazon, Starbucks, iTunes, etc. in
> the amount of anything between 5 to 50 dollars depending on the time
> commitment. Steven has a good point about high amounts being off-
> putting, but I believe this is particularly true with field
> intercepts and blind recruiting. When going through a firm with a
> registered network of available participants, those folks usually
> expect higher compensation.
>
> Kind regards,
>
> Angel Anderson
> Senior Interaction Designer
> HUGE
> ----------------------------------
> IxDA Los Angeles
> ----------------------------------
> Email: angel.j.anderson at gmail.com
> Twitter: AngelAnderson
> Skype: AngelJAnderson
>
>
> On Wed, Sep 30, 2009 at 2:19 AM, Steven Diebold <steven at stevendiebold.com
> > wrote:
> there is an interesting correlation to the amount of money required to
> engage participants and when they think its too much and it puts them
> off.
>
> They did a test recently for people filling out surveys and found
> $5-$10 gift cards to particular places were the most motivating to
> get people to respond. When they moved it up to $50-$250 people were
> less likely to respond because they thought it was too good to be
> true.
>
> If you are recruiting you should be careful about the incentive and
> test them. When they ran these tests they found the promotion that
> offered $250 actually received lower response rates than the $5-$10
> gift card to Lowes hardware store. They actually got triple the
> response rates to lower amount of gift and it cost them less.
>
> Always test your incentives and don't ask people in forums what
> works for them unless you want them to take your test. Everyone is
> different and is motivated by different things. This is why its
> important to test incentives and not to rely on cash.
>
> If you want proof of this test visit Marketing Sherpa B2B marketing
> summit. Design reviews are surveys and incentives to review anything
> requires testing.
>
> stevendiebold.com
>
>
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Posted from the new ixda.org
> http://www.ixda.org/discuss?post=46204
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>
>
>
> --
>

Syndicate content Get the feed