Need advice on an unorthodox methodology

10 Jun 2011 - 1:55am
2 years ago
10 replies
1125 reads
Zelbinian
2010

So, I'm doing a brief contract for a rather large, multinational software company located near Seattle (ahem) and I've been asked to critique a methodology I've never come across before. Clearly it's been put together on the fly a la MacGyver invention, and I'm not entirely sure it'll get the results they think it will, so I figured I'd ask more knowledgeable folk. The methodology is something of a cross between a focus group (ugh) and a feature presentation. They're calling it a Concept Verification Test, and the sessions will go something like this:

  • Groups of 3-4
  • Brief demonstration of current product to establish context
  • Presentation of several feature changes/new features
  • Passing out handouts to the participants and having them stack-rank the presented features (w/ room for suggesting others)
It doesn't seem that bad, but I'm not quite experienced enough to forsee how valid the results from such a thing might be. Help is appreciated - and please cite sources when possible!

Comments

10 Jun 2011 - 4:05am
William Hudson
2009

People react differently when you ask them to look at something as opposed to using it. It depends a lot on the context and the experience of the participants. If they've used earlier versions of the product extensively, you might get something useful out of just showing them the changes/new features. For inexperienced users or a new product, I don't think the method you describe is going to produce much in the way of useful results.

Regards,

William Hudson Syntagm Ltd User Experience Strategist UK 01235-522859 World +44-1235-522859 US Toll Free 1-866-SYNTAGM mailto:william.hudson@syntagm.co.uk http://www.syntagm.co.uk skype:williamhudsonskype twitter:SyntagmUCD

Syntagm is a limited company registered in England and Wales (1985). Registered number: 1895345. Registered office: 10 Oxford Road, Abingdon OX14 2DS.

UX and UCD courses in London, Brussels and Vancouver www.syntagm.co.uk/design/schedule.shtml

-----Original Message----- From: ixdaor@host.ixda.org [mailto:ixdaor@host.ixda.org] On Behalf Of Zelbinian Sent: 10 June 2011 08:25 To: William Hudson Subject: [IxDA] Need advice on an unorthodox methodology

So, I'm doing a brief contract for a rather large, multinational software company located near Seattle (ahem) and I've been asked to critique a methodology I've never come across before. Clearly it's been put together on the fly a la MacGyver invention, and I'm not entirely sure it'll get the results they think it will, so I figured I'd ask more knowledgeable folk. The methodology is something of a cross between a focus group (ugh) and a feature presentation. They're calling it a Concept Verification Test, and the sessions will go something like this:

  • Groups of 3-4
  • Brief demonstration of current product to establish context
  • Presentation of several feature changes/new features
  • Passing out handouts to the participants and having them stack-rank the presented features (w/ room for suggesting others)

It doesn't seem that bad, but I'm not quite experienced enough to forsee how valid the results from such a thing might be. Help is appreciated - and please cite sources when possible!

10 Jun 2011 - 8:05am
Alan James Salmoni
2008

I'd have to second that view although my inference was that the 'users' are experienced in the product pre-test (perhaps an inference too far?)

Having said that, there's nothing wrong with new research methods per se: a lot of work in UX is hard to consider as valid never mind reliable. It sounds academic but why measure something that doesn't measure what you think it measures? I'm guessing that they haven't aimed for validity analysis (and if so, best of luck to them)

Alan

On Fri, Jun 10, 2011 at 11:56 AM, William Hudson wrote: > People react differently when you ask them to look at something as opposed > to using it. It depends a lot on the context and the experience of the > participants. If they've used earlier versions of the product extensively, > you might get something useful out of just showing them the changes/new > features. For inexperienced users or a new product, I don't think the method > you describe is going to produce much in the way of useful results. > > Regards, > > William Hudson > Syntagm Ltd > User Experience Strategist > UK 01235-522859 > World +44-1235-522859 > US Toll Free 1-866-SYNTAGM > mailto:william.hudson@syntagm.co.uk > http://www.syntagm.co.uk > skype:williamhudsonskype > twitter:SyntagmUCD > > Syntagm is a limited company registered in England and Wales (1985). > Registered number: 1895345. Registered office: 10 Oxford Road, Abingdon OX14 > 2DS. > > UX and UCD courses in London, Brussels and Vancouver > www.syntagm.co.uk/design/schedule.shtml > > -----Original Message----- > From: ixdaor@host.ixda.org [mailto:ixdaor@host.ixda.org] On Behalf Of > Zelbinian > Sent: 10 June 2011 08:25 > To: William Hudson > Subject: [IxDA] Need advice on an unorthodox methodology > > So, I'm doing a brief contract for a rather large, multinational software > company located near Seattle (ahem) and I've been asked to critique a > methodology I've never come across before. Clearly it's been put together on > the fly a la MacGyver invention, and I'm not entirely sure it'll get the > results they think it will, so I figured I'd ask more knowledgeable folk. > The methodology is something of a cross between a focus group (ugh) and a > feature presentation. They're calling it a Concept Verification Test, and > the sessions will go something like this: > > * Groups of 3-4 > * Brief demonstration of current product to establish context > * Presentation of several feature changes/new features > * Passing out handouts to the participants and having them stack-rank the >  presented features (w/ room for suggesting others) > > It doesn't seem that bad, but I'm not quite experienced enough to forsee how > valid the results from such a thing might be. Help is appreciated - and > please cite sources when possible! > > (

10 Jun 2011 - 4:05am
Max L. Wilson
2009

its interesting, sounds a little like a participatory review, with a card-sort in it. almost a participatory design session, except its mostly talking through existing features rather than conceiving new ones? are these new features demoed as fully implemented or lofi-mockups of possible options?

does it get anywhere near a cognitive walkthrough (although youd usually do that with experts rather than end users) are they stepping through the interaction as a group?

max

n - Dr Max L Wilson e - m.l.wilson@swansea.ac.uk w - www.cs.swan.ac.uk/~csmax/

t - +44 (0) 1792 602611

On 10 Jun 2011, at 08:27, Zelbinian wrote:

> So, I'm doing a brief contract for a rather large, multinational software company located near Seattle (ahem) and I've been asked to critique a methodology I've never come across before. Clearly it's been put together on the fly a la MacGyver invention, and I'm not entirely sure it'll get the results they think it will, so I figured I'd ask more knowledgeable folk. The methodology is something of a cross between a focus group (ugh) and a feature presentation. They're calling it a Concept Verification Test, and the sessions will go something like this: > > * Groups of 3-4 > * Brief demonstration of current product to establish context > * Presentation of several feature changes/new features > * Passing out handouts to the participants and having them stack-rank the > presented features (w/ room for suggesting others) > > It doesn't seem that bad, but I'm not quite experienced enough to forsee how valid the results from such a thing might be. Help is appreciated - and please cite sources when possible! > >

10 Jun 2011 - 11:21am
Caroline Jarrett
2007

+1 for what William Hudson said.

The minute I read your description I thought "Ah ha, consultation by presentation!"

I've seen it done quite a lot before, and it's not a good idea.

I'm going to explain why by telling you a long story about Susan, a participant. I have changed some names and details, but believe me when I say I've seen version of this story many, many times.

The story opens.

Susan gets an invitation to the presentation about the new version of Product A. She uses A every day and doesn't really notice much about it now; it's become familiar. Sure, it's a bit awkward here and there but she's used to working around the problems. She wishes that it didn't mess up task B quite as often, but she's mostly working on tasks C through H and she knows how to do them. She also knows that tasks I, J, and K will be coming along soon but she doesn't yet know how she'll tackle them with Product A.

It's a lovely summer day. Susan arrives full of anticipation at the presentation along with several of her peers. Out comes Big Kahuna who does a presentation. "You'll love new product A+" says Kahuna. "I'm going to demonstrate it now, and show you how it will revolutionize task J, save you time on task F, and you'll be thrilled that the crash rate on task B has been reduced by 50%"

Susan is impressed. Kahuna hasn't said anything about the other tasks, but the new features sound great. Susan hasn't taken any specific notes about how often she does the tasks - why should she? she's busy enough just doing them - so she doesn't recall right now how often she does task F (in fact, it's less than once a week). She's a little concerned that task C is going to be harder, and she's pretty sure she does that 10 times a day. She's forgotten all about task D which is a big, big deal but only happens in the winter, and she's sure that everyone knows that the whole organization relies on task E, so Kahuna wouldn't even think of something that messes that up.

Excitedly, Susan fills in her stack-rank card for the new features. She wants all of them!  They'll be great!

Some months later, she gets product A+. Now task J is great, but task E! It's a horror story....And now it's the winter, she's got to do task D all the time, no-one thought of that...

If only Kahuna had asked Susan to bring typical work with her. If only they'd done proper user-centred design, finding out what was important to users and letting them experience the product in realistic contexts. If only they'd avoided .... the dreaded... consultation by presentation.

The story closes.

And the morals are:

- people don't remember the details of everyday activity
- people especially don't rememeber the details of everyday activity if you take them away from the environment of that activity and put them into a big presentation
- if a big boss presents something as a benefit, many more junior people will feel that they should not protest about it, especially in the enviroment of a bit presentation.

But if you have no option but to do 'consultation by presentation', then at least make sure that you ask people to prepare for it by bringing with them realistic examples of the types of things they need to do with the product. I generally ask for three things:

- a typical task

- an especially easy task that needs to continue to be easy

- an especially tricky task that is really important in some way.

Apart from anything else, this helps you to learn what 'typical' 'easy' 'tricky' and 'important' mean to a selection of users.

(By all means substitute 'work' or 'activity' or 'goal' or some other thing for 'task' if that works better for your product and your organization).

Best

Caroline Jarrett

Twitter @cjforms

 

 

10 Jun 2011 - 12:21pm
Zelbinian
2010

 Your comments are serving to give more tangible form to the ghosts of thoughts and concerns that were floating around in my head. This is awesome, and not only because it gives me more faith in my own abilities as a UX professional and that it generates a nifty feeling of camaraderie within this community. :)

My first reaction to this - and I said it in an email - was "Are you interested in what people think or what people do? This study will only give you the former." My other - and as yet unvoiced - reaction is that it strays wayyyy too close to making the user the designer. Users can say that they value that feature A over feature B, even when given context, but when they actually use it the truth may actually be the reverse.

10 Jun 2011 - 2:45pm
bjminihan
2010

You're all singing my chorus, and I couldn't agree more =]

The three worst phrases to see on a meeting agenda are:  Brainstorm, Touch Base, and "Get your thoughts"

Meetings are most useful for confirming concensus, not discovering it, unless they are focused, well-planned, small-scale, and realistic (I haven't been to one of those lately).

I would value results from a a Concept Verification Test on the same order as a brain-dump of requirements left in my voicemail, or (worse) instant message.  Best case, I would use that in the earliest possible conceptualization phase, when you rarely have any visual sense of what you're building, anyway (except in napkin form).

If you've gotten far enough to have stuff to show people, why not use the CVT to schedule a series of 15 minute one-on-one interviews in each member's office, after which the results will be shared and spur more detailed observations. 

Looking back at your description, it really seems they're using this as a brainstorming tool...maybe they've run out of ideas?

Bryan

10 Jun 2011 - 4:46pm
Zelbinian
2010

You know, as it happens I was reviewing the "Design Testing with Users" chapter in A Project Guide to UX Design by Russ Unger and Carolyn Chandler... and as it turns out this is an actual thing: Concept Exploration. The section on that (pgs. 221-225 if anyone happens to own it) is maddeningly unprescriptive, so I'm having trouble figuring out if we're doing it "right" (if there is such a thing). So let me get as specific about the rationale and methodology as I dare and maybe you guys can help me help them avoid a train wreck.

Basically, they're planning for the next version of the product, and they've got a bucket full of feature ideas. As far as I know, they've already estabilshed through previous research that these are all things that customers want, but they can't deliver all of them in the time they have so they want to get a rough idea of which ones are safest to leave on the cutting room floor. In these Concept Exploration groups, they'll present the product as a whole (for completeness, most people will be "experts"), then say "Ok, for feature X, we're thinking of changing it from this <<current state>> to this <<mockup of new state>> because of <<rationale> and/or to give you <<value proposition>>,"  or "We're thinking of adding new feature Y to the mix because of "<<rationale>> and/or to give you <<value proposition>>." The group gives feedback and then the process iterates until it's time to give everyone handouts of each of these feature changes/addons for them to stack-rank.

Hopefully in the aftermath it'll be a user researcher compiling the results and making the recommendations and not just raw user feedback being handed off to development, but who the hell knows. So, does all your previous feedback still apply as much as I think it does?

Edit: Thanks, IXDA.org forum for telling me I can use certain HTML tags in my post and then not processing them at all.

11 Jun 2011 - 3:05pm
Max L. Wilson
2009

from that description, it sounds like they are only trying to prioritise feature development, not actually do any design work. carolines email scenario would only be a worry if they were using to figure out their designs. if she's right then its a poor approach. if its about feature prioritisation, then its sounding a little like a market research session. in some ways, its gets the participants very well informed for a card sorting process, although i dont think it would meet the rigour of a full card sort.

although i agree all the above that both people reflecting on what they think they do, or speculating on how they might use something in the future, is never going to work out.

max

n - Dr Max L Wilson e - m.l.wilson@swansea.ac.uk w - www.cs.swan.ac.uk/~csmax/

t - +44 (0) 1792 602611

On 10 Jun 2011, at 23:34, Zelbinian wrote:

> You know, as it happens I was reviewing the "Design Testing with Users" chapter in /A Project Guide to UX Design/ by Russ Unger and Carolyn Chandler... and as it turns out this is an actual thing: Concept Exploration. The section on that (pgs. 221-225 if anyone happens to own it) is maddeningly unprescriptive, so I'm having trouble figuring out if we're doing it "right" (if there is such a thing). So let me get as specific about the rationale and methodology as I dare and maybe you guys can help me help them avoid a train wreck. > > Basically, they're planning for the next version of the product, and they've got a bucket full of feature ideas. As far as I know, they've already estabilshed through previous research that these are all things that customers want, but they can't deliver all of them in the time they have so they want to get a rough idea of which ones are safest to leave on the cutting room floor. In these Concept Exploration groups, they'll present the product as a whole (for completeness, most people will be "experts"), then say "Ok, for feature X, we're thinking of changing it from this <> to this <> because of < and/or to give you <>," or "We're thinking of adding new feature Y to the mix because of "<> and/or to give you <>." The group gives feedback and then the process iterates until it's time to give everyone handouts of each of these feature changes/addons for them to stack-rank. > > Hopefully in the aftermath it'll be a user researcher compiling the results and making the recommendations and not just raw user feedback being handed off to development, but who the hell knows. So, does all your previous feedback still apply as much as I think it does? > > /Edit: Thanks, IXDA.org forum for telling me I can use certain HTML tags in my post and then not processing them at all./ > >

11 Jun 2011 - 5:06pm
monkeyshine
2010

I would imagine that if they are this invested in user research, researchers will drive the results, working with ux and product managers to determine which features are included in the release cycle. It's pretty common to do this type of testing. One familiar model is to give users a dollar amount and have them "invest" in the most important features. I have seen this done and found it pretty useful for current customers of a product.
Deanna


On Fri, Jun 10, 2011 at 6:26 PM, Zelbinian <dustin.l.hodge@gmail.com> wrote:

You know, as it happens I was reviewing the "Design Testing with Users" chapter in /A Project Guide to UX Design/ by Russ Unger and Carolyn Chandler... and as it turns out this is an actual thing: Concept Exploration. The section on that (pgs. 221-225 if anyone happens to own it) is maddeningly unprescriptive, so I'm having trouble figuring out if we're doing it "right" (if there is such a thing). So let me get as specific about the rationale and methodology as I dare and maybe you guys can help me help them avoid a train wreck.

Basically, they're planning for the next version of the product, and they've got a bucket full of feature ideas. As far as I know, they've already estabilshed through previous research that these are all things that customers want, but they can't deliver all of them in the time they have so they want to get a rough idea of which ones are safest to leave on the cutting room floor. In these Concept Exploration groups, they'll present the product as a whole (for completeness, most people will be "experts"), then say "Ok, for feature X, we're thinking of changing it from this <<current state>> to this <<mockup of new state>> because of <<rationale> and/or to give you <<value proposition>>,"  or "We're thinking of adding new feature Y to the mix because of "<<rationale>> and/or to give you <<value proposition>>." The group gives feedback and then the process iterates until it's time to give everyone handouts of each of these feature changes/addons for them to stack-rank.

Hopefully in the aftermath it'll be a user researcher compiling the results and making the recommendations and not just raw user feedback being handed off to development, but who the hell knows. So, does all your previous feedback still apply as much as I think it does?

/Edit: Thanks, IXDA.org forum for telling me I can use certain HTML tags in my post and then not processing them at all./

((
11 Jun 2011 - 2:55am
Headspace
2010

Hi. It seems that everybody but me has a clear understanding of what the goals of this panel-session are.

I am wondering if the participants are expected to comment on usability, user experience, functionality or whatever.

Depending on the sought-after insight, the process you describe is either more or less appropriate - though it actually seems not really to address any issue in paricular.

Syndicate content Get the feed