Survey Dilemma

21 Aug 2007 - 1:14pm
6 years ago
7 replies
756 reads
bminihan
2007

This falls under the "other duties as assigned" category...

Our executives have asked my team (UCD & Web Analytics) to conduct a survey of "end users and business & IT stakeholders" (exact words) to collect data that will be mapped to proposals to migrate our email, calendar, web portal, team collab and real-time collaboration tools to either a full IBM stack or Micrsoft stack. We currently use Notes for email, calendars, meetings, team collab & Sametime for instant messaging and screen-sharing. We use BEA Aqualogic (Plumtree) for our web portal.

Our constraints are as follows:
* Don't ask any question or publish any responses that could be construed as favoring one or the other stack. Most people in our company want our Notes email environment to die slowly and loudly, while a significant block wants to push whole-hog for IBM
* Focus the survey purely on how people think they want to work in the future, and their current preferences for using technology either at work or at home.

Here are reasons why I can't wrap my brain around this and find a way forward:
* I can't figure out how to ask questions that will meaningfully reflect what people want, while completely ignoring the visceral response a large number of people have toward Notes (IBM will continue to push Notes & Quickplace into its NG workplace environments). Note: I have no preference either way for Notes or Outlook, but it's a valid issue with our employees.
* I honestly don't feel that we can collect any meaningful data from questions about how people "think they want to work", as people are historically inaccurate in describing what they think they would do in the future. My fear is that we provide a beautiful report about the "workplace of tomorrow" that reflects a set of imaginary users who will never exist. That data will be used to justify (politically or otherwise) a decision that will completely obliterate progress we've made responding to observed behavior in the past 5 years.

--
- Bryan
http://www.bryanminihan.com

Comments

21 Aug 2007 - 8:45pm
Chris Bernard
2007

Quite frankly this is a terrible way to figure out which way you should go (as I beat down an MS Account Rep). It's like designing applications by vendor and package selection.

Is it possible for your team to actually conduct a contextual research effort instead? You could perhaps frame this activity as the effort to 'figure out what to ask' (partially true anyway). Do a true big-D effort and to your insights and recommendations honestly because polling end users about platform is like asking me what type of preservatives I want in my cereal.

When it comes time to rationalize this to business decision makes you can do two things. One is do a search on Roger Martin over at Harvard Business Review and get some insights into how to present design choices and research to this audience. Two is to simply use your statistics to validate the validity of your insights (As Disrali says, "There are lies, damnable lies and statistics.")

One other option is to see if you have or can get any type of session data out of how people are using your existing toolset. This might be impossible but an implementation of a sensing mechanism for whatever you go to is critical.

Chris Bernard
Microsoft
User Experience Evangelist
chris.bernard at microsoft.com
630.530.4208 Office
312.925.4095 Mobile

Blog: www.designthinkingdigest.com
Design: www.microsoft.com/design
Tools: www.microsoft.com/expression
Community: http://www.visitmix.com

"The future is already here. It's just not evenly distributed." William Gibson

-----Original Message-----
From: discuss-bounces at lists.interactiondesigners.com [mailto:discuss-bounces at lists.interactiondesigners.com] On Behalf Of bjminihan at nc.rr.com
Sent: Tuesday, August 21, 2007 1:14 PM
To: discuss at ixda.org
Subject: [IxDA Discuss] Survey Dilemma

This falls under the "other duties as assigned" category...

Our executives have asked my team (UCD & Web Analytics) to conduct a survey of "end users and business & IT stakeholders" (exact words) to collect data that will be mapped to proposals to migrate our email, calendar, web portal, team collab and real-time collaboration tools to either a full IBM stack or Micrsoft stack. We currently use Notes for email, calendars, meetings, team collab & Sametime for instant messaging and screen-sharing. We use BEA Aqualogic (Plumtree) for our web portal.

Our constraints are as follows:
* Don't ask any question or publish any responses that could be construed as favoring one or the other stack. Most people in our company want our Notes email environment to die slowly and loudly, while a significant block wants to push whole-hog for IBM

* Focus the survey purely on how people think they want to work in the future, and their current preferences for using technology either at work or at home.

Here are reasons why I can't wrap my brain around this and find a way forward:
* I can't figure out how to ask questions that will meaningfully reflect what people want, while completely ignoring the visceral response a large number of people have toward Notes (IBM will continue to push Notes & Quickplace into its NG workplace environments). Note: I have no preference either way for Notes or Outlook, but it's a valid issue with our employees.
* I honestly don't feel that we can collect any meaningful data from questions about how people "think they want to work", as people are historically inaccurate in describing what they think they would do in the future. My fear is that we provide a beautiful report about the "workplace of tomorrow" that reflects a set of imaginary users who will never exist. That data will be used to justify (politically or otherwise) a decision that will completely obliterate progress we've made responding to observed behavior in the past 5 years.

--
- Bryan
http://www.bryanminihan.com
________________________________________________________________
Welcome to the Interaction Design Association (IxDA)!
To post to this list ....... discuss at ixda.org
List Guidelines ............ http://beta.ixda.org/guidelines
List Help .................. http://beta.ixda.org/help
Unsubscribe ................ http://beta.ixda.org/unsubscribe
Questions .................. list at ixda.org
Home ....................... http://beta.ixda.org

21 Aug 2007 - 9:04pm
Steve Baty
2009

Chris,

With all due respect, I hope you're not entirely serious about this
suggestion.

On 22/08/07, Chris Bernard <Chris.Bernard at microsoft.com> wrote:
>
> Two is to simply use your statistics to validate the validity of your
> insights (As Disrali says, "There are lies, damnable lies and statistics.")
>

"He uses statistics as a drunken man uses lampposts—
for support rather than illumination."
Andrew Lang (1844-1912)

In hope,
Steve

----------------------------------------------
Steve 'Doc' Baty B.Sc (Maths), M.EC, MBA
Director, User Experience Strategy
Red Square
P: +612 8289 4930
M: +61 417 061 292

Member, UPA - www.upassoc.org
Member, IxDA - www.ixda.org
Member, Web Standards Group - www.webstandardsgroup.org

21 Aug 2007 - 9:37pm
bminihan
2007

Thanks for the tip re: Roger Martin, I'll definitely take a look.

You're absolutely right; this is the wrong way to go about the whole thing.
I wish that were in my control. Unfortunately, the survey idea was sold way
up the chain without my (or my team's) input into the best data collection
methods. As usual, we have about a week to build the question-set,
remaining within all the aforementioned constraints.

For what it's worth, I pitched the idea of contextual inquiry, or even focus
groups to get at the heart of how people feel, but there isn't much time to
pull folks together from 7 business units, several US & UK sites and
representatives from a dozen other countries. I even drafted a
quasi-prototype of the portal-portion, but won't have a chance to test it
with anyone *sigh* (if you're interested, it's here:
http://www.bryanminihan.com/case-informationworkplace.html)

Thanks for the tip...if we get anything useful out of this thing, I'll share
with anyone interested.

- Bryan
http://www.bryanminihan.com

-----Original Message-----
From: Chris Bernard [mailto:Chris.Bernard at microsoft.com]
Sent: Tuesday, August 21, 2007 9:45 PM
To: bjminihan at nc.rr.com; discuss at ixda.org
Subject: RE: [IxDA Discuss] Survey Dilemma

Quite frankly this is a terrible way to figure out which way you should go
(as I beat down an MS Account Rep). It's like designing applications by
vendor and package selection.

Is it possible for your team to actually conduct a contextual research
effort instead? You could perhaps frame this activity as the effort to
'figure out what to ask' (partially true anyway). Do a true big-D effort and
to your insights and recommendations honestly because polling end users
about platform is like asking me what type of preservatives I want in my
cereal.

When it comes time to rationalize this to business decision makes you can do
two things. One is do a search on Roger Martin over at Harvard Business
Review and get some insights into how to present design choices and research
to this audience. Two is to simply use your statistics to validate the
validity of your insights (As Disrali says, "There are lies, damnable lies
and statistics.")

One other option is to see if you have or can get any type of session data
out of how people are using your existing toolset. This might be impossible
but an implementation of a sensing mechanism for whatever you go to is
critical.

Chris Bernard
Microsoft
User Experience Evangelist
chris.bernard at microsoft.com
630.530.4208 Office
312.925.4095 Mobile

Blog: www.designthinkingdigest.com
Design: www.microsoft.com/design
Tools: www.microsoft.com/expression
Community: http://www.visitmix.com

"The future is already here. It's just not evenly distributed." William
Gibson

-----Original Message-----
From: discuss-bounces at lists.interactiondesigners.com
[mailto:discuss-bounces at lists.interactiondesigners.com] On Behalf Of
bjminihan at nc.rr.com
Sent: Tuesday, August 21, 2007 1:14 PM
To: discuss at ixda.org
Subject: [IxDA Discuss] Survey Dilemma

This falls under the "other duties as assigned" category...

Our executives have asked my team (UCD & Web Analytics) to conduct a survey
of "end users and business & IT stakeholders" (exact words) to collect data
that will be mapped to proposals to migrate our email, calendar, web portal,
team collab and real-time collaboration tools to either a full IBM stack or
Micrsoft stack. We currently use Notes for email, calendars, meetings, team
collab & Sametime for instant messaging and screen-sharing. We use BEA
Aqualogic (Plumtree) for our web portal.

Our constraints are as follows:
* Don't ask any question or publish any responses that could be construed
as favoring one or the other stack. Most people in our company want our
Notes email environment to die slowly and loudly, while a significant block
wants to push whole-hog for IBM

* Focus the survey purely on how people think they want to work in the
future, and their current preferences for using technology either at work or
at home.

Here are reasons why I can't wrap my brain around this and find a way
forward:
* I can't figure out how to ask questions that will meaningfully reflect
what people want, while completely ignoring the visceral response a large
number of people have toward Notes (IBM will continue to push Notes &
Quickplace into its NG workplace environments). Note: I have no preference
either way for Notes or Outlook, but it's a valid issue with our employees.
* I honestly don't feel that we can collect any meaningful data from
questions about how people "think they want to work", as people are
historically inaccurate in describing what they think they would do in the
future. My fear is that we provide a beautiful report about the "workplace
of tomorrow" that reflects a set of imaginary users who will never exist.
That data will be used to justify (politically or otherwise) a decision that
will completely obliterate progress we've made responding to observed
behavior in the past 5 years.

--
- Bryan
http://www.bryanminihan.com
________________________________________________________________
Welcome to the Interaction Design Association (IxDA)!
To post to this list ....... discuss at ixda.org
List Guidelines ............ http://beta.ixda.org/guidelines
List Help .................. http://beta.ixda.org/help
Unsubscribe ................ http://beta.ixda.org/unsubscribe
Questions .................. list at ixda.org
Home ....................... http://beta.ixda.org

21 Aug 2007 - 11:13pm
Chris Bernard
2007

That was a joke. Sometimes sarcasm is lost in this medium.

Chris Bernard
Microsoft
User Experience Evangelist
chris.bernard at microsoft.com<mailto:chris.bernard at microsoft.com>
630.530.4208 Office
312.925.4095 Mobile

[cid:image001.png at 01C7E448.D2A9A560]<http://www.microsoft.com/silverlight>

Blog: www.designthinkingdigest.com<http://www.designthinkingdigest.com/>
Design: www.microsoft.com/design<http://www.microsoft.com/design>
Tools: www.microsoft.com/expression<http://www.microsoft.com/expression>
Community: http://www.visitmix.com<http://www.visitmix.com/>

"The future is already here. It's just not evenly distributed." William Gibson

From: Steve Baty [mailto:stevebaty at gmail.com]
Sent: Tuesday, August 21, 2007 9:04 PM
To: Chris Bernard; discuss at ixda.org
Subject: Re: [IxDA Discuss] Survey Dilemma

Chris,

With all due respect, I hope you're not entirely serious about this suggestion.
On 22/08/07, Chris Bernard < Chris.Bernard at microsoft.com<mailto:Chris.Bernard at microsoft.com>> wrote:
Two is to simply use your statistics to validate the validity of your insights (As Disrali says, "There are lies, damnable lies and statistics.")

"He uses statistics as a drunken man uses lampposts-
for support rather than illumination."
Andrew Lang (1844-1912)

In hope,
Steve

----------------------------------------------
Steve 'Doc' Baty B.Sc (Maths), M.EC<http://M.EC>, MBA
Director, User Experience Strategy
Red Square
P: +612 8289 4930
M: +61 417 061 292

Member, UPA - www.upassoc.org<http://www.upassoc.org>
Member, IxDA - www.ixda.org<http://www.ixda.org>
Member, Web Standards Group - www.webstandardsgroup.org<http://www.webstandardsgroup.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 6885 bytes
Desc: image001.png
Url : http://lists.interactiondesigners.com/pipermail/discuss-interactiondesigners.com/attachments/20070821/c6480d48/attachment.png

22 Aug 2007 - 6:53am
Faith Peterson
2007

Random thoughts about approach, in no particular order:

- Do you have to have just one survey for all three classes of
respondents?
- Does IT already have a feature matrix comparing the current
application set and the two contenders that you could use to focus questions
on the factors that will actually make a difference to the decision?
- Can the people who want the survey tell you what are the most
important factors in their decision? Do they care about data migration,
adoption, used features that might be missing in the new app, gaps in the
current set they hope to remedy?
- Do you have contacts in different departments whom you could talk
briefly with on the phone to get some insights about actual use that you
could use to frame questions?
- When in doubt follow the flow of information. If you ask about the
information people receive and send, to whom, when, independent of the
technology used, sometimes that can yield better information to information
application selection than asking them about software directly.
- Can you just not use the contextual inquiry label with the boss and
try to come up with context questions for the survey?
- Are you prohibited from asking open-ended questions?
- Could you frame questions about how often they do or don't do
certain activities or use e-mail, calendar, webex, document repository, or
whatever - almost all day, several times a day, daily, a few times a week,
more seldom. When you use email what do you use it for: quick messages,
sending documents, project discussions. How quickly do you expect a response
to an email you send - a few minutes, an hour, 1/2 a day, a day, next day.
Include open-ended questions - how satisfied are you with ABC? Why or why
not?
- If they insist that you ask about the future, you could spin it as
"Imagine a system that ______. Would you use ______? Why or why not? Would
your team use _____? Why or why not? What for?"

On 8/21/07, Bryan Minihan <bjminihan at nc.rr.com> wrote:
>
> As usual, we have about a week to build the question-set,
> remaining within all the aforementioned constraints.
>
> For what it's worth, I pitched the idea of contextual inquiry, or even
> focus
> groups to get at the heart of how people feel, but there isn't much time
> to
> pull folks together ...if we get anything useful out of this thing, I'll
> share
> with anyone interested.
> ....
> Our constraints are as follows:
> * Don't ask any question or publish any responses that could be construed
> as favoring one or the other stack. ...

* Focus the survey purely on how people think they want to work in the
> future, and their current preferences for using technology either at work
> or
> at home.
>
--
Faith Peterson
f.a.peterson at gmail.com
Schaumburg, IL
http://www.linkedin.com/in/fpeterson

22 Aug 2007 - 7:53am
bminihan
2007

Thanks for the questions and advice, Faith. It seems we're on pretty much the same path you suggested, which helps to hear =]. Our current approach is to try to gauge how people use some of the tools that are just emerging, to determine (roughly) their comfort level with an information workplace down the road that combines all these things. Q's in the vein of "Have you visited Wikipedia before, are you comfortable with it?" might help us understand that a workplace down the road that depends on Wikis for reference information might or might not be that bad.

To answer some of your questions:
- We decided to conduct three simultaneous surveys for the IT stakeholders (to gauge support considerations & "the expert skill set"), Business Stakeholdeers (to gauge the business fit for future technology) and End Users (to gauge attitudes and yearning for a better work env).
- IT has a loose feature matrics (honestly, they haven't organized it and aren't doing a great job drawing distinctions yet, so I'm relying on what I've learned in my own research), but we're not allowed to use it to draw questions (because it would differentiate Notes from Outlook)
- I've been asked to take all steps necessary to make the survey come out to support both platforms. That's the #1 objective, and the only one provided, unfortunately.
- We enlisted the aid of some folks from business units who, as luck would have it, vehemently want to prove that we should move to one platform or another.
- I pitched the contextual inquiry as "can we at least visit some folks to do a quick deep dive of how users think and want to work, rather than send a questionnaire that would cause confusion and require a lot of tortured language about what an Information Workplace is?". The answer was "no".
- Nope, we plan to ask at least one open-ended question, and hope to get some valuable information from those.

Our set, so far, is breaking down into three categories: Questions about how you work now, Questions about how you wish to work (using various tools people might be using right now), and Questions about how your workplace should fit your role in the company. We have some good starters, but the trick is in figuring out how to keep the questions to less than 20 without nixing a valuable one.

Thanks again =]

- Bryan
http://www.bryanminihan.com

---- Faith Peterson <f.a.peterson at gmail.com> wrote:
> Random thoughts about approach, in no particular order:
>
> - Do you have to have just one survey for all three classes of
> respondents?
> - Does IT already have a feature matrix comparing the current
> application set and the two contenders that you could use to focus questions
> on the factors that will actually make a difference to the decision?
> - Can the people who want the survey tell you what are the most
> important factors in their decision? Do they care about data migration,
> adoption, used features that might be missing in the new app, gaps in the
> current set they hope to remedy?
> - Do you have contacts in different departments whom you could talk
> briefly with on the phone to get some insights about actual use that you
> could use to frame questions?
> - When in doubt follow the flow of information. If you ask about the
> information people receive and send, to whom, when, independent of the
> technology used, sometimes that can yield better information to information
> application selection than asking them about software directly.
> - Can you just not use the contextual inquiry label with the boss and
> try to come up with context questions for the survey?
> - Are you prohibited from asking open-ended questions?
> - Could you frame questions about how often they do or don't do
> certain activities or use e-mail, calendar, webex, document repository, or
> whatever - almost all day, several times a day, daily, a few times a week,
> more seldom. When you use email what do you use it for: quick messages,
> sending documents, project discussions. How quickly do you expect a response
> to an email you send - a few minutes, an hour, 1/2 a day, a day, next day.
> Include open-ended questions - how satisfied are you with ABC? Why or why
> not?
> - If they insist that you ask about the future, you could spin it as
> "Imagine a system that ______. Would you use ______? Why or why not? Would
> your team use _____? Why or why not? What for?"
>
> On 8/21/07, Bryan Minihan <bjminihan at nc.rr.com> wrote:
> >
> > As usual, we have about a week to build the question-set,
> > remaining within all the aforementioned constraints.
> >
> > For what it's worth, I pitched the idea of contextual inquiry, or even
> > focus
> > groups to get at the heart of how people feel, but there isn't much time
> > to
> > pull folks together ...if we get anything useful out of this thing, I'll
> > share
> > with anyone interested.
> > ....
> > Our constraints are as follows:
> > * Don't ask any question or publish any responses that could be construed
> > as favoring one or the other stack. ...
>
>
> * Focus the survey purely on how people think they want to work in the
> > future, and their current preferences for using technology either at work
> > or
> > at home.
> >
> --
> Faith Peterson
> f.a.peterson at gmail.com
> Schaumburg, IL
> http://www.linkedin.com/in/fpeterson

26 Aug 2007 - 5:42pm
Oleh Kovalchuke
2006

This might be an unorthodox approach, but I am under influence of 'The
Wisdom of Crowds' by James Surowiecki at the moment.

Can you try a controlled office pool for predicting future outcome?

An example of the question could be: "If you had $100, how much would you
bet that the company will adopt IBM vs. MS stack in three years?" Give an
incentive of $100 to the person, whose guess will be the closest to the mean
value of all guesses.

To sell the result to your executives present them with a copy of the book.
I should say though, that from the way they framed the problem, the
executives might be already aware of this approach.

Oleh

On 8/21/07, bjminihan at nc.rr.com <bjminihan at nc.rr.com> wrote:
>
> This falls under the "other duties as assigned" category...
>
> Our executives have asked my team (UCD & Web Analytics) to conduct a
> survey of "end users and business & IT stakeholders" (exact words) to
> collect data that will be mapped to proposals to migrate our email,
> calendar, web portal, team collab and real-time collaboration tools to
> either a full IBM stack or Micrsoft stack. We currently use Notes for
> email, calendars, meetings, team collab & Sametime for instant messaging and
> screen-sharing. We use BEA Aqualogic (Plumtree) for our web portal.
>
> Our constraints are as follows:
> * Don't ask any question or publish any responses that could be construed
> as favoring one or the other stack. Most people in our company want our
> Notes email environment to die slowly and loudly, while a significant block
> wants to push whole-hog for IBM
> * Focus the survey purely on how people think they want to work in the
> future, and their current preferences for using technology either at work or
> at home.
>
> Here are reasons why I can't wrap my brain around this and find a way
> forward:
> * I can't figure out how to ask questions that will meaningfully reflect
> what people want, while completely ignoring the visceral response a large
> number of people have toward Notes (IBM will continue to push Notes &
> Quickplace into its NG workplace environments). Note: I have no preference
> either way for Notes or Outlook, but it's a valid issue with our employees.
> * I honestly don't feel that we can collect any meaningful data from
> questions about how people "think they want to work", as people are
> historically inaccurate in describing what they think they would do in the
> future. My fear is that we provide a beautiful report about the "workplace
> of tomorrow" that reflects a set of imaginary users who will never
> exist. That data will be used to justify (politically or otherwise) a
> decision that will completely obliterate progress we've made responding to
> observed behavior in the past 5 years.
>
> --
> - Bryan
> http://www.bryanminihan.com
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> List Guidelines ............ http://beta.ixda.org/guidelines
> List Help .................. http://beta.ixda.org/help
> Unsubscribe ................ http://beta.ixda.org/unsubscribe
> Questions .................. list at ixda.org
> Home ....................... http://beta.ixda.org
>

--
Oleh Kovalchuke
Interaction Design is the Design of Time
http://www.tangospring.com/IxDtopicWhatIsInteractionDesign.htm

Syndicate content Get the feed