Likert scale survey designs

26 Nov 2008 - 3:59pm
5 years ago
17 replies
4105 reads
Robert Hoekman, Jr.
2005

Anyone ever seen a great design for a long survey comprised entirely of
Likert scale questions?
One I need to design has 100 questions, which could obviously make it pretty
tedious. Looking for ways to make it feel fast and easy.

Thanks!

-r-

Comments

27 Nov 2008 - 5:40am
Alan James Salmoni
2008

One possibility is to perform a pilot and see which questions co-vary.
This can help identify questions which are perceived to answer the
same thing and reduce them down to one. Having said that, there might
be times when repeating a question (e.g., an L-score) is necessary.

In terms of examples, try Eysenck's EPQ questionnaire - the full
version has lots of questions and is frankly a bore to go through so
maybe it's not so good.

Another strategy might be to break questions down into sections so
that users can pace themselves. Sometimes a terse explanation about
each section's purpose might be useful if that doesn't compromise
the design.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=35980

27 Nov 2008 - 10:14am
Robert Hoekman, Jr.
2005

Thanks for your responses, everyone. Turns out, though, that while writing
down the design criteria for the survey design, the solution magically
presented itself. Need to run it by a few users, but I think I have a
winner.
Cheers!

-r-

26 Nov 2008 - 4:39pm
Sam Ladner
2008

Hi Robert,

Attached is a screen shot of a word document. It's a likert-scale survey of
the Revised Learning Process Questionnaire (R-LPQ) I used for some teaching
research. I have adapted this kind of layout many times for online versions.
Instead of just a grid, I use radio buttons.

One proviso: 100 questions for a survey will end up with invalid results. If
it's online (which I'm assuming it is) you have about 15 minutes of
attention span from your user. After this time, they tend to engage in what
they call "acquiescence bias," or simply answering the same way repeatedly.

I'm sure we've all done that -- simply putting in a whole whack of 5's after
we get bored.

I would recommend either breaking it up into several sections, with several
"grids" (a strategy I've used repeatedly), or attempt multiple surveys. If
you do the several grid option, you can pack in about 10 questions per grid.
They can follow the same theme, such as "online behaviour" (10 questions)
and then "demographics" (another 10), etc.

And I'd also question why you need 100 questions. Very few questionnaires
have this many, and those that do often are done through structured
interviewing, where you're answering an actual person.

Cheers,

--
~~~~~
Sam Ladner, PhD
Sociologist
Toronto
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Picture 8.png
Type: image/png
Size: 35485 bytes
Desc: not available
URL: <http://lists.interactiondesigners.com/pipermail/discuss-interactiondesigners.com/attachments/20081126/6e9ca752/attachment.png>

27 Nov 2008 - 11:42am
Robert Hoekman, Jr.
2005

>
> I have adapted this kind of layout many times for online versions.
> Instead of just a grid, I use radio buttons.

This is exactly what I want to avoid—row after row of radio button groups.
Very tedious, not the least bit enjoyable, and in a survey with 100
questions, it means bombarding the user with multiple pages containing lots
of questions. As I said, it can be demotivating.

One proviso: 100 questions for a survey will end up with invalid results. If
> it's online (which I'm assuming it is) you have about 15 minutes of
> attention span from your user.

I'd be amazed if I got that much time out of a user. I'm aiming for 3-4
minutes tops.

And I'd also question why you need 100 questions.

It's a personality analysis survey. It's actually the most widely-recognized
questionnaire there is, and changing it is not really an option, because
shortening it would sacrifice far too much accuracy in the analysis.

-r-

27 Nov 2008 - 12:02pm
Robert Hoekman, Jr.
2005

Here's what I designed <http://rhjr.net/tests/LikertScale.gif>.
I'd love any and all feedback. It's obviously an unconventional design, so
I'd like to run it by some users, and I'd love to hear the impressions of
other designers.

The task flow:

1. User clicks a response to the current active statement (blue, bold,
larger font)
2. List of statements auto-scrolls and the next statement becomes active
3. To change a previous answer, user clicks the arrows to navigate back and
forth

Here was my criteria for the design:

1. Minimal mouse movements (the answer buttons don't move—the statements do)
2. Keep the user motivated (the design shows upcoming statements, but
not enough to overwhelm)
3. Provide large hit areas for responses (something radio buttons don't
have)
4. Track progress (the counter in the upper-right)
5. Allow user to change his/her mind (the arrows to go back/forward to
change answers or review)
6. Set clear expectations (the text about typical survey duration, and the
counter)
7. To avoid tedium, it has to feel fast (hopefully, the design as a whole
achieves this)
8. Avoid making the user memorize the responses, as well as the order of the
responses (used a color scale for the buttons to help the user infer their
meaning without reading explicitly each time)

Thoughts?

Thanks, everyone.

-r-

27 Nov 2008 - 12:49pm
Adrian Howard
2005

On 27 Nov 2008, at 17:02, Robert Hoekman Jr wrote:

> Here's what I designed <http://rhjr.net/tests/LikertScale.gif>.
> I'd love any and all feedback. It's obviously an unconventional
> design, so
> I'd like to run it by some users, and I'd love to hear the
> impressions of
> other designers.
[snip]

I like it. Feels much more like a game / quiz machine than a
questionnaire.

Niggles that immediately occurred to me on a first viewing
* I initially thought green-is-good/red-is-bad... I expect that will
bias some folk from the actual accurate/inaccurate scale.
* Why does "neutral" have a smaller clickable areas? Again - bias folk
away from picking it.
* For some reason my eyes are drawn to the "next active question" (#6
in the mock up) - I'm guessing that (despite the numbering) that it
looks like the content under a heading. Might just be me. Might not
happen when you see it in "real life". Might go away if the display
was symmetric so the current question was centred...
* Would be nice to have an indication of how you answered questions in
the preview area. Would help when scanning for questions you didn't
answer. Would help verify that you actually did hit the button you
thought you did on the last question. I imagine that once folk are
into the flow of the task they'll just focus on the question text - so
may miss button feedback that would cue them they miss-clicked.

Neat concept!

Cheers,

Adrian

27 Nov 2008 - 12:59pm
Robert Hoekman, Jr.
2005

>
> * I initially thought green-is-good/red-is-bad... I expect that will bias
> some folk from the actual accurate/inaccurate scale.

Good point. I was thinking, "Red = negative as in disagree, Green = positive
as in agree". Trying to make use of that existing mental model. Perhaps
there are other ways.

> * Why does "neutral" have a smaller clickable areas? Again - bias folk away
> from picking it.

In reality, the buttons will probably be all the same size, but yeah, thanks
for pointing that out.

> * Would be nice to have an indication of how you answered questions in the
> preview area. Would help when scanning for questions you didn't answer.
> Would help verify that you actually did hit the button you thought you did
> on the last question. I imagine that once folk are into the flow of the task
> they'll just focus on the question text - so may miss button feedback that
> would cue them they miss-clicked.
>

In my storyboard for the design, I show that after you click your response,
the other buttons turn gray and there's a 1-second delay before the
auto-advance kicks in. Haven't decided yet if I'm going to keep it that way,
but it does help address the mis-click possibility, because at least you'd
be able to see what you clicked for a second before moving on. If you
mis-clicked, you can use the arrows to go back one question and change your
response.

On another note, this design also gets you through an entire survey without
the possibility of seeing a single error message. As in, click wrong button
> go back > change it. Can't advance until you've answered the current
question. No validation needed. I'm very happy about that. I'm a big
advocate of eliminating errors.

Regardless, I love coming up with new approaches to things, but
unconventional designs definitely bring up challenges. :)

-r-

27 Nov 2008 - 1:08pm
Loren Baxter
2007

Cool design! Looks less intimidating than a huge form, too.

Feedback:
* Agree with Adrian that finished questions should have some indicator of
whether you answered them or not in the preview. And, they need to show
what answer you gave, so that you can decide to go back and change it if
needed.
* The "I..." and the actual question aren't grouped very well (only via
color / font). Why not have the question scroll up and land directly to the
right of the "I..."? Seems like it would be possible. OR, simply place the
"I..." at the front of every question, instead of separating it out like
that.

Happy Thanksgiving,
Loren

-----
http://acleandesign.com

On Thu, Nov 27, 2008 at 9:59 AM, Robert Hoekman Jr <robert at rhjr.net> wrote:

> >
> > * I initially thought green-is-good/red-is-bad... I expect that will bias
> > some folk from the actual accurate/inaccurate scale.
>
>
> Good point. I was thinking, "Red = negative as in disagree, Green =
> positive
> as in agree". Trying to make use of that existing mental model. Perhaps
> there are other ways.
>
>
> > * Why does "neutral" have a smaller clickable areas? Again - bias folk
> away
> > from picking it.
>
>
> In reality, the buttons will probably be all the same size, but yeah,
> thanks
> for pointing that out.
>
>
> > * Would be nice to have an indication of how you answered questions in
> the
> > preview area. Would help when scanning for questions you didn't answer.
> > Would help verify that you actually did hit the button you thought you
> did
> > on the last question. I imagine that once folk are into the flow of the
> task
> > they'll just focus on the question text - so may miss button feedback
> that
> > would cue them they miss-clicked.
> >
>
> In my storyboard for the design, I show that after you click your response,
> the other buttons turn gray and there's a 1-second delay before the
> auto-advance kicks in. Haven't decided yet if I'm going to keep it that
> way,
> but it does help address the mis-click possibility, because at least you'd
> be able to see what you clicked for a second before moving on. If you
> mis-clicked, you can use the arrows to go back one question and change your
> response.
>
> On another note, this design also gets you through an entire survey without
> the possibility of seeing a single error message. As in, click wrong button
> > go back > change it. Can't advance until you've answered the current
> question. No validation needed. I'm very happy about that. I'm a big
> advocate of eliminating errors.
>
> Regardless, I love coming up with new approaches to things, but
> unconventional designs definitely bring up challenges. :)
>
> -r-
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

27 Nov 2008 - 2:42pm
Adrian Howard
2005

On 27 Nov 2008, at 17:59, Robert Hoekman Jr wrote:

[snip]
> In my storyboard for the design, I show that after you click your
> response,
> the other buttons turn gray and there's a 1-second delay before the
> auto-advance kicks in. Haven't decided yet if I'm going to keep it
> that way,
> but it does help address the mis-click possibility, because at least
> you'd
> be able to see what you clicked for a second before moving on. If you
> mis-clicked, you can use the arrows to go back one question and
> change your
> response.
[snip]

I'm guessing (I'd try it out with some users) that folk would would
find that pause annoying. On a happy path the user has correctly
answered the question - so the pause only prevents them moving on to
the next question.

A less annoying version of a "you pressed button A" window that
appeared then faded away... unless you make a mistake you're always
thinking "I know I pressed A - that's what I just did!". Feels more
like a place for an undo...

But I might be over thinking it :)

Adrian

27 Nov 2008 - 4:39pm
Matt Nish-Lapidus
2007

I'd agree with that.

A proposed flow:

- click an answer button
- immediately get the next question
- next question page includes a spot (top or bottom?) that confirms
the previous answer and offer to undo/go back

That way I can just ignore the confirmation unless I actually need to
change my answer.

On Thu, Nov 27, 2008 at 2:42 PM, Adrian Howard <adrianh at quietstars.com> wrote:
>
> On 27 Nov 2008, at 17:59, Robert Hoekman Jr wrote:
>
> [snip]
>>
>> In my storyboard for the design, I show that after you click your
>> response,
>> the other buttons turn gray and there's a 1-second delay before the
>> auto-advance kicks in. Haven't decided yet if I'm going to keep it that
>> way,
>> but it does help address the mis-click possibility, because at least you'd
>> be able to see what you clicked for a second before moving on. If you
>> mis-clicked, you can use the arrows to go back one question and change
>> your
>> response.
>
> [snip]
>
> I'm guessing (I'd try it out with some users) that folk would would find
> that pause annoying. On a happy path the user has correctly answered the
> question - so the pause only prevents them moving on to the next question.
>
> A less annoying version of a "you pressed button A" window that appeared
> then faded away... unless you make a mistake you're always thinking "I know
> I pressed A - that's what I just did!". Feels more like a place for an
> undo...
>
> But I might be over thinking it :)
>
> Adrian
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

--
Matt Nish-Lapidus
--
personal: mattnl at gmail.com
twitter: emenel

27 Nov 2008 - 8:56pm
Marcus Coghlan
2007

Like the idea a lot.

Couple of thoughts:

1) Can you move the scroll buttons from the right of the question
panel to the bottom - between the bottom of the questions panel and
the response buttons? Or even 'scroll down' below the panel and
'scroll up' above the panel? Either way, I think closer proximity
between 'answering' and 'moving on' will speed things up and
reduce effort for the user.

2) Is there a need for users to go back to questions they have
already answered? If not, could you simply clear the questions from
the list as they are answered, so that at any time the list only
contains unanswered questions? As well as making it easier to get
back to earlier unanswered questions without needing to scroll up for
5-10seconds, but with the right visual treatment would aid the user's
perception of making progress toward their goal.

Good luck with it.

Marcus

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=35980

27 Nov 2008 - 9:07pm
Robert Hoekman, Jr.
2005

>
> 1) Can you move the scroll buttons from the right of the question
> panel to the bottom - between the bottom of the questions panel and
> the response buttons?

I appreciate the thoughts, but I'm confused by this. I put the arrows in the
same position a scrollbar would go in a textbox, browser, or app window of
any kind, to leverage the design pattern. Putting them between the questions
and answers — wouldn't that then break (or at least interrupt) the
relationship between questions and answers?

2) Is there a need for users to go back to questions they have
> already answered?

Yes, but cool idea. :)

-r-

27 Nov 2008 - 9:53pm
Mark Schraad
2006

the ah ha syndrome strikes again... the subconscious processing of
the human brain?

On Nov 27, 2008, at 10:14 AM, Robert Hoekman Jr wrote:

> Thanks for your responses, everyone. Turns out, though, that while
> writing
> down the design criteria for the survey design, the solution magically
> presented itself. Need to run it by a few users, but I think I have a
> winner.
> Cheers!
>
> -r-
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help

28 Nov 2008 - 2:51am
Michael Stiso
2006

I really like it, especially if you incorporate the feedback that
everyone has given. I don't have much to add that hasn't already
been said, but I am curious about what you're using to create the
survey. Flash? Something else?

Mike

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=35980

28 Nov 2008 - 3:03am
Eric Scheid
2006

On 28/11/08 4:59 AM, "Robert Hoekman Jr" <robert at rhjr.net> wrote:

> If you
> mis-clicked, you can use the arrows to go back one question and change your
> response.

what happens if they click on the text of the previous question? (it's
bigger, it's not a proxy, it's right above where their mouse already is)

e.

27 Nov 2008 - 1:53pm
Anonymous

I like this much better than a wall of Likert Scale radio buttons.

The knowledge of exactly which question the user is focusing on, and
the layout of this survey, presents a really cool opportunity to
offer timed prompts. Suppose the user is mulling over a particular
question for an extended period of time. To the right of the the
statement, the quiz could display messages that clarify the question
or say things like "What is true most of the time?"

I'm a strong proponent for eliminating unnecessary labels. I don't
think you need the Question/Answer labels as that's very apparent.
Besides, these are really Statements, not Questions. ;-)

Do you really want to support the user behavior of reviewing and
changing answers, or do you want them to go with their first choice
for the most part? If the former, your design makes this downright
painful; if the latter, this works really well.

Kim, ENTJ

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=35980

28 Nov 2008 - 11:43am
Oleh Kovalchuke
2006

Nice, inspiring design, and good suggestions.

The feedback Adrian is talking about to facilitate the review can be in the
form of colored bullet (with numbers -2 -1 0 1 and 2 in it or with thumbs
up/down) next to the answered question (in addition to the button highlight
you are suggesting -- I would simply flatten the selected button).

Oleh Kovalchuke
Interaction Design is design of time
http://www.tangospring.com/IxDtopicWhatIsInteractionDesign.htm

On Thu, Nov 27, 2008 at 11:02 AM, Robert Hoekman Jr <robert at rhjr.net> wrote:

> Here's what I designed <http://rhjr.net/tests/LikertScale.gif>.
> I'd love any and all feedback. It's obviously an unconventional design, so
> I'd like to run it by some users, and I'd love to hear the impressions of
> other designers.
>
> The task flow:
>
> 1. User clicks a response to the current active statement (blue, bold,
> larger font)
> 2. List of statements auto-scrolls and the next statement becomes active
> 3. To change a previous answer, user clicks the arrows to navigate back and
> forth
>
> Here was my criteria for the design:
>
> 1. Minimal mouse movements (the answer buttons don't move—the statements
> do)
> 2. Keep the user motivated (the design shows upcoming statements, but
> not enough to overwhelm)
> 3. Provide large hit areas for responses (something radio buttons don't
> have)
> 4. Track progress (the counter in the upper-right)
> 5. Allow user to change his/her mind (the arrows to go back/forward to
> change answers or review)
> 6. Set clear expectations (the text about typical survey duration, and the
> counter)
> 7. To avoid tedium, it has to feel fast (hopefully, the design as a whole
> achieves this)
> 8. Avoid making the user memorize the responses, as well as the order of
> the
> responses (used a color scale for the buttons to help the user infer their
> meaning without reading explicitly each time)
>
> Thoughts?
>
> Thanks, everyone.
>
> -r-
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

Syndicate content Get the feed