Analyzing usability testing notes

1 Jul 2008 - 9:34am
6 years ago
10 replies
889 reads
Guillermo Ermel
2008

Hello folks!

I'm trying to find a better way to do usability test analysis.

My current approach is: after i finish a usability study, with 8 or 10
users, and collected my own and all observers' notes, I usually read all
notes and then immediately write down the issues I feel area appearing
more often (assuming my brain will remember issues that are repeated in
many notes more than issues than appear only once).

Now, how do YOU approach analyzing those notes? Reading and re-writing
by heart? Putting all notes on a wall and eye-balling? Tagging the text
with some piece of software?

Thanks!

--
Guillermo Ermel
Head of web usability
MercadoLibre.com

Comments

1 Jul 2008 - 10:52am
Patricia Garcia
2007

One thing I started to do was create a chart using pen and paper. On
the left side I list the numbers corresponding to the users. Across
the top, I leave open. As I read my notes, I write issues
encountered across the top of my chart and mark the user number with
a a tic. As I review all my notes I start to see the issues that
have the most tics and repeating across users. I can always go back
to the exact user to review video if I had more questions.

As a time saver, I started to create my chart before the testing
begins and takes notes during testing this way as well. But still
review my notes again afterward.

Knowing where the most issues were, I can write up my report focusing
on solutions to those issues first.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=30937

1 Jul 2008 - 1:05pm
Erin Walsh
2007

The most effective method we have used has been a cluster analysis
with comments on post-its. You can color code by participant to keep
things in perspective.

The last test we did was not only our site, but a competitive
analysis as well. (It this case, we color coded post-its with
sites.) Once complete with the analysis, we melded our groupings
with a take-off of the mental model. We drew a line under the groups
and then lined up with competitors provided the desired
functionality. In a quick glance you could spot the impending
threats, industry-wide opportunities, etc. Throw in some simplistic
SWOT type icons and viola, we had a visual summary of a ridiculous
number of tests, the shortest amount of analysis time in my career
and the most easily distributable results across all groups:
business, product, UI, etc.

1 Jul 2008 - 1:47pm
Anonymous

2008/7/1 Erin Walsh <erin.walsh at dominionenterprises.com>:
> The most effective method we have used has been a cluster analysis with
> comments on post-its. You can color code by participant to keep things in
> perspective.
>

Hello,

I though cluster analysis was only for card sorting. How do you do
cluster analysis on user testing ? what are the variables ?

thanks

1 Jul 2008 - 2:30pm
Sarah Kampman
2008

Though I take notes, I rely more heavily on quantitative measures when
assessing the results of a usability test. All of the tasks I have
participants complete have degrees of success, and often a time
component as well. These measures lend themselves to comparison and
analysis in a way that quotes do not. This is important for me, as
usability testing only part of what I do, and I don't have the time to
write out transcripts. The easier & faster I can make analysis, the
better.

As for the comments that I record, they fall into two categories:
marketing and feedback. The "marketing" quotes are used to make a point
internally, often to help position a change/enhancement as meaningful to
a particular persona. The "feedback" quotes I use to fix whatever was
problematic in the usability test, and I'll typically have my mockups up
in Dreamweaver as I go through the feedback notes so that I can make the
needed changes immediately.

-Sarah Kampman

-----Original Message-----
I'm trying to find a better way to do usability test analysis.

My current approach is: after i finish a usability study, with 8 or 10
users, and collected my own and all observers' notes, I usually read all
notes and then immediately write down the issues I feel area appearing
more often (assuming my brain will remember issues that are repeated in
many notes more than issues than appear only once).

Now, how do YOU approach analyzing those notes? Reading and re-writing
by heart? Putting all notes on a wall and eye-balling? Tagging the text
with some piece of software?

1 Jul 2008 - 3:46pm
Sarah Kampman
2008

I'd be happy to elaborate -- and I'd love feedback, as this is something I'm always trying to streamline and improve.

I try to identify the items under investigation ahead of time, so that I can mark up a prewritten test script during the test. My usability tests are often short and target a small number of issues, so that the results are manageable and my DHTML-based "moving mockups" can keep up with the expected variations they'll need to handle. I'd rather perform three short tests than one unwieldy one.

Here's what a sample moderator script looks like. Notice that the expected problem areas are identified ahead of time, with space for notes. It's much faster for me to work from my notes, and they're essential if I can't record the session for any reason.

1. Please log into the system using username: ME and password: PW.
[Clock stops when they click Log In. Time: ____]
[Username typo? Y/N]
[Password typo? Y/N]
[Clicked the correct button on first try? Y/N]

2. Please create a new login for John Smith.
[Clock stops when they click Save at the end. Time:____]
[Correct navigation? Y/N -- If N, where to first?_____]
[Required fields entered? Y/N -- If N, which weren't?_____]
[Default password >6 char? Y/N]

I hope that helps.
-Sarah

-----Original Message-----
Could you explain further how you take those measures. e.g. how do you
take time for tasks (whole tasks, parts...?), what other metrics you
look into, how you measure success ("yes/no", "yes but...", etc.

Thanks!

Sarah Kampman escribió:
> Though I take notes, I rely more heavily on quantitative measures when
> assessing the results of a usability test. All of the tasks I have
> participants complete have degrees of success, and often a time
> component as well. These measures lend themselves to comparison and
> analysis in a way that quotes do not. This is important for me, as
> usability testing only part of what I do, and I don't have the time to
> write out transcripts. The easier & faster I can make analysis, the
> better.
>
> As for the comments that I record, they fall into two categories:
> marketing and feedback. The "marketing" quotes are used to make a point
> internally, often to help position a change/enhancement as meaningful to
> a particular persona. The "feedback" quotes I use to fix whatever was
> problematic in the usability test, and I'll typically have my mockups up
> in Dreamweaver as I go through the feedback notes so that I can make the
> needed changes immediately.
>
> -Sarah Kampman
>
>
> -----Original Message-----
> I'm trying to find a better way to do usability test analysis.
>
> My current approach is: after i finish a usability study, with 8 or 10
> users, and collected my own and all observers' notes, I usually read all
> notes and then immediately write down the issues I feel area appearing
> more often (assuming my brain will remember issues that are repeated in
> many notes more than issues than appear only once).
>
> Now, how do YOU approach analyzing those notes? Reading and re-writing
> by heart? Putting all notes on a wall and eye-balling? Tagging the text
> with some piece of software?
>

--
Guillermo Ermel
Responsable de usabilidad
MercadoLibre.com

1 Jul 2008 - 5:43pm
Anonymous

Erin, your method "cluster analysis with comments on post-its"
sounds effective, but I wasn't able to get a handle on the process
as you described it. I think I have a 1/2 picture of it. Could you
break it down a bit more for a newbie?

Thanks for everyone's postings, such a valuable dialog!

>t.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=30937

1 Jul 2008 - 8:36pm
Todd Warfel
2003

On Jul 1, 2008, at 10:34 AM, Guillermo Ermel wrote:

> Now, how do YOU approach analyzing those notes? Reading and re-
> writing by heart? Putting all notes on a wall and eye-balling?
> Tagging the text with some piece of software?

We use a custom framework we've developed that was inspired by ELITO
(used at IIT). Each observation is tagged and can have artifacts and
concepts (design solution) attached to them. We run analysis through
this framework, currently in a spreadsheet, looking for patterns.
Additionally, each observation gets a significance rating of 1-5,
along with a judgement (why the business cares about it) of 1-5, and a
technical feasibility score of 1-5 for the design solution (concept).
These scores go into a weighted formula that produces a priority
rating at the end.

This gives us our observations, design solutions, and prioritization
for the business on which items to address first.

We've built an internal prototype of the framework as a Rails app,
which should make the analysis much faster.

Cheers!

Todd Zaki Warfel
President, Design Researcher
Messagefirst | Designing Information. Beautifully.
----------------------------------
Contact Info
Voice: (215) 825-7423
Email: todd at messagefirst.com
AIM: twarfel at mac.com
Blog: http://toddwarfel.com
Twitter: zakiwarfel
----------------------------------
In theory, theory and practice are the same.
In practice, they are not.

2 Jul 2008 - 5:30am
Chauncey Wilson
2007

This is an excellent topic and a critical one for our profession. It
would make for a good workshop or presentation at UPA, or CHI, or
IxDA. There are some good issues with taking notes of usability
sessions:

1. Many note taking sessions focus on point problems -- the person
chooses the wrong menu item, but problems that occur over a longer
interval may be harder to extract from notes. For example, a person
might do something early in a test that results in an incorrect result
later, but the initial action that started the chain of events may not
be obvious to anyone.
2. Watching videos is time consuming, but whenever I watch (or skim)
the videos, I see more problems and also note actions that were't
coded as problems (like a person spending 5 seconds looking for
something) but that might indicate a usability issue. In one test I
tried about 6 years ago, I had watched a tape of a session and listed
problems and compared them with the problems recorded by multiple
notetakers and I found 50% more problems.
3. I think that asking observers to record problems can be
problematic :-). I like to do some training to cue observers about
what things might constitute a "problem".
4. Most of us track issues for a single study, but sometimes the more
important issues might only show up in a meta-analysis across multiple
studies (using different methods).
5. What is the difference in noting problems if you see the person's
face versus just seeing the interaction with the software? I think
that there is some research that seeing a person's face will result in
problems being rated as more serious on average. I have noticed that
people who aren't watching the person note more details than observers
who actually see the person's face.
6. How you do deal with repeated examples of the same problem within
a session and across sessions. What is someone has the same problem 4
times in one session .... is that one problem or 4 instances of the
same problem?
7. What levels of granularity are represented in problem reports.
This is a big issue in notetaking. Some people might report problems
at a high-level of granularity ("Person has problem with dialog box
X") while others may report something much more specific ("Person has
problem with filtering UI in dialog box X").

I think that this is an area that should receive more attention in our
field. Thanks for this posting and responses.

Chauncey

On Tue, Jul 1, 2008 at 3:30 PM, Sarah Kampman <skampman at planview.com> wrote:
> Though I take notes, I rely more heavily on quantitative measures when
> assessing the results of a usability test. All of the tasks I have
> participants complete have degrees of success, and often a time
> component as well. These measures lend themselves to comparison and
> analysis in a way that quotes do not. This is important for me, as
> usability testing only part of what I do, and I don't have the time to
> write out transcripts. The easier & faster I can make analysis, the
> better.
>
> As for the comments that I record, they fall into two categories:
> marketing and feedback. The "marketing" quotes are used to make a point
> internally, often to help position a change/enhancement as meaningful to
> a particular persona. The "feedback" quotes I use to fix whatever was
> problematic in the usability test, and I'll typically have my mockups up
> in Dreamweaver as I go through the feedback notes so that I can make the
> needed changes immediately.
>
> -Sarah Kampman
>
>
> -----Original Message-----
> I'm trying to find a better way to do usability test analysis.
>
> My current approach is: after i finish a usability study, with 8 or 10
> users, and collected my own and all observers' notes, I usually read all
> notes and then immediately write down the issues I feel area appearing
> more often (assuming my brain will remember issues that are repeated in
> many notes more than issues than appear only once).
>
> Now, how do YOU approach analyzing those notes? Reading and re-writing
> by heart? Putting all notes on a wall and eye-balling? Tagging the text
> with some piece of software?
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

1 Jul 2008 - 2:51pm
Guillermo Ermel
2008

Could you explain further how you take those measures. e.g. how do you
take time for tasks (whole tasks, parts...?), what other metrics you
look into, how you measure success ("yes/no", "yes but...", etc.

Thanks!

Sarah Kampman escribió:
> Though I take notes, I rely more heavily on quantitative measures when
> assessing the results of a usability test. All of the tasks I have
> participants complete have degrees of success, and often a time
> component as well. These measures lend themselves to comparison and
> analysis in a way that quotes do not. This is important for me, as
> usability testing only part of what I do, and I don't have the time to
> write out transcripts. The easier & faster I can make analysis, the
> better.
>
> As for the comments that I record, they fall into two categories:
> marketing and feedback. The "marketing" quotes are used to make a point
> internally, often to help position a change/enhancement as meaningful to
> a particular persona. The "feedback" quotes I use to fix whatever was
> problematic in the usability test, and I'll typically have my mockups up
> in Dreamweaver as I go through the feedback notes so that I can make the
> needed changes immediately.
>
> -Sarah Kampman
>
>
> -----Original Message-----
> I'm trying to find a better way to do usability test analysis.
>
> My current approach is: after i finish a usability study, with 8 or 10
> users, and collected my own and all observers' notes, I usually read all
> notes and then immediately write down the issues I feel area appearing
> more often (assuming my brain will remember issues that are repeated in
> many notes more than issues than appear only once).
>
> Now, how do YOU approach analyzing those notes? Reading and re-writing
> by heart? Putting all notes on a wall and eye-balling? Tagging the text
> with some piece of software?
>

--
Guillermo Ermel
Responsable de usabilidad
MercadoLibre.com

3 Jul 2008 - 9:02am
Erin Walsh
2007

Doh! I should have said affinity diagram with clustered results of
sites features and functions. I apologize for mistyping and causing
confusion!

We used Morae to perform the tests, so we initially spent time
logging comments, observations and errors. Once complete we had a
giant spreadsheet of the individual issues, quotes, etc. The
affinity idea sprang from our previous success with (and love of) the
K-J Technique (http://www.uie.com/articles/kj_technique/).

Once we had our grouped clusters we brought in some ideas from the
Mental Model virtual seminar Indi Young did with UIE. While not a
formal mental model, we pulled in quotes from user and stakeholder
interviews to see how they lined up with our usability test result
diagram. Under all of it we drew our line and then pulled in the
competitive analysis of who in our industry delivered each feature/
function in our clusters, regardless if it was utilized in the test.

If still interested, please write and I send you a cleaned up version
since it is difficult to conceptualize due to the melding of so many
different methods.

Sorry about the earlier confusion!
Erin

Syndicate content Get the feed