Challenge of analyzing qualitative data

6 Feb 2004 - 3:29am
10 years ago
4 replies
1741 reads
kaushikg at vsnl.net
2004

Does anyone have good resources on tools/methods/techniques of how to use the data generated out of qualitative ethnographic studies to construct appropriate formative assessment framework useful for design at the final stages of the process. I have been having some issues with this. Obviously there are many methods available, including scenarios and persona, to create valuable design insights towards a design space of possible products or services.
But then how do I make sure that I have captured each little detail of the 'thick description' generated from the primary design research into the models and constructs of the secondary phase.
In one particular instance, along with primary ethnographic studies, business and technology researches were constantly being carried out, informing each other in parallel to produce grounded, relevant and concurrent data in a short time. So we had massively saved on time and had better quality of data at the end of the fieldwork. But the challenge we faced is how to maximize the utilization of these data to create definite design 'vision's. How does this vision lead to the concrete design space of the future products or services?

Kaushik Ghosh
Sr. Researcher
Center for Ethnography & Contextual Innovation
Human Factors International

Comments

6 Feb 2004 - 10:44am
Todd Warfel
2003

Kaushik,

That's a tough question to answer. I'm interested to see how others
tackle this.

First answer the following question: Why are you doing user research in
the first place?

For us, we do research to establish business objectives, user goals,
and evaluate the desirability, feasibility, and sustainability of a
particular product, or set of features. All of these ultimately feed
into making products better and easier-to-use.

Here's our approach.
First of all, our ethnographic research flows directly into the
following:
1. User research summary report (may also include findings from
surveys, focus groups, competitive analysis)
2. Task analysis grid
3. Task flow diagrams
4. Personas

In the user research report, we try and answer some of the direct
questions the client has (e.g. adoption factors, mental model,
desirability, feasibility, sustainability). Then we look for some "I
never would have guessed that" pieces of information. For example, we
recently discovered that academia is still struggling with recruiting.
You'd think that with the dot.com bust the field would be flooded with
individuals with plenty of experience. And the field is flooded with
these types. However, while they have the content knowledge, they don't
have pedagogy (classroom management). So, many of these industry
professionals end up leaving teaching within the first month - they
can't hack it. That was a "wow, wouldn't have guessed that" piece of
info for us and the client. Schools invest a lot in recruiting. And to
have 20% of your new recruits walk off the job in the first month is
very costly.

Try and find nuggets of info that can be related to business
objectives, user goals (tasks), feasibility, etc. Those are the items
we find will have the greatest significance, impact design, and present
the greatest value.

On Feb 6, 2004, at 3:29 AM, kaushikg at vsnl.net wrote:

> But then how do I make sure that I have captured each little detail of
> the 'thick description' generated from the primary design research
> into the models and constructs of the secondary phase.

Cheers!

Todd R. Warfel
User Experience Architect
MessageFirst | making products easier to use
--------------------------------------
Contact Info
voice: (607) 339-9640
email: twarfel at messagefirst.com
web: www.messagefirst.com
aim: twarfel at mac.com
--------------------------------------
In theory, theory and practice are the same.
In practice, they are not.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: text/enriched
Size: 2719 bytes
Desc: not available
Url : http://lists.interactiondesigners.com/private.cgi/discuss-interactiondesigners.com/attachments/20040206/8ebc37ff/attachment.bin

6 Feb 2004 - 12:55pm
Peter Merholz
2004

This is an excellent question, and something I've been trying to get my
head around.

Some thoughts:

This was exactly the theme of the About, With, and For conference, held
at the Institute of Design in Chicago. The theme of the topic was
synthesizing user research. You might want to poke around the session
notes for some more ideas:
http://www.id.iit.edu/events/aboutwithandfor/awf_home.html

I wrote some notes about the conference:
http://tinyurl.com/2hule
(this points to a Google query, which in turn reveals posts from my
site)

Your questions also explicitly the subject of the talk I gave there. I
still haven't written it up (it's remarkably hard "writing" a
presentation, it turns out), but I can give you an overview of the
discussion:

(I will be presenting this material at the IA Summit,
http://iasummit.org/)

I. Work Modeling
I begin with work modeling, to which I was introduced through Beyer and
Holtzblatt, taking a course at CHI '98.
http://www.incent.com/pubs/workmodeling.html

I'm guessing that since the words "Contextual Innovation" are in your
department name, you're familiar with their work.

I appreciate their work, but criticize it for being too cumbersome for
the design projects I've taken part in.

II. Mental Model and Thematic Model

I then discuss two forms of simplified work modeling that have been
developed at Adaptive Path. Mental modeling you can learn more about
here:
http://www.uiconf.com/7west/young_interview.htm
http://www.boxesandarrows.com/archives/
rearchitecting_peoplesoft_from_the_top_down.php

I can't point you to the thematic model, because we've never written
about it, and it's a new approach. Suffice to say it's REALLY basic --
from your user research, you pull out prevailing themes. You then
diagram those themes, connecting them to form psychological narratives.

III. Looking outward

I end the talk by encouraging those of us who synthesize user research
to look outside our discipline, and see how other social scientists
approach the problem. The example that gets the most attention is
archaeologist James Deetz' seminal paper on gravestone design in
colonial America:
http://etext.lib.virginia.edu/users/deetz/Plymouth/deathshead.html

In it, Deetz applied statistical analysis to photographic data to
demonstrate trends in design. I find it encouraging, and I implore user
researchers to get out of the text-and-diagramming world and consider
other media for their research.

To respond to a particular element of your statement:
> But then how do I make sure that I have captured each little detail of
> the 'thick description' generated from the primary design research
> into the models and constructs of the secondary phase.

Not to be judgmental, but this is a foolish desire. The whole point of
research synthesis is to get out of the weeds, to not stress over "each
little detail", to develop a framework that allows for creative
insight. It's necessary to overlook many "little details" to achieve
this.

--peter

On Feb 6, 2004, at 12:29 AM, kaushikg at vsnl.net wrote:

> Does anyone have good resources on tools/methods/techniques of how to
> use the data generated out of qualitative ethnographic studies to
> construct appropriate formative assessment framework useful for design
> at the final stages of the process. I have been having some issues
> with this. Obviously there are many methods available, including
> scenarios and persona, to create valuable design insights towards a
> design space of possible products or services.
> But then how do I make sure that I have captured each little detail of
> the 'thick description' generated from the primary design research
> into the models and constructs of the secondary phase.
> In one particular instance, along with primary ethnographic studies,
> business and technology researches were constantly being carried out,
> informing each other in parallel to produce grounded, relevant and
> concurrent data in a short time. So we had massively saved on time and
> had better quality of data at the end of the fieldwork. But the
> challenge we faced is how to maximize the utilization of these data to
> create definite design 'vision's. How does this vision lead to the
> concrete design space of the future products or services?
>
> Kaushik Ghosh
> Sr. Researcher
> Center for Ethnography & Contextual Innovation
> Human Factors International
>
>
> _______________________________________________
> Interaction Design Discussion List
> discuss at interactiondesigners.com
> --
> to change your options (unsubscribe or set digest):
> http://discuss.interactiondesigners.com
> --
> Questions: lists at interactiondesigners.com
> --
> Announcement Online List (discussion list members get announcements
> already)
> http://interactiondesigners.com/announceList/
> --
> http://interactiondesigners.com/
>

6 Feb 2004 - 6:59pm
sandeepblues
2003

Typically, affinity charts are used (you
know...writing post-its for every new point
discovered, and organizing them into contextual
clumps) to capture the main points in the thick
ethnography info.

Sandeep

11 Feb 2004 - 8:44pm
Tac Leung
2004

An excellent answer from Peter.

Our team has just completed a Phase 0 project where we unwittingly used what
Peter describes as the 'thematic model' and it was very effective.

> I can't point you to the thematic model, because we've never written
> about it, and it's a new approach. Suffice to say it's REALLY basic --
> from your user research, you pull out prevailing themes. You then
> diagram those themes, connecting them to form psychological narratives.

The project involved rethinking an enterprise application management tool
from the ground up. The client wanted to change the way that not just a
particular role, but how entire organizations did their work. One of the
major challenges of the project was how to devise interfaces for roles and
tasks that did not yet exist.

After conducting contextual inquiry and interviews with the target users
(who spanned at least 4 distinct job roles), we constructed personas and
scenarios. In this case, the software product we are developing aims to
redistribute role responsibilities: creating a mental model for each role
seemed inappropriate. Instead, we distilled the high-level goals for the
entire target group into 5 main themes. We then designed interface concepts
to support these themes, which included organization-wide goals such as
"Pattern matching" and "Exception management" and refined them with users in
participatory design sessions.

Abstracting these goals from the granular experience of specific user
allowed us to clearly set the functional direction of the application while
avoiding rat holing over particular user preferences. Creating role-specific
narratives interfaces then became a simple issue of mapping the needs and
scenarios associated with each user type to the functional groups.

We have an application that is radically different than the original and at
the same time clearly supports the tasks of the current and future users.

I highly recommend this process for designing products or services that aim
to significantly change the responsibilities or task flow of your target
users.

Cheers,
Tac

> From: Peter Merholz <peterme at peterme.com>
> Date: Fri, 6 Feb 2004 09:55:00 -0800
> To: discuss at interactiondesigners.com
> Subject: Re: [ID Discuss] Challenge of analyzing qualitative data
>
> This is an excellent question, and something I've been trying to get my
> head around.
>
> Some thoughts:
>
> This was exactly the theme of the About, With, and For conference, held
> at the Institute of Design in Chicago. The theme of the topic was
> synthesizing user research. You might want to poke around the session
> notes for some more ideas:
> http://www.id.iit.edu/events/aboutwithandfor/awf_home.html
>
> I wrote some notes about the conference:
> http://tinyurl.com/2hule
> (this points to a Google query, which in turn reveals posts from my
> site)
>
> Your questions also explicitly the subject of the talk I gave there. I
> still haven't written it up (it's remarkably hard "writing" a
> presentation, it turns out), but I can give you an overview of the
> discussion:
>
> (I will be presenting this material at the IA Summit,
> http://iasummit.org/)
>
> I. Work Modeling
> I begin with work modeling, to which I was introduced through Beyer and
> Holtzblatt, taking a course at CHI '98.
> http://www.incent.com/pubs/workmodeling.html
>
> I'm guessing that since the words "Contextual Innovation" are in your
> department name, you're familiar with their work.
>
> I appreciate their work, but criticize it for being too cumbersome for
> the design projects I've taken part in.
>
> II. Mental Model and Thematic Model
>
> I then discuss two forms of simplified work modeling that have been
> developed at Adaptive Path. Mental modeling you can learn more about
> here:
> http://www.uiconf.com/7west/young_interview.htm
> http://www.boxesandarrows.com/archives/
> rearchitecting_peoplesoft_from_the_top_down.php
>
> I can't point you to the thematic model, because we've never written
> about it, and it's a new approach. Suffice to say it's REALLY basic --
> from your user research, you pull out prevailing themes. You then
> diagram those themes, connecting them to form psychological narratives.
>
> III. Looking outward
>
> I end the talk by encouraging those of us who synthesize user research
> to look outside our discipline, and see how other social scientists
> approach the problem. The example that gets the most attention is
> archaeologist James Deetz' seminal paper on gravestone design in
> colonial America:
> http://etext.lib.virginia.edu/users/deetz/Plymouth/deathshead.html
>
> In it, Deetz applied statistical analysis to photographic data to
> demonstrate trends in design. I find it encouraging, and I implore user
> researchers to get out of the text-and-diagramming world and consider
> other media for their research.
>
> To respond to a particular element of your statement:
>> But then how do I make sure that I have captured each little detail of
>> the 'thick description' generated from the primary design research
>> into the models and constructs of the secondary phase.
>
> Not to be judgmental, but this is a foolish desire. The whole point of
> research synthesis is to get out of the weeds, to not stress over "each
> little detail", to develop a framework that allows for creative
> insight. It's necessary to overlook many "little details" to achieve
> this.
>
> --peter
>
> On Feb 6, 2004, at 12:29 AM, kaushikg at vsnl.net wrote:
>
>> Does anyone have good resources on tools/methods/techniques of how to
>> use the data generated out of qualitative ethnographic studies to
>> construct appropriate formative assessment framework useful for design
>> at the final stages of the process. I have been having some issues
>> with this. Obviously there are many methods available, including
>> scenarios and persona, to create valuable design insights towards a
>> design space of possible products or services.
>> But then how do I make sure that I have captured each little detail of
>> the 'thick description' generated from the primary design research
>> into the models and constructs of the secondary phase.
>> In one particular instance, along with primary ethnographic studies,
>> business and technology researches were constantly being carried out,
>> informing each other in parallel to produce grounded, relevant and
>> concurrent data in a short time. So we had massively saved on time and
>> had better quality of data at the end of the fieldwork. But the
>> challenge we faced is how to maximize the utilization of these data to
>> create definite design 'vision's. How does this vision lead to the
>> concrete design space of the future products or services?
>>
>> Kaushik Ghosh
>> Sr. Researcher
>> Center for Ethnography & Contextual Innovation
>> Human Factors International
>>
>>
>> _______________________________________________
>> Interaction Design Discussion List
>> discuss at interactiondesigners.com
>> --
>> to change your options (unsubscribe or set digest):
>> http://discuss.interactiondesigners.com
>> --
>> Questions: lists at interactiondesigners.com
>> --
>> Announcement Online List (discussion list members get announcements
>> already)
>> http://interactiondesigners.com/announceList/
>> --
>> http://interactiondesigners.com/
>>
>
> _______________________________________________
> Interaction Design Discussion List
> discuss at interactiondesigners.com
> --
> to change your options (unsubscribe or set digest):
> http://discuss.interactiondesigners.com
> --
> Questions: lists at interactiondesigners.com
> --
> Announcement Online List (discussion list members get announcements already)
> http://interactiondesigners.com/announceList/
> --
> http://interactiondesigners.com/
>

Syndicate content Get the feed