qualitative and quantitative research

30 May 2008 - 11:35am
6 years ago
13 replies
1987 reads
christine chastain
2008

Good morning!

Does anyone have any experience with "making" qualitative research as
quantitative as possible? This is a rather nebulous idea in my mind for the
moment but I was thinking about something like using images from still or
video footage and tagging those such that they could be coded in a
quantitative way. Sort of like a heuristic evaluation on steroids.

Just thinking...thanks!

Comments

30 May 2008 - 1:30pm
Christine Boese
2006

Generally (and forgive me if you are already familiar with these methods)
the way to go is with methods of content analysis such as sociolinguists
use. I always used to think part of the beauty of Q-sort methods is that
they have wonderful open-ended approaches, yet the data can be crunched.

But this is an area where invention and inventiveness could really push on
boundaries of what is possible. The problem is, most researchers tend to
have either a quantitative or qualitative bias. Multi-modal stuff is
interesting, tho! Triangulate!

My own issues with sociolinguistic-style content analysis data (which is
what you get when you take oral speak-aloud transcripts from video usability
testing and have independent raters code the content according to a
particular schema, checking for inter-rater reliability) is that it often
turns into "counts" of certain keywords that show frequency, but fail to
reveal true emphasis or weight.

For instance (and I'm dating myself here), look at a content analysis study
of the discourse of an online forum or some other social or dialogic
context. You code and count, crunch in SPSS etc and what can be revealed is
a map of sorts of the ways that that conversation or dialogue went down.

But the blind spot is the ethos of the different speakers, known to the
other speakers. So while many speakers may be chattering away in the forum,
generating lines and lines of text, this one speaker can enter the
conversation with just a few well-chosen lines and turn the direction of the
entire discussion, or shut down one line of argument entirely with a pithy
or significant counter-argument. A counting-style analysis of qualitative
content would completely miss the emphasis and power of these few
well-chosen and influential statements.

Now, go visual, in a less social interface design example. Supposed you are
looking at one of those fun "red dot" studies on a web page, red dot
representing mouse actions. Great data. Fascinating to look at. Or the
eyeball studies, that show where eyeballs go on a page. In many ways, these
are attempts to quantify what is essentially idiosyncratic, qualitative user
mouse and eyeball behaviors. And they keep getting better. You could even do
studies with these tools that are completely phenomenological, as open-ended
and qualitative as you could possibly get.

But they end up coming back to counting something, and you can usually find
blind spots there too, where quantity does not reveal the real strength and
emphasis that should be put on a single under-counted thing, something that
works in the gestalt of the experience, or sits on the periphery, and yet
strongly influences and colors the experience far out of proportion to
anything being counted. For some sites, that might be a compelling (not
lame) interface metaphor. Others, it might be a style of photography that
just makes the keyed image dazzling far out of proportion to anything else
on the site. Or it might be an unenunciated emotional effect, a common
reaction to a combination of colors on a particular community site that
makes it feel like "home" to the people who come to live there, so that when
those colors are changed, the community inexplicably loses its energy and
center.

Anyway, I'm just throwing this out there, for something to think about.

Chris

On Fri, May 30, 2008 at 12:35 PM, christine chastain <
chastain.christine at gmail.com> wrote:

> Good morning!
>
> Does anyone have any experience with "making" qualitative research as
> quantitative as possible? This is a rather nebulous idea in my mind for the
> moment but I was thinking about something like using images from still or
> video footage and tagging those such that they could be coded in a
> quantitative way. Sort of like a heuristic evaluation on steroids.
>
> Just thinking...thanks!
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

30 May 2008 - 2:43pm
lachica
2006

Change Sciences attempts to plug qualitative measures into a quantitative
formula. They have a whitepaper that describes the process:
http://www.changesciences.com/

Best,
Julie

On Fri, May 30, 2008 at 11:35 AM, christine chastain <
chastain.christine at gmail.com> wrote:

> Good morning!
>
> Does anyone have any experience with "making" qualitative research as
> quantitative as possible? This is a rather nebulous idea in my mind for the
> moment but I was thinking about something like using images from still or
> video footage and tagging those such that they could be coded in a
> quantitative way. Sort of like a heuristic evaluation on steroids.
>
> Just thinking...thanks!
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

30 May 2008 - 1:40pm
christine chastain
2008

Thanks for your thoughtful input, Chris! Being a designer and ethnographer,
I think what I have in my head is actually tagging behaviors, interactions,
even materiality in video or other self-reported data and then coming up
with a way to code those that would show patterns over time. So imagine
taking all visual materials from a time-motion study, for example, and
tagging all behaviors, interactions and things, feeding that into some
magical formula that would allow you to cross tab and identify patterns from
which you could then produce a lifestyle narrative. So awesome...time
consuming but could be amazing. And you could still have "other" qualitative
input overlayed onto that.

On Fri, May 30, 2008 at 11:30 AM, Christine Boese <christine.boese at gmail.com>
wrote:

> Generally (and forgive me if you are already familiar with these methods)
> the way to go is with methods of content analysis such as sociolinguists
> use. I always used to think part of the beauty of Q-sort methods is that
> they have wonderful open-ended approaches, yet the data can be crunched.
>
> But this is an area where invention and inventiveness could really push on
> boundaries of what is possible. The problem is, most researchers tend to
> have either a quantitative or qualitative bias. Multi-modal stuff is
> interesting, tho! Triangulate!
>
>
>

30 May 2008 - 5:30pm
Jared M. Spool
2003

On May 30, 2008, at 12:35 PM, christine chastain wrote:

> Does anyone have any experience with "making" qualitative research as
> quantitative as possible? This is a rather nebulous idea in my mind
> for the
> moment but I was thinking about something like using images from
> still or
> video footage and tagging those such that they could be coded in a
> quantitative way. Sort of like a heuristic evaluation on steroids.

We've been doing it for years. Most of our research is based on it.
It's hard work, but quite doable.

At one point, we had a sign in the office that read: "Scientific
advancement through mind-numbing manual labor."

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: jspool at uie.com p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks

30 May 2008 - 6:10pm
gretchen anderson
2005

Tables. Percentages. Pie Charts. I'm serious.

Sometimes I refer to this as being "scientistic".

I always approach research problems with a somewhat scientific framework
(coding activities, keeping tallies). It's just important to be clear
with yourself that the samples sizes you are dealing with generally mean
that true quant claims are meaningless. And, heck even dangerous,
sometimes.

That said, it's especially helpful for skeptical clients who are quant
jocks. Just make sure to let anyone on the project who actually gets the
difference know what's up.

-----Original Message-----
From: discuss-bounces at lists.interactiondesigners.com
[mailto:discuss-bounces at lists.interactiondesigners.com] On Behalf Of
Jared Spool
Sent: Friday, May 30, 2008 3:31 PM
To: christine chastain
Cc: discuss at ixda.org
Subject: Re: [IxDA Discuss] qualitative and quantitative research

On May 30, 2008, at 12:35 PM, christine chastain wrote:

> Does anyone have any experience with "making" qualitative research as
> quantitative as possible? This is a rather nebulous idea in my mind
> for the
> moment but I was thinking about something like using images from
> still or
> video footage and tagging those such that they could be coded in a
> quantitative way. Sort of like a heuristic evaluation on steroids.

We've been doing it for years. Most of our research is based on it.
It's hard work, but quite doable.

At one point, we had a sign in the office that read: "Scientific
advancement through mind-numbing manual labor."

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: jspool at uie.com p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks

________________________________________________________________
Welcome to the Interaction Design Association (IxDA)!
To post to this list ....... discuss at ixda.org
Unsubscribe ................ http://www.ixda.org/unsubscribe
List Guidelines ............ http://www.ixda.org/guidelines
List Help .................. http://www.ixda.org/help

30 May 2008 - 8:45pm
Itamar Medeiros
2006

If you're talking about how to turn qualitative data into something
"measureable", there are several etnography methods now that are
moving in that direction.

You should check out -- for example -- the "Insight Matrix",
developed by Vijay Kumar and Brandon Schauer at the IIT Institute of
Design (http://www.id.iit.edu/568/).

{ Itamar Medeiros } Information Designer
http://designative.info/
http://www.autodesk.com/

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=29646

30 May 2008 - 9:02pm
Christine Boese
2006

Hi Christine,

Yup, that sort of tagging is just like more formal methods of content
analysis, such as like socio-linguists use. You might pick up one of those
small paperback Sage publications that run through an overview of content
analysis methods, just so you are rigorous in your tags and how you assign
them, or perhaps look to multiple raters/coders/taggers and establish a
baseline of inter-rater reliability.

See, the trouble with winging it is you could spend a lot of time on a
taggin/coding schema, and you might discover (or worse, not discover and be
oblivious) to the fact that your schema is giving you bad data, which then
becomes bad conclusions. That would be a nightmare.

So you'd want a really STRONG pilot project, and lots and lots of feedback
to make sure the method will yield both reliable and useful results. Hone
the method out, THEN turn it into your hamburger grinder and see what kind
of burgers you get.

And then trot your great new method out at the next IxDA conference and tell
us all about it, so we can try it too, and replicate it, and further test
the usefulness of the results!

I don't say this to put you off by the amount of work entailed. Following
vigorous and careful research methods may seem like a huge mountain to
climb, but it can also be creative, interesting, and really valuable. I'm
hoping you do it! We could use some fresh and innovative methods.

Chris

On Fri, May 30, 2008 at 2:40 PM, christine chastain <
chastain.christine at gmail.com> wrote:

> Thanks for your thoughtful input, Chris! Being a designer and ethnographer,
> I think what I have in my head is actually tagging behaviors, interactions,
> even materiality in video or other self-reported data and then coming up
> with a way to code those that would show patterns over time. So imagine
> taking all visual materials from a time-motion study, for example, and
> tagging all behaviors, interactions and things, feeding that into some
> magical formula that would allow you to cross tab and identify patterns from
> which you could then produce a lifestyle narrative. So awesome...time
> consuming but could be amazing. And you could still have "other" qualitative
> input overlayed onto that.
>
>
> On Fri, May 30, 2008 at 11:30 AM, Christine Boese <
> christine.boese at gmail.com> wrote:
>
>> Generally (and forgive me if you are already familiar with these methods)
>> the way to go is with methods of content analysis such as sociolinguists
>> use. I always used to think part of the beauty of Q-sort methods is that
>> they have wonderful open-ended approaches, yet the data can be crunched.
>>
>> But this is an area where invention and inventiveness could really push on
>> boundaries of what is possible. The problem is, most researchers tend to
>> have either a quantitative or qualitative bias. Multi-modal stuff is
>> interesting, tho! Triangulate!
>>
>>
>>
>

30 May 2008 - 9:25pm
christine chastain
2008

I'm going to give it a go...;) I mean, how much worse can it be than a
time-motion study of knee replacements including all players using various
surgical methods for the purpose of creating new instrumentation? I'm a
glutton for punishment...

Your suggestions are great!

On Fri, May 30, 2008 at 7:02 PM, Christine Boese <christine.boese at gmail.com>
wrote:

> Hi Christine,
>
> Yup, that sort of tagging is just like more formal methods of content
> analysis, such as like socio-linguists use. You might pick up one of those
> small paperback Sage publications that run through an overview of content
> analysis methods, just so you are rigorous in your tags and how you assign
> them, or perhaps look to multiple raters/coders/taggers and establish a
> baseline of inter-rater reliability.
>
> See, the trouble with winging it is you could spend a lot of time on a
> taggin/coding schema, and you might discover (or worse, not discover and be
> oblivious) to the fact that your schema is giving you bad data, which then
> becomes bad conclusions. That would be a nightmare.
>
> So you'd want a really STRONG pilot project, and lots and lots of feedback
> to make sure the method will yield both reliable and useful results. Hone
> the method out, THEN turn it into your hamburger grinder and see what kind
> of burgers you get.
>
> And then trot your great new method out at the next IxDA conference and
> tell us all about it, so we can try it too, and replicate it, and further
> test the usefulness of the results!
>
> I don't say this to put you off by the amount of work entailed. Following
> vigorous and careful research methods may seem like a huge mountain to
> climb, but it can also be creative, interesting, and really valuable. I'm
> hoping you do it! We could use some fresh and innovative methods.
>
> Chris
>
>
> On Fri, May 30, 2008 at 2:40 PM, christine chastain <
> chastain.christine at gmail.com> wrote:
>
>> Thanks for your thoughtful input, Chris! Being a designer and
>> ethnographer, I think what I have in my head is actually tagging behaviors,
>> interactions, even materiality in video or other self-reported data and then
>> coming up with a way to code those that would show patterns over time. So
>> imagine taking all visual materials from a time-motion study, for example,
>> and tagging all behaviors, interactions and things, feeding that into some
>> magical formula that would allow you to cross tab and identify patterns from
>> which you could then produce a lifestyle narrative. So awesome...time
>> consuming but could be amazing. And you could still have "other" qualitative
>> input overlayed onto that.
>>
>>
>> On Fri, May 30, 2008 at 11:30 AM, Christine Boese <
>> christine.boese at gmail.com> wrote:
>>
>>> Generally (and forgive me if you are already familiar with these methods)
>>> the way to go is with methods of content analysis such as sociolinguists
>>> use. I always used to think part of the beauty of Q-sort methods is that
>>> they have wonderful open-ended approaches, yet the data can be crunched.
>>>
>>> But this is an area where invention and inventiveness could really push
>>> on boundaries of what is possible. The problem is, most researchers tend to
>>> have either a quantitative or qualitative bias. Multi-modal stuff is
>>> interesting, tho! Triangulate!
>>>
>>>
>>>
>>
>

30 May 2008 - 9:32pm
Paul Eisen
2007

Christine asked:
> Does anyone have any experience with "making" qualitative research as quantitative as possible?

I'm not sure if this is what you are looking for: Fuzzy logic is a mathematics that has been used to model imprecise values. It allows you to apply values from 0 to 1 to constructs that a researcher may be interested in such as, "Easy", "Engaging", etc. Fuzzy logic has been used with varying levels of success to represent subjective impressions, as we often like to do in user research. Check out Wikipedia if you want to learn more. A Google search will lead you to some examples of the use of fuzzy logic in usability-related research.

Regards,

Paul

Paul Eisen
Principal User Experience Architect
tandemseven

31 May 2008 - 12:09pm
Jeff Howard
2004

christine wrote:
> Does anyone have any experience with "making" qualitative
> research as quantitative as possible?

It is not worth the while to go round the world to count the cats in
Zanzibar. --Thoreau

That being said, a couple books come to mind with examples of this
type of research. William Whyte wrote a short book called the Social
Life of Small Urban Spaces and Paco Underhill (who worked for Whyte
on the Project for Public Spaces) wrote Why We Buy and calls his
approach "the science of shopping."

They're both engaged in some pretty serious ethnography (IIRC,
primarily shadowing and video analysis) and neither skimp on the
quantification.

// jeff

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=29646

31 May 2008 - 4:56pm
Mark Schraad
2006

As Jeff eludes, It is a mistake to put ethnography solely in the
qualitative box. Agar, Maanen and many of the other early
ethnographers took meticulous counts and quantified their findings.

On May 31, 2008, at 10:09 AM, Jeff Howard wrote:

> They're both engaged in some pretty serious ethnography (IIRC,
> primarily shadowing and video analysis) and neither skimp on the
> quantification.

31 May 2008 - 9:04pm
Formulate
2007

Gosh I'm loving the way IxDA is getting input from all sorts of other
fields. This forum rocks!

I'm have a market research background and what you describe sounds
to me like qualitative coding. If you do a search for this term, you
will find heaps of references for:
- software to assist the process (usually assuming you're working
from transcripts of interviews or focus groups, so probably not so
useful for you)
- papers and books about how to do it
- papers and books about its reliability and usefulness.

My experience in market research was that the subjective nature of
qualitative coding was recognised and mechanisms were put in place to
manage it. These included ensuring the same person or people coded,
that the coding frame was discussed and reviewed by all parties, and
that a random sample of coded information was quality checked to look
for glaring inconsistencies in how the frame was applied.

The thing to remember is that if you want to draw conclusions about
patterns you are observing, then you need to be working with a
complete or at least statistically representative sample. This is
usually *not* the case in qualitative market research but it might be
in your situation.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=29646

9 Jun 2008 - 1:49pm
John D'Ignazio
2008

This is called content analysis. It's a recognized research method that was
developed for texts but has been applied to video or images. The best
strategy for consistent coding in content analysis is to create a codebook
of what it is your looking for and keep to it.

Good luck!

On Fri, May 30, 2008 at 12:35 PM, christine chastain <
chastain.christine at gmail.com> wrote:

> Good morning!
>
> Does anyone have any experience with "making" qualitative research as
> quantitative as possible? This is a rather nebulous idea in my mind for the
> moment but I was thinking about something like using images from still or
> video footage and tagging those such that they could be coded in a
> quantitative way. Sort of like a heuristic evaluation on steroids.
>
> Just thinking...thanks!
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

Syndicate content Get the feed