Planning internal usability lab

6 Jan 2011 - 6:57am
3 years ago
24 replies
1917 reads
Graham Sear
2010

We're setting up a small internal usability lab at work to use for usability testing and PET interviews. I'm planning to use 2 rooms: one for interviewer and participant and one for observer. We don't have a one-way mirror but I am intending to set up a video camera in the interviewing room to be streamed/broadcast to the observation room and screencast the laptop.

Using Morae would be my preferred route for software as I feel comfortable using it, however I would like to explore cheaper/free alternatives. Has anyone ever used any different software or have any advice on the set up I am intending to use?

Thanks

Graham

Comments

6 Jan 2011 - 7:47am
Confused Rose
2011

Sounds exactly like my set up Wink

I use Morae with Recorder / Observer in one room (I observe on my laptop during the test), and Observer in another room on an LED screen. Works perfectly well - I haven't ever felt the need to move to a one-way mirror, which I feel is more obtrusive than the webcam.

Can't advise on cheaper software alternatives though, sorry...

6 Jan 2011 - 9:40am
Jared M. Spool
2003

My suggestion: Bag the two-room set up and use a single room. It's much cheaper and you'll get more effective results.

Wrote about this awhile back:

Streamlining Usability Testing by Avoiding The Lab
http://www.uie.com/articles/streamlining_usability/

Jared

10 Jan 2011 - 9:10am
Louise Hewitt
2010

Hahaha - I had that article running through my head but had no plan do dig it out - Thanks Jared.

6 Jan 2011 - 10:40am
smitty777
2010

Hi Jared, 

I agree with your old post that you don't have to have the observation deck with all the mirrors, bells and whistles.  I also think it's a great point about having the room close to the team to allow quick visits.  However, I would feel very uncomfortable having observers in the room due to the effects of  social facilitation (http://www.psychwiki.com/wiki/Social_Facilitation).  It's funny, in that article, they even mention the experiment they did showing cockroaches with poor performance due to other roaches watching them.  

I would feel more comfortable with a setup using two conference rooms and Morae.  Keeps things quiet and anxiety free, and you still have the benefits of the local, low cost lab.  Plus, I can't count how many times I've regretted letting stakeholders into the room because they jump out of the cone of silence to offer "help" to someone who's stuck. 

Thanks,

Bill

6 Jan 2011 - 9:30pm
Jared M. Spool
2003

Hi Bill,

Social Facilitation is always a bias in usability tests because (1) there's a moderator and (2) the participants know they are being watched or recorded. A good moderator can nulify the effects of social facilitation through simple techniques that focus the participant on the task at hand and comfort them through the process.

The same is true for any problem with observers "helping." A good moderator will train and brief the observers before hand, explaining the proper protocol and behavior.

I've personally conducted hundreds of tests with observers in the room and haven't had any issues with misbehaving observers. I've trained many moderators to work the same way.

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: jspool@uie.com p: +1 978 327 5561
http://uie.com  Blog: http://uie.com/brainsparks  Twitter: @jmspool

7 Jan 2011 - 6:52am
smitty777
2010

Hi Jared, 

Yes, I completely agree that there is a degree of artificiality in conducting the normal facilitator-participant laboratory usability tests.  The issue of even having moderated vs unmoderated tests has been going on for years.  But IMHO there's a difference between a moderator and observer from the social facilitation context. First, the moderator is an active participant in the study.  There is an understood social contract between the facilitator and the participant on how they will interact with each other.  The observers only role is to watch.  Their presence cannot contribute anything positive, and you're only introducing the unnecessary risk of something negative (cell phones, talking, making the participant feel uncomfortable).  Social facilitation is also called the "audience effect" for a reason, after all.  Because of the social contract, the facilitator couldn't really be considered part of the audience, but the observers would.  Second, I would be willing to wager that the effects of social facilitation grows with the audience.  

7 Jan 2011 - 9:08am
krushford
2008

 

When I've facilitated sessions with observers in the room (limited to a handful of folks per session), I find the chance of interruption that you describe here (cell phones and talking) to be non-existent. In fact, that sort of behavior is exactly what happens in the *safety* of the separate room. I'm drawn to the accessibility and universality of the human experience and I have found that witnessing the session from the same room levels the playing field. I also get copious session notes from observers, which is a big plus and I think exposes a higher level of engagement. And, like Jared said, a good facilitator can neutralize the anxiety stemming from social facilitation. Doing it this way is more work (for me), but well worth it.

Participants can forget the observers are in the room while observers never forget the participants. That alone has helped me introduce and spread an enthusiasm for user centered design.

Cheers,

 Kaden

 

7 Jan 2011 - 9:15am
Jared M. Spool
2003

Bill,

What you describe sounds to me like poor moderation and planning.

In my experience, I don't have problems with observers. In fact, they are a positive influence on the session, as the participants love to know that people are paying attention to the frustrations and delights they're experiencing.

Planning to use a second room to compensate for poor moderation and planning skills is wrong, in my opinion.

Jared

7 Jan 2011 - 10:17am
smitty777
2010

Actually, the social facilitation effects I was talking about doesn't have anything to do with moderation at all.  I'll bet you could get the same effect with mannequins. 

9 Jan 2011 - 7:58am
Jared M. Spool
2003

I'm thinking if you have mannequins as usability testing observers, there is something seriously wrong with your process.

I don't use mannequins in my process. Instead, I take advantage of the smart, competant team members, who, when given a little instruction and encouragement, behave respectably, turning the testing session into a fount of insights and design ideas. After all, that was the purpose of doing the test in the first place.

10 Jan 2011 - 9:10am
Richard Carson
2010

I didn't read the full thread, but the idea of mannequins caught my interest. Although you're not going to get much mental intelligence from a wooden dummy, however you can flush out many ideas when thinking about proximity and location of people in 3D space. Ergonomics can benefit greatly using mannequins to flush out ideas within frozen moments in time. This could be helpful, depending on what you are designing and mostly to strengthen the ideas you've already developed on paper. But nothing is better than the use of live mannequins (real people) in usability testing.

Richard

On Jan 9, 2011, at 9:06 AM, Jared M. Spool wrote:

> I'm thinking if you have mannequins as usability testing observers, there is something seriously wrong with your process. > > I don't use mannequins in my process. Instead, I take advantage of the smart, competant team members, who, when given a little instruction and encouragement, behave respectably, turning the testing session into a fount of insights and design ideas. After all, that was the purpose of doing the test in the first place.

10 Jan 2011 - 9:10am
Chauncey Wilson
2007

This is an interesting discussion and one that I've debated for a few decades with colleagues as well.  Some additional thoughts here.   1.  In these discussions, there is often an underlying assumption that the observers in the same room can see what is going on as an individual (or in the case of co-participation, two people) work together.  I've seen some labs where the observers are looking at a large screen at an angle where the participant can see the screen or without turning a lot, the observers.  In a very few cases, there was also an overhead camera that showed the person's use of the mouse and keyboard which for some tasks, might also be important. So, a question, - for those who have observers in the same room as the participant, what have you done to allow the participants to clearly see what the user is doing or do you depend on the "thinking-aloud" as the source for feedback from the user?  One issue that might affect how successful people are would be the physical arrangement of the observers with respect to the viewing angle of the participant. 

2.  One thing that there seems to be agreement on is that observers need some kind of training so they know how to  behave in a way that doesn't unduly influence the study.  This training could also apply to observers who are in the same room in focus groups, interviews, etc.  I've put together a set of rules for observers where some rules apply whether they are observing remotely using something like GoToMeeting or Morae or another collaboration tool or observing in the same room.  Even remote viewing should consider things like derogatory comments and the use of names or other personally identifiable information outside the viewing room.  The notes that I've put together about observing participants includes things like:  quesion-asking protocol, verbal and non-verbal behaviors that might influence the participant, how do deal with the participant's leaving, introductions of the observers, the role of the observers, etc.  So one best practice would be to train observers in proper behavior, remind observers who don't exhibit proper behaviors, and reminders each time there is a session (since observing usability tests is often a rare event).  After a study, the facilitator might, apart from discussing the results, provide feedback on how well the observers did.

  3.  How large should the observation group be in face-to-face testing if you choose the observe-in-the-room approach?  I have often included another person, the "notetaker" in sessions and had the person sit behind the person and out of general sight.  The role is clear and again there are some rules for interaction between the 3 parties.  How do things differ when the group size grows larger.  I've mellowed some over the years and would consider it OK for a small and well-trained group of observers in the same room, but I would feel uncomfortable with more than a few people.

  4.  The type of "testing" can be a factor in the choice to have observers (and the moderator as well) in the same room. There are some studies that I would call "participatory design" where you would include both users and stakeholders (still with some pretty explicit rules of interaction).   5. The skill of the moderator might influence whether observers in the same room would be effective.  A person who is just learning how to moderate (something that is often considered easy when in fact, I think that it takes years to become truly proficient) might be influenced more by observers than the participant.  I don't recall a discussion of the impact of observers on the moderator, but that is something to consider.

  6.  There is a good discussion of when the moderator should be in the room or not in Dumas & Loring (2008, chapter 9). The chapter deals with a number of issues about the "choice of arrangement".  This chapter focuses mostly on the moderator but some of the considerations also apply to observers.

  7.  While there have been many anecdotes thrown around in this thread and volumns of research in social psychology about both social facilitation and social inhibition, I dug back into the usability testing literature and didn't find too much.  There was a study in the HFES journal by Barker and Biers (1994) that examined the impact of one-way mirrors and cameras and in that study, the researchers found that the presence of equipment did not have an impact on the performance of the participants or ratings of usability.  In my experience with many kinds of labs with mirrors and cameras, most, (though not all) participants quickly adapt.  

  8.  In the 1980s and early 1990s, much usability testing involved a person working in a room alone with the moderator communicating by phone or intercom.  One of the concerns about that type of study was an ethical consideration of power relationships - the white-coat research and his/her human guinea pig (if a participant ever uses that phrase, then I feel like I have failed my ethical obligations). The move of the moderator from the observation room to the testing room was sometimes thought to mitigate the power relationship somewhat.

  9.  There might be some cultural issues that dictate whether there are observers in the same room or remote.  If you have a bunch of older, serious looking stakeholders watching a young person, this could be a problem.  The make-up of the user group might dictate whether observers should be in the room or not.

  10.  While some of the discussants have noted that the presence of observers has provided many insights and has not had a demonstrable impact on the participants' feedback, I would ask if you have collected any metrics to substantiate those assumptions.  For example, did you do a follow-up survey asking participants about their reactions to the observers in the room or other aspects of the session. If there is any recent research on this topic, I would love to read it.  

  11.  Apart from the issue of observers (and/or moderator) being in the same room or not is the issue about how we do think-aloud testing in general and how thinking aloud influences the results of usability testing.  There are a few studies that look at that issue.  There are several studies showing that think-aloud studies produce more "problems" than studies where the participant does not think-aloud. 

  12.  A very general issue here is what contributes to the artificiality of the situation.  Compared to working on tasks at work or home, there are many things that contribute to the artificiality of usability testing.  There was a move in the late 1980s to try and make usability testing less artificial by making labs less sterile and even introducing background noise.  A group of observers will generally not make the situation any more realistic and there generally isn't a moderator around at home or work so we need to think about that as well.  There are techniques now like letting people buy something they really want, asking people to bring in their own files, and generating tasks based on a pre-study interview, but those methods have some drawbacks as well.

  Good topic, good discussion.  To me, I think that a limited number of observers, with training, and led by an experienced moderator in an environment where the physical arrangement is appropriate, would be OK.  But, there are some kinds of research questions where I would want the person to be working with a moderator only and yet others where I would want the person to work without observers or moderator.  I support a thoughtful "it depends on the context" approach here.

  Chauncey          
  On Sun, Jan 9, 2011 at 9:10 AM, Jared M. Spool <jspool@uie.com> wrote:

I'm thinking if you have mannequins as usability testing observers, there is something seriously wrong with your process.

I don't use mannequins in my process. Instead, I take advantage of the smart, competant team members, who, when given a little instruction and encouragement, behave respectably, turning the testing session into a fount of insights and design ideas. After all, that was the purpose of doing the test in the first place.

(((Please leave all content below this line)))
___________________________________________
10 Jan 2011 - 10:40am
smitty777
2010

Hi Jared -

That's a pretty funny idea, but I really feel like your missing my point.  The fact is we have a lot of research proving observers in the room effects performance. Your earlier post stated that "a good moderator can nulify [sic] the effects of social facilitation".  I really didn't know this was possible - can you post any citations to back this up?   

10 Jan 2011 - 9:10am
Louise Hewitt
2010

You know, cutting costs is a plus too. 1 room to hire, no additional moderators required (you weren't seriously thinking you could leave them alone in that observation room, were you?).

BTW, I don't think anyone is suggesting that you outright shouldn't  have a separate room - just that if it is easier/more viable to have observers in the same room that is perfectly OK and may even have some benefits in itself.

Lou.

9 Jan 2011 - 12:21pm
Paul Eisen
2007

Jared, I agree that training and briefing the observers in the proper protocol generally works. However, occasionally you get the client stakeholder, usually a somewhat arrogant executive who flies in and out at will, doesn't have time to attend the briefing, and even if he or she did, would probably not bother to play by the rules anyway. When I sense the risk of this in an upcoming usability assessment, I will ensure a safe remote observation area is available.

How do you handle this situation?

By the way, the other side of the coin that I've experienced is that motivated and well-briefed observers will enthusiastically log issues and remember the participant behavior regardless of their location.

10 Jan 2011 - 9:11am
willdonovan
2009

I jump in with some tech suggestions, and I did this and other setups before I had a chance to get my hands on Morae.

Morae can be expensive and I agree with Chris McLay, the networking thing can be very tedious.
One way to save on cost is Silverback or any screen capture package and do some movie editing in a package like iMovie. It's very easy to use.
As for the second room, use a dual screen with the second screen mirroring the other and get a long cable. I have used the video streaming idea. 
UStream Producer is a free package you can download and have a private, one-way streaming conference to broadcast to others. The observers can also make comments and chat in the background to integrate Josie Scott's idea of anyone leaving notes.  

As for comments by anyone, a chat package or even something like yammer for private short comments along a timeline for a social idea off the top of my head that anyone in the company can watch. The group feature can be utilised in Yammer for those who want to follow the stream of comments and re-use later.
I look forward to hearing how you go.

William Donovan
t, fb, in, b: @willdonovanhttp://www.willdonovan.com.au/


On 10 January 2011 06:06, Paul Eisen <peisen@tandemseven.com> wrote:

Jared, I agree that training and briefing the observers in the proper protocol /generally/ works. However, occasionally you get the client stakeholder, usually a somewhat arrogant executive who flies in and out at will, doesn't have time to attend the briefing, and even if he or she did, would probably not bother to play by the rules anyway. When I sense the risk of this in an upcoming usability assessment, I will ensure a safe remote observation area is available.

How do you handle this situation?

By the way, the other side of the coin that I've experienced is that motivated and well-briefed observers will enthusiastically log issues and remember the participant behavior regardless of their location.

((
14 Jan 2011 - 10:05am
Richard Carson
2010

Hi Folks,

I was looking into the future of reading of all my essential Interaction Design books. I'm not sure where it stands today because I do not own a reading device yet. Since I'm in between living, all my books are in storage for a few months. It would make sense for me to have a reading device and have my entire library where I can access it at all times. Paying the money for storage just does not make sense as half of my storage is books, they're heavy and I have to go back and forth to get just one book. My question is... can I get all these books on these devices yet?

Richard Carson

10 Jan 2011 - 9:10am
James Page
2008

Of course testing is better than no testing, but context is important. 
Context, and not just Social Facilitation has a very large effect on behavior. Sabrina Mach has a presentation she did last year at Ux Brighton 2010 on how context effects behavior http://www.slideshare.net/webnographer/uxbrighton2010-howcontextaffectsbehaviour-5206261
She started to get curious about this when she did a very large scale Remote Usability test using our tool Webnographer with over 2000 participants, and another test was carried out by a researcher visiting peoples homes. In the home visits, nearly every participant blamed themselves, in the remote study the participants blamed the software been tested. We call this the myth of self blaming user. 
Before anybody starts saying that this effect is caused by poor moderation, we have done multiple studies with multiple moderators (from in house to some large well known agencies) and this pattern repeats it self. 
One of the slides on the presentation about Brazilian Street kids, Sabrina expands in this blog post. http://blog.webnographer.com/2010/11/why-exams-mean-nothing-out-of-context/
All the best
Jamesblog: blog.webnographer.com
On 7 January 2011 03:42, Jared M. Spool <jspool@uie.com> wrote:

Hi Bill,

Social Facilitation is always a bias in usability tests because (1) there's a moderator and (2) the participants know they are being watched or recorded. A good moderator can nulify the effects of social facilitation through simple techniques that focus the participant on the task at hand and comfort them through the process.

The same is true for any problem with observers "helping." A good moderator will train and brief the observers before hand, explaining the proper protocol and behavior.

I've personally conducted hundreds of tests with observers in the room and haven't had any issues with misbehaving observers. I've trained many moderators to work the same way.

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: jspool@uie.com p: +1 978 327 5561
http://uie.com  Blog: http://uie.com/brainsparks  Twitter: @jmspool

(((Please leave all content below this lin
6 Jan 2011 - 10:48am
smitty777
2010

Hi Graham, 

Sorry - I fired off that message without even getting to your question.  I've used the Morae/two room solution with great results. There's really not much to it - it almost sets itself up.  You should probably have someone to run the Observer software to take notes and make sure the system is running from the observation room.  Getting the IPs talking to each other if you have DHCP can be very tricky if you don't know what you're doing as well, so make sure you have a techie there to troubleshoot if you're unsure.   Also, do a dry run the day before - mandatory. 

One other cheaper solution I've used is Webex as well.  The advantage here is that you can broadcast the sessions right to any number of people's desks. I've used this in conjunction with Morae Observer to broadcast out the observation feed. 

Hope this helps, 

 

Bill

6 Jan 2011 - 4:50pm
Chris McLay
2005

Hi Graham,

I would avoid a networked or streaming solution as they are slow and prone to technical problems. It's much simpler to run a long VGA cable to a second monitor, and to run a long video cable from a camera to a TV. This will give you the best resolution a speed for observation.

You can then run whatever other software you like to record screens, cameras etc, on top of this simple hardware solution.

Chris

7 Jan 2011 - 10:14am
Josie Scott
2009

I'll weigh in on another idea if you use Morae; the observers (clients and stakeholders) should be asked to log their own thoughts as well. Caveat:  I am a UX researcher for the company that makes Morae.

This works for in-person or remote.  My setup typically consists of a "lab"-based test area (really, just an office with chairs and a desk) and a conference room or meeting area in another part of the building, where Observer is projected or shown on a large screen.  The keyboard and mouse are available to all. The whole team may gather to observe, although a few still log in from their desks.  Either way,  we ask *everyone* to make notes. I usually have a partner to help everyone feel comfortable logging and to make sure the Observer setup is running smoothly. 

This technique also handles the ADD issues that Jared writes about:  Now the observers are part of the action.   I emphasize that their thoughts are just as important to me as my observations -- and they are.  I love the comments and ideas that come from this interaction with the team and the stakeholders. 

And, for the record, if I can, I use my Wii remote to capture important moments as well.  When I go to Manager to analyze, my work is more than half done. 

 

 

7 Jan 2011 - 11:57am
Josh B Williams
2010

"I use my Wii remote to capture important moments"

How does the Wii work into this?

7 Jan 2011 - 11:01am
Graham Sear
2010

Hi all,

Thanks a lot for your comments, really helpful.

It doesn't look like we'll be going with the Morae option now as it's out of budget. My current approach to this is drawing up a usability test plan for an upcoming project and use it as a test case for the different options to see what we really need internally and what we can do without. From there I will iterate the tests applying different advice from your comments (set-up and software).

Thanks again, i'll post some updates for what worked/didn't work.

Graham

10 Jan 2011 - 9:10am
Louise Hewitt
2010

Have you thought about having the observers in the room? It can be quite fine, as long as they agree to behave themselves. Can't get 'free-er' than that. Once a user knows that they are being 'watched' it can be much less disconcerting to look those people in the eye than to wonder who's sitting in the invisible room - Spooks style.


As for recordings/analysis etc. I'd still use Morae every time I get the chance, but I've used things like Webex before for remote sessions or a webcam with a an ordinary video recording, and then analysed using an excel spreadsheet. Not nearly as pretty, but just as effective if you plan ahead.


Lou.

Syndicate content Get the feed