Selenium (or similar) for Usability Testing?

28 Jan 2009 - 4:57am
5 years ago
6 replies
1263 reads
Nik
2009

Hi all,

Does anyone have any experience of using Selenium for testing
(http://seleniumhq.org/)? I have been asked to research it's potential
for UX testing. I have used it briefly as a developer, but it seems to
be more suited to testing the functionality of a site rather than from a
usability point of view.

In fact, can any automated tools really test usability?

Thanks,

Nik

Comments

28 Jan 2009 - 5:31am
Andrew Boyd
2008

On Wed, Jan 28, 2009 at 8:57 PM, Nik Lazell
<nik.lazell at realadventure.co.uk>wrote:

> Hi all,
>
>
>
> Does anyone have any experience of using Selenium for testing
> (http://seleniumhq.org/)? I have been asked to research it's potential
> for UX testing. I have used it briefly as a developer, but it seems to
> be more suited to testing the functionality of a site rather than from a
> usability point of view.
>
>
>
> In fact, can any automated tools really test usability?
>
>
>
>
Nik,

I would maintain that an understanding of automated test suites/harnesses
was a very fine thing - if only such that they may be discussed with their
proponents in UX circles, test managers, UAT participants, and managers.

Are they directly useful to a UxD? I can think of a few instances:
1. where the EAs/SAs have screwed up and system response times are a total
disaster, automated test tools can be a real graph-filler - i.e.
incontrovertible evidence that the "usability" issue cannot be solved by the
UxD. Try telling a senior manager that a slow system is not your issue as a
UxD - at best, they will be skeptical.
2. where bulk use causes issues - for example, where simulating peak system
load kills the system stone cold motherless dead (see cause in case 1).
3. where large numbers of boundary cases (and other weird data) can usefully
be included in the test process.

This is my way of saying, I guess, that poor coding and architecting can
cause system errors that adversely affect the user experience - and that
this is something that we can help to fix by engaging with the professional
system testing members of the project team.

As a postscript, I have worked with developers that love Selenium - it seems
to serve for load testing as well as some of the heavy-duty commercial test
systems (which I have not had much to do with since WinRunner days). I found
it easy to use - that said, I have not used it since 2006.

Best regards, Andrew

--
---
Andrew Boyd
http://uxbookclub.org -- connect, read, discuss
http://govux.org -- the government user experience forum

28 Jan 2009 - 5:35am
Harry Brignull
2004

Hi Nik

Your question sounds interesting, but it's hard to make out exactly what
you're asking.

The idea of having a piece of software that can test UX / usability by
pushing a button - this is a wonderful idea, but a complete fantasy. User
experience can only be understood by observing the behaviour of real users
in some manner. This is absolutely fundamental to the whole concept of user
experience. The best shortcuts you can hope for are expert evaluation
techniques like heuristics, checklists and standards - even these require an
experienced expert to operate.

Looking at the demo video, Selenium looks entirely inappropriate. Perhaps
you're after a screen recording app like morae, silverback or camtasia - but
without knowing more about your needs, it's hard to guess.

Tell us more and we'd be happy to help :-)

Harry

On Wed, Jan 28, 2009 at 9:57 AM, Nik Lazell
<nik.lazell at realadventure.co.uk>wrote:

> Hi all,
>
>
>
> Does anyone have any experience of using Selenium for testing
> (http://seleniumhq.org/)? I have been asked to research it's potential
> for UX testing. I have used it briefly as a developer, but it seems to
> be more suited to testing the functionality of a site rather than from a
> usability point of view.
>
>
>
> In fact, can any automated tools really test usability?
>
>
>
> Thanks,
>
> Nik
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

--
Dr. Harry Brignull
User Experience Consultant
http://www.90percentofeverything.com
+ 44 (0)7920 474784

28 Jan 2009 - 5:59am
Nik
2009

Hi Harry,

Thanks for your reply.

I was really just trying to establish whether anyone had found a use for
Selenium within a user-experience field.

As you said, it does look entirely inappropriate, far more relevant for
testing errors in the functionality than any aspect of usability.

I guess I was simply looking to double check my own thoughts on it being
more suited for developers testing the functional aspects of a site, and
as Andrew mentioned previously about load testing.

Thanks,

Nik

28 Jan 2009 - 6:06am
James Page
2008

We use Selenium for Functionality Testing, and have tried it for Usability
work. The main way that we have tried to use for Usability is to record and
playback user interactions with a site. The challenge is often the script
needs some editing. So I don't think it is ready out of the box.

I have also tried to use it as way of automating GOMS KLM metrics (doing
this by hand is tedious), but this needs allot more work.

James Page
http://blog.feralabs.com

2009/1/28 Nik Lazell <nik.lazell at realadventure.co.uk>

> Hi Harry,
>
>
>
> Thanks for your reply.
>
>
>
> I was really just trying to establish whether anyone had found a use for
> Selenium within a user-experience field.
>
> As you said, it does look entirely inappropriate, far more relevant for
> testing errors in the functionality than any aspect of usability.
>
>
>
> I guess I was simply looking to double check my own thoughts on it being
> more suited for developers testing the functional aspects of a site, and
> as Andrew mentioned previously about load testing.
>
>
>
> Thanks,
>
>
>
> Nik
>
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

28 Jan 2009 - 9:15am
jabbett
2008

Selenium really helps me in my work -- it eliminates a lot of the
tedium that gets in the way of usability testing. For example, I can
write a script that will register a new account, login, and navigate
to a particular section. This is very helpful when our applications
are being redeployed frequently, and accounts do not exist yet: at the
touch of a button, I can quickly get into the application and test
only the sections that need my attention.

Best,
Jon

On Wed, Jan 28, 2009 at 6:06 AM, James Page <jamespage at gmail.com> wrote:
> We use Selenium for Functionality Testing, and have tried it for Usability
> work. The main way that we have tried to use for Usability is to record and
> playback user interactions with a site. The challenge is often the script
> needs some editing. So I don't think it is ready out of the box.
>
> I have also tried to use it as way of automating GOMS KLM metrics (doing
> this by hand is tedious), but this needs allot more work.
>
> James Page
> http://blog.feralabs.com
>
> 2009/1/28 Nik Lazell <nik.lazell at realadventure.co.uk>
>
>> Hi Harry,
>>
>>
>>
>> Thanks for your reply.
>>
>>
>>
>> I was really just trying to establish whether anyone had found a use for
>> Selenium within a user-experience field.
>>
>> As you said, it does look entirely inappropriate, far more relevant for
>> testing errors in the functionality than any aspect of usability.
>>
>>
>>
>> I guess I was simply looking to double check my own thoughts on it being
>> more suited for developers testing the functional aspects of a site, and
>> as Andrew mentioned previously about load testing.
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Nik
>>
>>
>>
>> ________________________________________________________________
>> Welcome to the Interaction Design Association (IxDA)!
>> To post to this list ....... discuss at ixda.org
>> Unsubscribe ................ http://www.ixda.org/unsubscribe
>> List Guidelines ............ http://www.ixda.org/guidelines
>> List Help .................. http://www.ixda.org/help
>>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

28 Jan 2009 - 9:27am
Harry Brignull
2004

Jon and Andrew have made some very good points - I can see my comment about
Selenium being "entirely inappropriate" was a sweeping generalisation.

So - while a tool like Selenium would never be appropriate as your "one stop
shop" for UX research, it clearly can be useful in certain situations, as
they've pointed out.

Harry

Syndicate content Get the feed