How can QA help UI team

3 Jul 2007 - 3:41pm
7 years ago
11 replies
738 reads
Pawson, Mark
2007

I have been asked by the QA Manager how her team can help UI,
particularly in regards to any spec docs her team can refer to when
testing. Her team is under resourced so its not a matter of they are
looking for something to do. :)
Since I am a one man team in a mid size ( >2000 employees) international
company my UI involvement tends to be idea generation, lots of research
and acting as a catalyst to let other dev teams know about unique design
solutions from their colleagues around the globe. As such I do not see
value in spending my time writing huge spec documents which no one
reads. Rather I have written a high level style guide which has more to
do with usability guidelines, and proposed a standard UI framework from
some Expert reviews I have done of several apps all built on the same
core architecture. I have shared these light weight docs with QA and
encouraged them to be familiar with their content so when they are
testing an app they can note if the app could or should be improved by
following these guides. I could certainly use them as ambassadors of
good UI. Common sense plus a past life in QA tells me that product
managers will treat these issues as enhancements. Should they be? What
other informal ways can QA assist? Usability test observers comes to
mind...

Thanks

Mark Pawson

Comments

3 Jul 2007 - 4:23pm
cfmdesigns
2004

>From: "Pawson, Mark" <Mark.Pawson at ihs.com>
>
> I have been asked by the QA Manager how her team can help UI,
>particularly in regards to any spec docs her team can refer to when
>testing. [...]
>Common sense plus a past life in QA tells me that product
>managers will treat these issues as enhancements. Should they be? What
>other informal ways can QA assist? Usability test observers comes to mind...

One of the best things you can do, in my opinion, is to build good bridges with the QA team. QA is often treated as somewhere between an afterthought and an annoyance, a barrier to shoving the application through the pipeline and into the hands of the customer. Getting QA better informed earlier in the process and having a voice at that stage can be a useful thing. You get better and fewer bugs if QA has a chance to ask "Why are we doing it this way?" early on.

(Even just the UI guideline docs you mentioned are a boon to QA, since they can then know how to tell if something that looks wrong really is wrong.)

Also, while you've done extensive user research into what the problems are that need solving and what form the solution may take, it's the (black box) QA folks who end up knowing how the product *really* works, and thus they are the closest to how real experienced users will interact with it. They can provide feedback in that arena that is better than you can get anywhere else (although you have to weed out the issues that are magnified due to "QA process" and the bypasses that are created due to intimate knowledge of the underpinnings of the product).

-- Jim Drew
Seattle, WA

5 Jul 2007 - 9:27am
Gordon, Richard E.
2006

>From: "Pawson, Mark" <Mark.Pawson at ihs.com>
>
> I have been asked by the QA Manager how her team can help UI,
>particularly in regards to any spec docs her team can refer to when
>testing. [...]

What process do you currently use to verify that design = implementation? I have seen developers take liberties with UI design more often than I would like to admit. As you mentioned, often there is a specific reason a UI element is a certain way and that may not be obvious to the person writing the code. Depending on the testing tools and methods used, the QA process can be good for that, but only if they have adequate specs to go by. It is hard to generate a test plan or test cases based on what you think something ought to do.

If there are conditional states in the UI such as non-available buttons etc., wherever/however that behavior is specified would be something they would want to have. Functional requirements and/or business rules are normally not at that level of detail.

Error handling is an area in which we have found QA to be very useful for usability improvement. They will likely see most of the error conditions and can let you know which ones might have been overlooked in terms of providing useful error messages to the users instead of a bunch of developer gobbledygook. That is an area that sometimes gets short-changed but is certainly part of good interaction design.

Rick Gordon

5 Jul 2007 - 9:35am
Margaret Hanley
2007

>From: "Pawson, Mark" <Mark.Pawson at ihs.com>
>
> I have been asked by the QA Manager how her team can help UI,
>particularly in regards to any spec docs her team can refer to when
>testing. [...]

I worked closely with the Test Manager and his team at an agency I worked at. He used my wireframes and process flows to test the system; the test team would come to me with any deviations in the application as a first port of call. It gave me great visibility into the actual implementation and made me realise that I had "another" use and audience for this detailed documentation.

I know to ensure that I get wireframes and functional specifications to the Test Manager as a part of the design process, so he can comment on areas he considers are missing and hopefully reducing the test cycle and time.

Mags

----------------------------------------------------

Mags Hanley
Email: mairead at yahoo.com
Mobile: +44 7789 132726
Blog: http://magsmoments.blogspot.com

5 Jul 2007 - 10:35am
Michael Micheletti
2006

Our QA team really likes it when I create a detailed UI spec for a product.
They use it to test states and features for compliance. The QA team creates
the test plans and a UI spec gives them clues about how the product controls
work early in the game.

QA folks are typically late-loaded into a schedule and have more free time
early (just the opposite of designers, who are usually front-loaded). Giving
them a strong spec as development begins is perfect - they have time to do
their test plans and gear up for their busy period, and understand more
about what they'll be testing.

Even though we treat the UI specs a little like a jazz chart here (everybody
improvises around the edges) they still keep us moving in a common
direction. Without a spec, there's a lot of bugs posted saying "I think the
so-and-so ought to work this way" that take time to work through late in the
dev schedule.

I get the impression that QA people are sort of always in the market for
coherent specification documents. Hope this helps,

Michael Micheletti

On 7/3/07, Pawson, Mark <Mark.Pawson at ihs.com> wrote:
>
> I have been asked by the QA Manager how her team can help UI,
> particularly in regards to any spec docs her team can refer to when
> testing.

5 Jul 2007 - 1:37pm
Will Parker
2007

I'm currently straddling UxD, IA and QA for our company , moving
forward from a straight QA background. Michael has pointed the way
with his post, and I'm going to add detail.

On Jul 5, 2007, at 8:35 AM, Michael Micheletti wrote:

> Our QA team really likes it when I create a detailed UI spec for a
> product.
> They use it to test states and features for compliance. The QA team
> creates
> the test plans and a UI spec gives them clues about how the product
> controls
> work early in the game.

More than 'clues'! Much more!

If you (can) take the time to write out a detailed spec, including
behaviors and error states, you'll find that QA can quite easily do a
great deal of work debugging the design before a line of code is
written. You should also strongly consider getting feedback on the
spec from the front-line tech support staff. They've had to deal
directly with all the previous design mistakes and/or bad assumptions
regarding user needs.

> QA folks are typically late-loaded into a schedule and have more
> free time
> early (just the opposite of designers, who are usually front-
> loaded). Giving
> them a strong spec as development begins is perfect - they have
> time to do
> their test plans and gear up for their busy period, and understand
> more
> about what they'll be testing.

Although I've had numerous managers (unsuccessfully) disagree with
me, I believe that test planning should not commence until QA
feedback on the spec has been addressed by the design team.

This doesn't mean that Design should abdicate responsibility for
determining the direction of the product. The chief benefit of asking
QA to debug the spec is that your experienced testers have have a
talent for spotting the rough and/or ill-defined portions of the spec.

With a motivated, experienced QA team, you'll find the bulk of their
advice will mostly cover what you _forgot_ to specify or specified in
an unclear manner. Getting those issues out of the way before Dev
gets involved will cut down on a lot of the ad-hoc issue resolution
moving forward.

> Even though we treat the UI specs a little like a jazz chart here
> (everybody
> improvises around the edges)

IEEEEEEEEE!!! That's an excellent way to drive QA crazy and insure at
least 30% more email for the design team. I understand time
constraints limit what you can do, but at a minimum, nail down _all_
the core user-experience details, or QA will have no standards to
test against other than functionality. There goes your chance to get
a quality user experience.

> I get the impression that QA people are sort of always in the
> market for
> coherent specification documents. Hope this helps,

And believe it or not, a properly trained and led QA department knows
as well as Design what constitutes a coherent user experience. The
Design team's responsibility is that of author, while QA plays the
role of copy editor.

- Will

Will Parker
wparker at ChannelingDesign.com

“I wish developing great products was as easy as writing a check. If
that were the case, then Microsoft would have great products.” -
Steve Jobs

5 Jul 2007 - 2:41pm
cfmdesigns
2004

>From: Michael Micheletti <michael.micheletti at gmail.com>
>
>I get the impression that QA people are sort of always in the market for
>coherent specification documents.

And one of the worst things you can do (in my experience; 17 years in QA) is to give an "incoherent" specification document.

One project I was on, QA was given a PDF with a half-dozen pages of screenshots and told to create our test plans based on those. (No text, no explanations of infrastructure, etc.) Six weeks later, 50% of our bugs were being bounced. Turned out that the document was marketing-produced mockups actually used for putting in front of high-level expected clients, and bore little relation to anything the design team was producing or the dev team was coding while we were busy making inferences and reading between the lines (of pixels). About a month's work from a half-dozen people had to be thrown out.

The less actual information available to QA, the more well-framed the info they do have needs to be. When stuff is unknown or deeply in flux, the QA team can still do productive work, but only if they know what is going to be fruitful to work on. As with anything, ample and clear communication is a must.

-- Jim

5 Jul 2007 - 3:00pm
Jack L. Moffett
2005

I once witnessed a project in which an early UI spec was sent to
newly contracted developers in a foreign country. They were told to
build it, but given no support in terms of graphics, color
specifications, etc. Some weeks later, I took a look at the software.
They had taken screenshots of the PDF spec and cut them up for
graphics. They built the whole thing from scratch, even though it
followed templates and stylesheets already in existence. It was a mess.

QA then used that same spec document to test against, even though the
actual functionality had changed drastically.

Jack

On Jul 5, 2007, at 3:41 PM, Jim Drew wrote:

> And one of the worst things you can do (in my experience; 17 years
> in QA) is to give an "incoherent" specification document.

Jack L. Moffett
Interaction Designer
inmedius
412.459.0310 x219
http://www.inmedius.com

The World is not set up to facilitate the best
any more than it is set up to facilitate the worst.
It doesn't depend on brilliance or innovation
because if it did, the system would be unpredictable.
It requires averages and predictables.

So, good deeds and brilliant ideas go against the
grain of the social contract almost by definition.
They will be challenged and will require
enormous effort to succeed.

Most fail.
- Michael McDonough

5 Jul 2007 - 4:21pm
Janna DeVylder
2006

Good question...one I've been chewing on lately. QA knows the fringe cases
of our site the best, and while QA is involved in reviewing our documents
from the very beginning, it is inherently a reactive situation rather than a
proactive situation.

I wonder if I could engage the project's QA members in a quick brainstorm
before too much concepting has occurred, to ensure I'm baking flexibility
into the solution I provide, rather than coming up with band-aids while in
development. Conversely, I need to look at their test cases, as well, to
ensure that nothing (or at least close to nothing) has been lost in
translation...that includes checking for the often-overlooked 'look and
feel' aspects that can make good functionality unpalatable in the end.

And the best thing we can do, in the end? Debrief. Talk about how that
project went, what improved, what stank... figure out how to do it better
next time. <repeat this step often>

janna

On 7/3/07, Pawson, Mark <Mark.Pawson at ihs.com> wrote:
>
> I have been asked by the QA Manager how her team can help UI,
> particularly in regards to any spec docs her team can refer to when
> testing. Her team is under resourced so its not a matter of they are
> looking for something to do. :)
>
>

5 Jul 2007 - 8:43pm
Will Parker
2007

On Jul 5, 2007, at 2:21 PM, Janna Hicks DeVylder wrote:

> I wonder if I could engage the project's QA members in a quick
> brainstorm
> before too much concepting has occurred, to ensure I'm baking
> flexibility
> into the solution I provide, rather than coming up with band-aids
> while in
> development.

If your development cycles are short, this is an excellent idea.
However, if the Design phase of the next cycle starts at Code
Complete (as it often does on larger projects), you'd be distracting
the QA team when they're most needed for the current dev cycle.

Try to find a point after the current dev cycle ends and the QA team
has had time to relax -- and more importantly, do any internal QA
procedures to get reset for the next cycle. Immediately after QA rest
and refit is the perfect time to get QA input. Your QA team will be
as fresh as they ever get and ready for a moderate challenge.

> Conversely, I need to look at their test cases, as well, to
> ensure that nothing (or at least close to nothing) has been lost in
> translation...that includes checking for the often-overlooked 'look
> and
> feel' aspects that can make good functionality unpalatable in the end.

Brava! I've _never_ heard of a designer willing to review test plans
and cases. If your development environment allows for it, consider
posting test case change requests as spec bugs, if the QA manager(s)
agree. That'll keep the QA team reasonably happy.

> And the best thing we can do, in the end? Debrief. Talk about how that
> project went, what improved, what stank... figure out how to do it
> better
> next time. <repeat this step often>
>
>
> janna

- Will

Will Parker
wparker at ChannelingDesign.com

“I wish developing great products was as easy as writing a check. If
that were the case, then Microsoft would have great products.” -
Steve Jobs

6 Jul 2007 - 5:33am
vutpakdi
2003

When I do a more formal design for a team (rather than quickie
consulting), the QA lead and tech writing lead normally are very happy
to have me onboard. Sometimes, it seems that my interaction design
spec is the closest or most comprehensive look at seeing and reading how
things are supposed to behave (as well as look), so that gives them a
better grounding from which to create their test plans.

Some teams have architects and product managers that are better about
recording requirements, but sometimes, those are just requirements
without the connective text much less a visual image of how things are
supposed to look. And, there is often very little about transitions.
So, the QA and tech writing leads really like my design specs to help
fill those gaps.

The QA and tech writing teams also often have pretty decent ideas about
things that annoy them in the product. You do have to take these
comments with a grain of salt since their "use" of the product is a bit
different than an end user's, but their observations about usability
issues are often very helpful.

Ron

6 Jul 2007 - 3:20pm
DrWex
2006

Right now my main relationship with QA is that they are the primary
users of the GUI Standards/Guidelines I wrote. They are testing
whether these guidelines are useful for detecting problems in
developed screens, for helping to ensure cross-product consistency,
and whether there are major omissions in the guidelines.

Another way that I've worked with QA in the past is to do statistical
sorts of analyses. Areas where many defects are found in products are
prime candidates for redesign (you can't keep patching forever); also,
areas where QA have a hard time developing realistic test plans are
areas that are highly likely to be confusing for people when the
product gets into widespread use because they're probably confusing.

--Alan

Syndicate content Get the feed