Severity/Priority criteria for UX/usability issues

22 Dec 2010 - 5:10pm
3 years ago
7 replies
3371 reads
Bruce Melendy
2009

Hi all,

We have a set of standard criteria for classifying the severity of an issue, eg for 'major' or 'sev-2':

The Application, or an essential part of it, is not working or working with reduced functionality.  However, the error is not seriously impacting the business because there is a practical bypass.  For example:

  • Essential function severely restricted
  • Essential function not working, practical bypass available
  • Restricted performance, logic problem
  • Causes severe restriction to essential up-stream or down-stream processes
  • Amounts incorrect on internal report


There is some debate about whether these are appropriate for usability- or UX-related issues. Do you have criteria you can share that are tailored to those kinds of issues or that kind of testing?

Thanks and regards,

Bruce Melendy

Comments

23 Dec 2010 - 4:23am
Chris Collingridge
2007

Really interesting topic. In many software organisations, the practical ability to get usability issues fixed is directly related to your ability to get them appropriately prioritised. Failure to do so often results in "UI issues" getting a relatively low priority and consequently always being likely to be ignored until it's much much too late.

We have a number of approaches here that we're using (and indeed the process that the team are using has an impact on which methods are most effective). One scale I'm promoting is this:

1-Causes damage:The user could inadvertently damage their system or data because of poor design

2-Impedes task: The user's task will be significantly impeded due to the design. The user is likely to be unable to complete the task, or very likely to make important errors.

3-Causes mistakes: The user is likely to misunderstand what they need to do at certain points, or to pick the wrong choices/options even when they do know what they should do. Despite this, they will probably still be able to complete the task successfully.

4-Confusing, inconsistent, or frustrating: The user is likely to experience confusion or uncertainty due to insufficient clarity or internal/external inconsistency. They will probably have "pause for thought", but will not actually make a slip. They may be frustrated by the steps they have to take or information they are forced to enter.

5-Cosmetic: The visual display of items is not ideal (alignment, spacing, etc)

This scale separates what something looks like from what the impact of the interaction is on the task. An issue can also be tagged as a 'quality' issue, with a quality issue defined as "Cosmetic or other 'minor' issues that are likely to have a strongly negative impact on the perception of quality. These are likely to be either mistakes that are very obvious from a user persepective or those that exist in high-profile/frequently used parts of the application". This means that spelling mistakes, for instance, would be 5-Cosmetic, but a quality issue - indicating that they should be given priority.

You'll note from the scale above that you must have some understanding of who the "user" is for the particular context, and you must understand the task they're trying to perform - it's not possible to rate them on the scale without that.

What this scale doesn't capture is a few key elements - how important/central/key the task is, the task frequency, and the breadth of users who perform the task. We're working on incorporating these factors without making it horrendously complicated. I'd be delighted to see other examples and experiences though...

Cheers,

Chris

 

23 Dec 2010 - 9:05am
Chauncey Wilson
2007

 Hello Bruce,   Kara Coyne and I co-authored a paper in the ACM Interactions Magazine a number of years ago where we debated how to log usability bugs.  My position was that we need to create a usability bug severity scale that was equivalent to the severity scale used by the QA team.  This was something that colleagues of mine first tried in the 1980s at Digital Equipment Corporation and I've helped implement such a system at several companies. 

  There is a short version of the usability bug severity descriptions in the ACM article.  If you are intersted, I will send you a public version.  The main point is that there are catastrophic usability bugs that are equivalent to crashes (a feature that is so hard to understand that people could loose large amounts of data for example, could be a catastrophic (sev 1)  just as a crash is a sev 1.  The philosophical issue is that usability bugs need to elevated to the same level as any other kind of bug. 

  The ACM article (if you have access to the ACM Digital LIbrary has the article that Kara and I co-authored where we discuss this issue.    Wilson, C., & Coyne, K. P. (2001). The whiteboard: Tracking usability issues: to bug or not to bug?. interactions 8, 3 (May 2001), 15-19.   There is quite a bit in the literature about usability bug severity - I'll post some references in a subsequent note.  There are schemes by Nielsen, Rubin, and others for rating severity that involve:   1.  The extent of the problem (is it a global or local problem) 2.  The consequences of the problem 3.  The frequency (or predicted frequency) of the problem 4.  Whether there is a workaround   Here is an old reference that describes an early example of a scheme for coding usability bug severity.    http://www.stcsig.org/usability/newsletter/9904-severity-scale.html   In 2003, I led up an Idea Market session at the UPA conference on Usability Bugs.  There is a summary of the discussion there with an example of a coding scheme for severity.  Here is the link to the document.

  http://www.upassoc.org/conferences_and_events/upa_conference/2004/call/sample/ideamarket_usability%20problems_wilson.doc

  I hope that is helpful.   Chauncey

 
  On Wed, Dec 22, 2010 at 6:27 PM, Bruce Melendy <bruce.melendy@gmail.com> wrote:

Hi all,

We have a set of standard criteria for classifying the severity of an issue, eg for 'major' or 'sev-2':

The Application, or an essential part of it, is not working or working with reduced functionality.  However, the error is not seriously impacting the business because there is a practical bypass.  For example:

* Essential function severely restricted
* Essential function not working, practical bypass available
* Restricted performance, logic problem
* Causes severe restriction to essential up-stream or down-stream processes
* Amounts incorrect on internal report

There is some debate about whether these are appropriate for usability- or UX-related issues. Do you have criteria you can share that are tailored to those kinds of issues or that kind of testing?

Thanks and regards,

Bruce Melendy

(((Please leave all content below t
23 Dec 2010 - 7:23pm
C K Vijay Bhaskar
2009

The criteria can only be set if you have the appropriate tools, the right definition and the people with the knowledge of understanding the process that results in the capture of the data. 
It is very important that the system that is being used has the right plugs to enable the data required to arrive at the right metrics for your defect severity. For example, if you look at "Restricted Performance, logic problem", how restricted is restricted and as logic problem is a generic term, how would that help in making it precise? OR "Cosmetic" - the visual display is very subjective depending on the persona for which the application has been designed. 

The intent of defining any parameter that is used to arrive at an objective analysis should have an objective basis. If I do arrive at some criteria with the right definition for a particular type of persona for whom I have designed the application, I will not be able to use the same definition and criteria for a design for another type of persona. If I do, then it would be comparing apples to oranges. 

It is a good idea to compare the usability related severity criteria to a bug/code related severity criteria, but again, once you bring in the persona aspect, it would turn the table over. Also, unlike the coding standards which are pretty stable, usability standards are yet to gain the stability of the coding standards. We can attempt to use the standard as the basis of arriving of the right criteria, but typically a standard would say that we need to have a mechanism to control defects and bugs and is typically not prescriptive in nature.

I would suggest that it would be objective and better if you can arrive at the criteria based on the Persona, so that you can objectively relate to the main goal of the project (impact to the end user) - with the exact specifics on how a severity would impact the Persona. 

Hope this helps. 

27 Dec 2010 - 2:06am
cfmdesigns
2004

I my experience, most organizations use both Severity and Priority.  Severity is fairly objective and stable (barring additional info that changes things), while priority is subjective and may be fluid (changing based of project phases, business needs, etc.).
A fairly standard, plain English scale for Severity is:
S1 = crash or data loss, or only way to free user block is to restart the appS2 = major feature is broken and unable to be used fully, minor feature is missing or so broken as to be unable to be used at allS3 = lesser problem: major feature use is hampered, minor feature is partially blocked, feature is broken but has workaroundS4 = problem with doesn't (or only lightly) affect ability to use a feature (UX quirks, message wording, etc.)
UX issues almost never hit the S1 level (and I'm good with that), but that doesn't mean they can't hit a P1 (or even P0) level and become vital for fixing immediately/in the next sprint/whatever.  The more "customer facing" the app and the problem is, the higher Priority UX issues should become.
One of the issues I've found in the past couple years with our current engineering organization, working in Scrum, is that we are unable to deal with a spread of Priority levels.  Our current breakdown is roughly:
P1 = intend to address in this sprint or this release (usually 2-4 sprints)P2 = would like to address in this sprint or release, but we may not get to itP3 = why do we even keep this bug around, we'll never actually look at it
(The answer for P3s is "At some point, we'll deal with that general area in a sprint, and then we'll comb the Bug Backlog for issues related to it.  Then those bugs may get reprioritized, but for now, don't even think we're going to look at it."
-- Jim


On Dec 22, 2010, at 3:45 PM, Bruce Melendy wrote:

Hi all,

We have a set of standard criteria for classifying the severity of an issue, eg for 'major' or 'sev-2':

The Application, or an essential part of it, is not working or working with reduced functionality.  However, the error is not seriously impacting the business because there is a practical bypass.  For example:

* Essential function severely restricted
* Essential function not working, practical bypass available
* Restricted performance, logic problem
* Causes severe restriction to essential up-stream or down-stream processes
* Amounts incorrect on internal report

There is some debate about whether these are appropriate for usability- or UX-related issues. Do you have criteria you can share that are tailored to those kinds of issues or that kind of testing?

Thanks and regards,

Bruce Melendy

27 Dec 2010 - 12:05pm
Paul J. Sherman
2010

FWIW, I've cobbled together a user experience issue severity scale that also touches on how the product or company's brand or financials may be affected.

Critical A critical usability issue will definitely result in a user not being able to complete their intended task. It will also result in an immediate, noticeable and significant impact to the organization’s brand equity, revenue and/or profitability.

High A high severity usability issue is one that is likely to result in a user not being able to complete their intended task. From the business perspective, the issue is likely to negatively affect the organization’s brand, revenue, or profitability.

Medium Medium severity usability issues include those that are likely to significantly impede or frustrate a user, but are not likely to prevent users from eventually accomplishing their task. They might also negatively affect the organization’s brand, revenue, or profitability.

Low Low severity usability issues include those that are likely to present momentary or transient difficulty or confusion to users, but do not prevent users from accomplishing their task. There should be no effect on the organization’s brand or financials.

More here: http://www.usabilityblog.com/2010/05/behaviorally-anchored-user-experience-issue-severity-ratings/ (Disclosure: Site not monetized. More from laziness than anything else...)

On Dec 27, 2010, at 4:34 AM, Jim Drew / CFM Designs wrote:

I my experience, most organizations use both Severity and Priority. Severity is fairly objective and stable (barring additional info that changes things), while priority is subjective and may be fluid (changing based of project phases, business needs, etc.). A fairly standard, plain English scale for Severity is: S1 = crash or data loss, or only way to free user block is to restart the appS2 = major feature is broken and unable to be used fully, minor feature is missing or so broken as to be unable to be used at allS3 = lesser problem: major feature use is hampered, minor feature is partially blocked, feature is broken but has workaroundS4 = problem with doesn't (or only lightly) affect ability to use a feature (UX quirks, message wording, etc.) UX issues almost never hit the S1 level (and I'm good with that), but that doesn't mean they can't hit a P1 (or even P0) level and become vital for fixing immediately/in the next sprint/whatever. The more "customer facing" the app and the problem is, the higher Priority UX issues should become. One of the issues I've found in the past couple years with our current engineering organization, working in Scrum, is that we are unable to deal with a spread of Priority levels. Our current breakdown is roughly: P1 = intend to address in this sprint or this release (usually 2-4 sprints)P2 = would like to address in this sprint or release, but we may not get to itP3 = why do we even keep this bug around, we'll never actually look at it (The answer for P3s is "At some point, we'll deal with that general area in a sprint, and then we'll comb the Bug Backlog for issues related to it. Then those bugs may get reprioritized, but for now, don't even think we're going to look at it."

28 Dec 2010 - 9:05am
Audrey Crane
2009

Sometimes this depends on where you are in the process. Pre-release, or even as part of the question of how the user is affected, it may make sense to prioritize by which issues are dependencies for lots of other issues. E.g.:

P1 - core issue that prevents lots of other items from being resolved ... P3 - isolated issue ... P5 - detail, resolution not required for this release

(In hindsight, this as the entire definition is useful for waterfall but less so for agile -- though the idea still warrants consideration IMO)

A presentation related to this is at http://www.dubberly.com/articles/middle-out-design.html (you have to download the PDF to see the images, which are really critical to this particular point.)

On Dec 27, 2010, at 12:01 PM, Paul J. Sherman wrote:

> FWIW, I've cobbled together a user experience issue severity scale that also touches on how the product or company's brand or financials may be affected. > > Critical > A critical usability issue will definitely result in a user not being able to complete their intended task. It will also result in an immediate, noticeable and significant impact to the organization’s brand equity, revenue and/or profitability. > > High > A high severity usability issue is one that is likely to result in a user not being able to complete their intended task. From the business perspective, the issue is likely to negatively affect the organization’s brand, revenue, or profitability. > > Medium > Medium severity usability issues include those that are likely to significantly impede or frustrate a user, but are not likely to prevent users from eventually accomplishing their task. They might also negatively affect the organization’s brand, revenue, or profitability. > > Low > Low severity usability issues include those that are likely to present momentary or transient difficulty or confusion to users, but do not prevent users from accomplishing their task. There should be no effect on the organization’s brand or financials. > > More here: > http://www.usabilityblog.com/2010/05/behaviorally-anchored-user-experience-issue-severity-ratings/ > (Disclosure: Site not monetized. More from laziness than anything else...) > > On Dec 27, 2010, at 4:34 AM, Jim Drew / CFM Designs wrote: > > I my experience, most organizations use both Severity and Priority. Severity is fairly objective and stable (barring additional info that changes things), while priority is subjective and may be fluid (changing based of project phases, business needs, etc.). > A fairly standard, plain English scale for Severity is: > S1 = crash or data loss, or only way to free user block is to restart the appS2 = major feature is broken and unable to be used fully, minor feature is missing or so broken as to be unable to be used at allS3 = lesser problem: major feature use is hampered, minor feature is partially blocked, feature is broken but has workaroundS4 = problem with doesn't (or only lightly) affect ability to use a feature (UX quirks, message wording, etc.) > UX issues almost never hit the S1 level (and I'm good with that), but that doesn't mean they can't hit a P1 (or even P0) level and become vital for fixing immediately/in the next sprint/whatever. The more "customer facing" the app and the problem is, the higher Priority UX issues should become. > One of the issues I've found in the past couple years with our current engineering organization, working in Scrum, is that we are unable to deal with a spread of Priority levels. Our current breakdown is roughly: > P1 = intend to address in this sprint or this release (usually 2-4 sprints)P2 = would like to address in this sprint or release, but we may not get to itP3 = why do we even keep this bug around, we'll never actually look at it > (The answer for P3s is "At some point, we'll deal with that general area in a sprint, and then we'll comb the Bug Backlog for issues related to it. Then those bugs may get reprioritized, but for now, don't even think we're going to look at it." > > (((Pleas

10 Jan 2011 - 9:09am
Paul J. Sherman
2010

Audrey, I agree with your point, especially now that I'm working more with in-development products.

It may still make some sense to characterize issues as to (potential) business impact, even when the product is still being constructed. My thinking is that when we point out the "presumed distal effects" - i.e., dollar or brand impact - it helps the decision-makers make priority calls that are independent of UX and engineering issues such as tight vs loose coupling of features or design elements, issue prevalence, etc.

-Paul

On Dec 28, 2010, at 10:57 AM, Audrey Crane wrote:

Sometimes this depends on where you are in the process. Pre-release, or even as part of the question of how the user is affected, it may make sense to prioritize by which issues are dependencies for lots of other issues. E.g.:

P1 - core issue that prevents lots of other items from being resolved ... P3 - isolated issue ... P5 - detail, resolution not required for this release

(In hindsight, this as the entire definition is useful for waterfall but less so for agile -- though the idea still warrants consideration IMO)

A presentation related to this is at http://www.dubberly.com/articles/middle-out-design.html (you have to download the PDF to see the images, which are really critical to this particular point.)

On Dec 27, 2010, at 12:01 PM, Paul J. Sherman wrote:

> FWIW, I've cobbled together a user experience issue severity scale that also touches on how the product or company's brand or financials may be affected. > > Critical > A critical usability issue will definitely result in a user not being able to complete their intended task. It will also result in an immediate, noticeable and significant impact to the organization’s brand equity, revenue and/or profitability. > > High > A high severity usability issue is one that is likely to result in a user not being able to complete their intended task. From the business perspective, the issue is likely to negatively affect the organization’s brand, revenue, or profitability. > > Medium > Medium severity usability issues include those that are likely to significantly impede or frustrate a user, but are not likely to prevent users from eventually accomplishing their task. They might also negatively affect the organization’s brand, revenue, or profitability. > > Low > Low severity usability issues include those that are likely to present momentary or transient difficulty or confusion to users, but do not prevent users from accomplishing their task. There should be no effect on the organization’s brand or financials. > > More here: > http://www.usabilityblog.com/2010/05/behaviorally-anchored-user-experience-issue-severity-ratings/ > (Disclosure: Site not monetized. More from laziness than anything else...) > > On Dec 27, 2010, at 4:34 AM, Jim Drew / CFM Designs wrote: > > I my experience, most organizations use both Severity and Priority. Severity is fairly objective and stable (barring additional info that changes things), while priority is subjective and may be fluid (changing based of project phases, business needs, etc.). > A fairly standard, plain English scale for Severity is: > S1 = crash or data loss, or only way to free user block is to restart the appS2 = major feature is broken and unable to be used fully, minor feature is missing or so broken as to be unable to be used at allS3 = lesser problem: major feature use is hampered, minor feature is partially blocked, feature is broken but has workaroundS4 = problem with doesn't (or only lightly) affect ability to use a feature (UX quirks, message wording, etc.) > UX issues almost never hit the S1 level (and I'm good with that), but that doesn't mean they can't hit a P1 (or even P0) level and become vital for fixing immediately/in the next sprint/whatever. The more "customer facing" the app and the problem is, the higher Priority UX issues should become. > One of the issues I've found in the past couple years with our current engineering organization, working in Scrum, is that we are unable to deal with a spread of Priority levels. Our current breakdown is roughly: > P1 = intend to address in this sprint or this release (usually 2-4 sprints)P2 = would like to address in this sprint or release, but we may not get to itP3 = why do we even keep this bug around, we'll never actually look at it > (The answer for P3s is "At some point, we'll deal with that general area in a sprint, and then we'll comb the Bug Backlog for issues related to it. Then those bugs may get reprioritized, but for now, don't even think we're going to look at it." > > (((Pleas

(((Pleas

Syndicate content Get the feed