How would you use this report?

3 Jul 2007 - 9:46am
7 years ago
18 replies
653 reads
tdellaringa
2006

Hi folks,

Just had a meeting with my boss about our process and it looks like we need
to 'reface' 30 sites within the next 6 months, so... yeah time is gonna be
short. That means I'm not going to be able to do really any user research
that takes any amount of time at all. I think I'm going to try and use Todd
Warfel's task analysis grid for these, seems like it might be a good fit.

Anyway - what he did give me is a white paper by Noel-Levitz entitled "What
do prospective students expect from your web site and e-communications
program?" It's the result of 1000 phone interviews of two groups, graduate
and upcoming H.S. seniors. (Our sites are marketing sites for our colleges).

There seems to be some good data in there, and some good recommendations. My
question is, what is the best way to use the resulting data in this report?
The sample size is significant, which I would think would be good. Seems
like the recommendations based on this data would be fairly sound.

It doesn't seem like this would be appropriate data for personas. Any
comments are appreciated.

Thanks

Tom

Comments

3 Jul 2007 - 11:12am
tdellaringa
2006

On 7/3/07, Muntone, Jim <Jim.Muntone at dowjones.com> wrote:
>
> Hi Tom,
> Some follow up questions and a comment...
>
> Are the tasks in the report typical to what you believe people typically
> do on your site now?

Yes, they are. Our sites are marketing/school sites that people are using to
research and apply to our schools. Many of the features listed are things
they already do on our site (and some are not).

Why the "refacing"? What makes you guys think you have a usability
> problem that can be solved with a redesign? (Customer feedback, site
> metrics, etc.)

Whew, well big question. We have 80+ sites, many are simply out of date and
"look old." Put it this way, we have to fix 30+ of them in the next 5
months. That's 6 sites per month... eeeyah. See the problem? :) So, we can't
do a full blown process for each site, as much as I would love to, and I'm
not sure it's necessary either in a sense - 12 sites might be all culinary
arts schools, even though they are in different locations.

It also leaves little or no time for developing personas and doing other
user research. I'm pushing to at least do phone interviews and getting a lot
of hemming and hawing back, they just don't feel there is enough time. (And
they are probably right, unfortunately).

As far as specific reasons why this is being done? A) They don't like they
way many of them look - they look "dated." B) The sites are old and the
site owners (schools) haven't taken ownership because the sites don't match
the stakeholder goals or interests. Unfortunately, I am not aware of *any*
user feedback, although I am pushing for that and trying to get into the
site analytics.

As I said in other posts, I am new here (2 weeks). There is clearly a
culture here of not involving the users due to time constraints. My boss
wants usability involved in the processes, but he has other constraints on
him - mainly time. So I need to work with him to do as much as we can in the
small amount of time possible.

As far as personas, even without doing formal research, you might want
> to take a stab at creating at least high-level archetypes so you guys
> have a consistent vocabulary when talking about your customers (ie.
> Poweruser vs, Novices).

That's a good idea. And I *think* although this may be another subject, that
they could be used nearly across the board. We have 3 types of users at a
high level - graduates seeking Masters and PhDs, currently enrolled students
(of all types) and High School seniors looking for a college. What further
divisions there are among those groups, I don't really know, and I don't
know how they differ among our verticals of culinary vs. design vs. general
vs. health care schools.

Maybe you can cross reference the report with internal marketing data,
> or quickly interview some of your staff that do regularly interact with
> your customers(salespeople, reps, stakeholders) who might have an
> understanding of day to day use patterns/problem points.

Makes sense, I'll ask if such data exists. We have people who deal with
students, so I could find out if I can spend some time with them.

Thanks for the suggestions Jim!

Tom

3 Jul 2007 - 10:35am
.pauric
2006

The report seems ripe for directing/prioritizing menu & content. That
is, to me a lot of the stats are not far off card sort results (e.g.
pages 5 & 6).

Some of the questions are suggestive and questionable. E.g. page 14:
'I would like to have an IM chat with a representative from a
graduate program on my list. 39%'
Then on the following page 15: 'Would you consider sending an IM to
a college representative through the school's website? - yes 76%'

I think this is where the report is a little wanting and maybe you
can focus your limited research time on IM placement/functionality
within the site?

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://beta.ixda.org/discuss?post=17900

3 Jul 2007 - 11:20am
White, Jeff
2007

It seems like the report, if it's the only research you have to go by,
should really drive the features and functionality that the sites
deliver. I would start by matching what the report says with the
business goals of these sites and go from there. You could use the
findings of the report, balanced with your business goals, to derive
page hierarchy, messaging, etc.

And, maybe the data could be used to develop personas. Does the report
break down the expectations of prospective students based on what "type"
of student they are? For example, maybe a freshman, from in-state, has
totally different needs than a sophomore who is considering a transfer
from another state or country, etc. Seems like data like this, if you
have it, could lend itself well to personas. And since you don't have
much time to do other research, those personas could be really helpful
as you go through the process.

Are you supposed to totally redesign the sites or just "reface them"? If
it's the latter, seems like you're being set up for failure in a way.
Re-branding poor content isn't really going to help much, and I hope
your stakeholders realize this.

At any rate, HTH and good luck.

Jeff

>>
Hi folks,

Just had a meeting with my boss about our process and it looks like we
need
to 'reface' 30 sites within the next 6 months, so... yeah time is gonna
be
short. That means I'm not going to be able to do really any user
research
that takes any amount of time at all. I think I'm going to try and use
Todd
Warfel's task analysis grid for these, seems like it might be a good
fit.

Anyway - what he did give me is a white paper by Noel-Levitz entitled
"What
do prospective students expect from your web site and e-communications
program?" It's the result of 1000 phone interviews of two groups,
graduate
and upcoming H.S. seniors. (Our sites are marketing sites for our
colleges).

There seems to be some good data in there, and some good
recommendations. My
question is, what is the best way to use the resulting data in this
report?
The sample size is significant, which I would think would be good. Seems
like the recommendations based on this data would be fairly sound.

It doesn't seem like this would be appropriate data for personas. Any
comments are appreciated.

Thanks

Tom
>>

3 Jul 2007 - 11:31am
tdellaringa
2006

On 7/3/07, White, Jeff <Jeff.White at jtv.com> wrote:
>
> It seems like the report, if it's the only research you have to go by,
> should really drive the features and functionality that the sites
> deliver. I would start by matching what the report says with the
> business goals of these sites and go from there. You could use the
> findings of the report, balanced with your business goals, to derive
> page hierarchy, messaging, etc.

And they do even make specific recommendations at the end, particularly
where there are 2 specific gaps. Our business goals are actually individual
school goals (per site) along with a few higher level corporate goals (make
the site worthwhile for alumni, for example), but that makes sense to match
up the two.

And, maybe the data could be used to develop personas. Does the report
> break down the expectations of prospective students based on what "type"
> of student they are?

No, that's the problem. There's no individual data, only the results of the
questions asked. They will say what group it was, but it doesn't get any
more granular than that. I wish that data was there. Maybe the overall
archetype idea is the way to go.

Are you supposed to totally redesign the sites or just "reface them"? If
> it's the latter, seems like you're being set up for failure in a way.

Yeah, that is my concern. It's not a total redesign in the sense that we
would think of it. The page colors and templates may change, and we may
rearrange sections, but the content will be 90% the same and we won't have
time to do much more than "improve things". It's going to be challenging,
and it's going to be hard to set goals beforehand to measure afterward to
see what kind of successes we have.

Re-branding poor content isn't really going to help much, and I hope
> your stakeholders realize this.

I guess it's my job to make sure they do. But thanks for the reminder!

At any rate, HTH and good luck.

Thanks for the feedback!

Tom

3 Jul 2007 - 11:00am
jim muntone
2006

Hi Tom,
Some follow up questions and a comment...

Are the tasks in the report typical to what you believe people typically
do on your site now?

Why the "refacing"? What makes you guys think you have a usability
problem that can be solved with a redesign? (Customer feedback, site
metrics, etc.)

As far as personas, even without doing formal research, you might want
to take a stab at creating at least high-level archetypes so you guys
have a consistent vocabulary when talking about your customers (ie.
Poweruser vs, Novices).

Maybe you can cross reference the report with internal marketing data,
or quickly interview some of your staff that do regularly interact with
your customers(salespeople, reps, stakeholders) who might have an
understanding of day to day use patterns/problem points.

Hope that helps.

-j

---------------------------------
Jim Muntone
User Experience Manager
Dow Jones & Company, Inc.
Blog: www.sadrhino.net

-----Original Message-----
From: discuss-bounces at lists.interactiondesigners.com
[mailto:discuss-bounces at lists.interactiondesigners.com] On Behalf Of Tom
Dell'Aringa
Sent: Tuesday, July 03, 2007 10:46 AM
To: IxDA Discuss
Subject: [IxDA Discuss] How would you use this report?

Hi folks,

Just had a meeting with my boss about our process and it looks like we
need
to 'reface' 30 sites within the next 6 months, so... yeah time is gonna
be
short. That means I'm not going to be able to do really any user
research
that takes any amount of time at all. I think I'm going to try and use
Todd
Warfel's task analysis grid for these, seems like it might be a good
fit.

Anyway - what he did give me is a white paper by Noel-Levitz entitled
"What
do prospective students expect from your web site and e-communications
program?" It's the result of 1000 phone interviews of two groups,
graduate
and upcoming H.S. seniors. (Our sites are marketing sites for our
colleges).

There seems to be some good data in there, and some good
recommendations. My
question is, what is the best way to use the resulting data in this
report?
The sample size is significant, which I would think would be good. Seems
like the recommendations based on this data would be fairly sound.

It doesn't seem like this would be appropriate data for personas. Any
comments are appreciated.

Thanks

Tom
________________________________________________________________
Welcome to the Interaction Design Association (IxDA)!
To post to this list ....... discuss at ixda.org
List Guidelines ............ http://beta.ixda.org/guidelines
List Help .................. http://beta.ixda.org/help
Unsubscribe ................ http://beta.ixda.org/unsubscribe
Questions .................. list at ixda.org
Home ....................... http://beta.ixda.org

3 Jul 2007 - 1:06pm
tdellaringa
2006

On 7/3/07, pauric <radiorental at gmail.com> wrote:
>
> The report seems ripe for directing/prioritizing menu & content. That
> is, to me a lot of the stats are not far off card sort results (e.g.
> pages 5 & 6).

Good point.

Some of the questions are suggestive and questionable. E.g. page 14:
> 'I would like to have an IM chat with a representative from a
> graduate program on my list. 39%'
> Then on the following page 15: 'Would you consider sending an IM to
> a college representative through the school's website? - yes 76%'
>
> I think this is where the report is a little wanting and maybe you
> can focus your limited research time on IM placement/functionality
> within the site?

I wondered about the IM stuff myself. I would definitely like to know more
about how they would want to use IM, and I think it would be different
between groups. I see IM as slightly intrusive from those you don't know,
but OTOH if I have supplied my IM name... then I guess you are giving
permission to be contacted. Seems like an odd way to go about searching for
a school, but I'm not those users.

On our side, we don't have a corporate policy on IM other than to say that
it is tolerated (in other words, they don't particularly like it, and they
see it as a security threat). So there's a disconnect between our policy and
the possible user desire to use IM in this manner as well, which would need
to be solved.

Thanks for the input!

Tom

3 Jul 2007 - 1:30pm
.pauric
2006

One thing to consider is.. well, I dont know how you describe this
functionality, lets call it virtual IM. I've been on some sales/shop sites
where if I spend a lot of time browsing without having put something in my
basket, a pop-up asks if I'd like to IM with a representative. Its a server
side IM too, not a client on my machine. I see this aiding conversion but
dont know of any stats to back that hunch up.

Here's an example: http://www.bestkiteboarding.com/ you can find the direct
link to the IM client in the bottom window. I'm assuming the timed
initiation function still works too.

I've also used virtual IM functionality when looking for support at
www.audible.com. I preferred that method over faq and email, by a long
shot.

I see this virtual IM as eliminating a number of barriers when compared with
initiating a conversation via local clients. Its not hard to visualise a
potential student utilising this if they were browsing one of your sites
(another hunch) 'Its there, its easy, why not?'

I do appreciate this takes a significant amount of financial commitment to
implement & support though. As per point 8 in the recent UIEtips: Ten Ways
to Kill Good Design, it might not be "technically feasible".

regards - pauric

3 Jul 2007 - 1:33pm
tdellaringa
2006

On 7/3/07, pauric <radiorental at gmail.com> wrote:
>
> I see this virtual IM as eliminating a number of barriers when compared
> with initiating a conversation via local clients. Its not hard to visualise
> a potential student utilising this if they were browsing one of your sites
> (another hunch) 'Its there, its easy, why not?'

Good point, and a little more controlled than typical IM

I do appreciate this takes a significant amount of financial commitment to
> implement & support though. As per point 8 in the recent UIEtips: Ten Ways
> to Kill Good Design, it might not be "technically feasible".

Yep, that would be my concern too. I don't know that we have the resources
to have enough people on the other side of that widget, but it's worth
exploring.

Thanks!

3 Jul 2007 - 3:46pm
jstrande
2007

Tom,

One other thing you can do very quickly and inexpensively is talk to the
customer service folks. Ask them about the phone calls and emails they
receive most from people, my guess is that you'll discover some common
themes. Those might help sort of some of the answers that people gave on the
report to what they actually care about.... sure, it's easy to answer "yes,
I'd IM someone", but what they really care about is being able to quickly
get the answer to the question about financial assistance (for example).

We have a customer service group as well as a very technical product
information team, my group spends at least a few hours a month with someone
from each of those areas, the insights we get are always rewarding.

Jon

On 7/3/07, Tom Dell'Aringa <pixelmech at gmail.com> wrote:
>
> On 7/3/07, pauric <radiorental at gmail.com> wrote:
> >
> > I see this virtual IM as eliminating a number of barriers when compared
> > with initiating a conversation via local clients. Its not hard to
> visualise
> > a potential student utilising this if they were browsing one of your
> sites
> > (another hunch) 'Its there, its easy, why not?'
>
>
> Good point, and a little more controlled than typical IM
>
> I do appreciate this takes a significant amount of financial commitment to
> > implement & support though. As per point 8 in the recent UIEtips: Ten
> Ways
> > to Kill Good Design, it might not be "technically feasible".
>
>
> Yep, that would be my concern too. I don't know that we have the resources
> to have enough people on the other side of that widget, but it's worth
> exploring.
>
> Thanks!
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> List Guidelines ............ http://beta.ixda.org/guidelines
> List Help .................. http://beta.ixda.org/help
> Unsubscribe ................ http://beta.ixda.org/unsubscribe
> Questions .................. list at ixda.org
> Home ....................... http://beta.ixda.org
>

3 Jul 2007 - 3:31pm
jkolko
2010

Tom,

> Yes, they are. Our sites are marketing/school sites that people
> are using to research and apply to our schools. Many of the
> features listed are things they already do on our site (and some
> are not).

When I was teaching, we administered the Noel Levitz survey once a year in
classrooms. Like any survey, their core questionnaire has huge problems -
the first and foremost being that the participants simply don't care while
they are answering it, and therefore fill out all of the first answer, or
abandon the thing halfway through, or put A-B-C-D-A-B-C-D for fun. It was
interesting to hear the students say things like "what a waste of time" or
"I can't believe I'm paying for class and they are doing this - I'm going to
put random answers". I can only imagine the online version fails due to the
same problems - the participants have little vested interest in putting
forth a complete effort to produce the results.

If I were in your shoes, I would push very, very hard to do even the most
rudimentary user research: find yourself three or four college students and
watch them use your sites. Then talk to them. It will take you a day, tops,
and while the data certainly isn't quantitative, it is probably a lot more
useful.

Perhaps the bigger question is - are surveys really a good tool for
Interaction Designers to be using?

Jon Kolko | http://www.thoughtsOnInteraction.com

4 Jul 2007 - 3:57pm
Jared M. Spool
2003

On Jul 3, 2007, at 11:12 AM, Tom Dell'Aringa wrote:

> As far as specific reasons why this is being done? A) They don't
> like they
> way many of them look - they look "dated." B) The sites are old
> and the
> site owners (schools) haven't taken ownership because the sites
> don't match
> the stakeholder goals or interests. Unfortunately, I am not aware
> of *any*
> user feedback, although I am pushing for that and trying to get
> into the
> site analytics.

Ah, change for change sake.

Well, once you remove "quality" as a requirement, everything else
becomes much easier to achieve.

After you've gone thru and updated all these sites, what will be
different, from a user experience perspective? How will you (or
anyone else on the team) know if you've actually "improved" the
experiences of people using these sites?

If you can answer those questions, you'll know what to do.

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: jspool at uie.com p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks

5 Jul 2007 - 8:07am
tdellaringa
2006

On 7/4/07, Jared M. Spool <jspool at uie.com> wrote:
>
> Ah, change for change sake.
>

Yeah, unfortunately it falls under the general "let's improve these sites"
banner without much thought or direction.

Well, once you remove "quality" as a requirement, everything else becomes
> much easier to achieve.
>
> After you've gone thru and updated all these sites, what will be
> different, from a user experience perspective? How will you (or anyone else
> on the team) know if you've actually "improved" the experiences of people
> using these sites?
>

I asked that exact question in our last meeting. Didn't get much of an
answer, and my point was how are we going to measure success? I'll need to
push that a bit.

If you can answer those questions, you'll know what to do.
>

Yep, thanks!

Tom

5 Jul 2007 - 8:56am
Todd Warfel
2003

Tom, a couple of things...

On Jul 3, 2007, at 10:46 AM, Tom Dell'Aringa wrote:

> [...] I think I'm going to try and use Todd Warfel's task analysis
> grid for these, seems like it might be a good fit.

This should help out here as it will outline tasks, goals, etc. and
help you prioritize what needs to be created. If you need any help
using this at all, please don't hesitate to contact me directly.

> Anyway - what he did give me is a white paper by Noel-Levitz
> entitled "What do prospective students expect from your web site
> and e-communications program?" It's the result of 1000 phone
> interviews of two groups, graduate and upcoming H.S. seniors. (Our
> sites are marketing sites for our colleges).
>
> There seems to be some good data in there, and some good
> recommendations. [...] It doesn't seem like this would be
> appropriate data for personas. Any comments are appreciated.

Actually, this is just the type of data we try and work into our
personas. I did a tutorial on data-driven design research personas at
UPA this year (around 40 attendees). I'd be more than happy to give
you some guidance to work these into some data-driven design research
personas which should help as well.

Cheers!

Todd Zaki Warfel
President, Design & Usability Specialist
Messagefirst | Designing Information. Beautifully.
----------------------------------
Contact Info
Voice: (215) 825-7423
Email: todd at messagefirst.com
AIM: twarfel at mac.com
Blog: http://toddwarfel.com
----------------------------------
In theory, theory and practice are the same.
In practice, they are not.

5 Jul 2007 - 9:00am
Todd Warfel
2003

On Jul 4, 2007, at 4:57 PM, Jared M. Spool wrote:

> After you've gone thru and updated all these sites, what will be
> different, from a user experience perspective? How will you (or
> anyone else on the team) know if you've actually "improved" the
> experiences of people using these sites?

Two excellent points. Make sure you set some goals so you can measure
success/failure at the end of the redesign. Otherwise, this is going
to be a bad situation no matter what. At least this way, you'll be
taking a bad situation and simply reducing the risk a bit.

Cheers!

Todd Zaki Warfel
President, Design & Usability Specialist
Messagefirst | Designing Information. Beautifully.
----------------------------------
Contact Info
Voice: (215) 825-7423
Email: todd at messagefirst.com
AIM: twarfel at mac.com
Blog: http://toddwarfel.com
----------------------------------
In theory, theory and practice are the same.
In practice, they are not.

5 Jul 2007 - 9:01am
Todd Warfel
2003

What's the significance of the date? Is there any way to push it out
to do quality work? Is there some board meeting coming up or
something that makes this date significant? Or is it some high level
executive that just picked a date out of thing air?

On Jul 5, 2007, at 9:07 AM, Tom Dell'Aringa wrote:

> Yeah, unfortunately it falls under the general "let's improve these
> sites"
> banner without much thought or direction.

Cheers!

Todd Zaki Warfel
President, Design & Usability Specialist
Messagefirst | Designing Information. Beautifully.
----------------------------------
Contact Info
Voice: (215) 825-7423
Email: todd at messagefirst.com
AIM: twarfel at mac.com
Blog: http://toddwarfel.com
----------------------------------
In theory, theory and practice are the same.
In practice, they are not.

8 Jul 2007 - 9:12pm
Jared M. Spool
2003

On Jul 5, 2007, at 6:07 AM, Tom Dell'Aringa wrote:

> On 7/4/07, Jared M. Spool <jspool at uie.com> wrote:
> Ah, change for change sake.
>
> Yeah, unfortunately it falls under the general "let's improve these
> sites" banner without much thought or direction.

I'd definitely bang on the "improve" notion a bit more.

Here's the deal: there have been *many* cases were teams have set out
to improve things and have ended up making them worse. Facebook's
"improvements" landed them with a petition of 700,000 users in 24
hours and major Wall Street Journal coverage later that week. A large
"big box" retailer "improved" their web site, instantly seeing a 20%
reduction in revenues which took them 3 years to recover from. A
top-10 web site "improvements" reduced page views by 40% (thus
reducing ad revenue by 40%).

And those were just the really *public* improvement failures. There
are hundreds of stories that are similar.

It's easy to break something. You want to be really careful.

On the other hand, you might get a good story out of it.

Here's some articles on the topic:

The Quiet Death of the Major Re-Launch
http://www.uie.com/articles/death_of_relaunch/

Designing for Embraceable Change
http://www.uie.com/articles/embraceable_change/

Of course, when you don't know where you're going, any road will get
you there.

Hope that helps,

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: jspool at uie.com p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks

9 Jul 2007 - 7:55am
.pauric
2006

Jared, sort of an open ended question...

If there's a perception internally that its time for a refresh.
Isnt there also a good chance that users perceive the presentation
layer to be outdated, thus detracting some value from the 'brand'.

I agree wholeheartedly about breaking functionality, the examples you
quoted for Facebook was a function refresh, not presentation (not sure
about the others).

My thinking is that if management come along and ask for
'improvements', its probably presentation layer, versus usability
or CSO feedback pointing to functionality issues.

As such, you can focus on improving layout without going through the
full throws of a major re-design, running the risk of breaking
function.

yeah, I know.. It depends (o;

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://beta.ixda.org/discuss?post=17900

9 Jul 2007 - 11:24am
Nancy Broden
2005

The Facebook feature in question is the News Feed which displays
member activity on the site, including changes in status. Although
there was an initial hue and cry over the potential breach in
privacy, this feature is one of the main reasons Facebook has become
so sticky and enjoyed rapid growth now that it is open to the general
public: http://tinyurl.com/yvb9yl

On Jul 8, 2007, at 7:12 PM, Jared M.Spool wrote:
> Facebook's
> "improvements" landed them with a petition of 700,000 users in 24
> hours and major Wall Street Journal coverage later that week.

Nancy Broden
nancy.broden at gmail.com

Syndicate content Get the feed