June 8, 2000

Attendees:

Barker, Dean Interface Architecture

Heffron Monahan, Carla American Express

Stanley, Mary Datacard

Borofka, Jill Digi International

Karyukina, Barbara SGI

Suits, Lucy Mae Plural

Damm, Gregory Personnel Decisions, Inc.

Madsen, John Interim Technology

Walsh, James Datacard

Funck, Mike NCS

Prigge, David Cray Research

Zange-Josephson, Caryn Renaissance Worldwide

Gardiner, Terry Personnel Decisions, Inc

Sit, Richard IBM

Future Meetings:

A. July topics

B. July location -- we need a place to host our July meeting. If you can do it, please let Lucy know ASAP. Thanks!!

C. August meeting -- will take place at the national conference in Ashville, NC.

Hosting our Website:

Digi International may be able to continue hosting for a time -- Jill will confirm. However, we still need to identify a permanent home for it. Please let Lucy or Jill know if your company can host or if you have other ideas.

Discussion #1 -- Online usability testing:

Lucy Suits shared various articles addressing the topic of online usability testing from Jakob Nielsen's Alertbox, Business 2.0 and Vividence's web site.

Reviews are mixed with Nielsen landing strongly on the side of live testing vs. online sessions. The group seemed to feel that it was online market research more than online usability testing because of the number of users suggested (100 -- 200 per test) and the type of data they purport to gather.

Lucy has participated in one test with Vividence. The process involved

completing a screener with demographic, computer and preference data, then downloading a special browser from Vividence. Upon notification that a test was open (sent to her via e-mail), she went to the company's site via the special Vividence browser. She reviewed the site and provided feedback on windows that popped-up periodically to solicit information about her reaction to elements of the site.

Based on our discussion, questions remained in the group about the following:

  1. Can they track time on task?
  2. Can they track success vs. failure?
  3. Can they track mouse movements?
  4. How do they determine who will participate? Does the online nature of it bias it toward savvier web users thereby diminishing the credibility of the findings for a "real" user group?
  5. How do they interpret the data they've collected? (Time on task may be skewed by the pop-up windows asking for feedback along the way, what do the sites do if you fail to complete a task?)
  6. What about the non-verbal feedback that you get in a live lab that's missing here (e.g., deep sighs, furrowed brows, angry outbursts, etc.)?
  7. How do they account for the fact that it's a lot of self-reported feedback (accuracy problems)?

The group agreed that it would be great to hear from the vendors themselves -- Vividence, WebCriteria and Evity.

Carla Heffron Monahan agreed to speak with the Amex Market Research group who entertained presentations from Vividence and WebCriteria to see if contact information was available.

Dean Barker will check with Dave Mitropolous-Rundus (a former colleague) to see if he is knowledgeable about other online testing opportunities, such as NetMeeting, and if he'd be interested in talking with our group by conference call.

Discussion #2 -- Where does usability reside in your organization?

Lee Zukor couldn't make the meeting to lead this topic so, Lucy kicked off a round-robin discussion about where it resides in the companies represented.

Individual results are shown in the attendance chart above. Consolidated results are as follows:

Last update: February 07, 2003