Reference: User Testing Protocol

Supporting Materials:

Greeting script

User Testing Demographic Questionnaire

Scenarios

Student

Your university has a portal that you frequent regularly for your classes and submitting assignments, as well as being informed of campus news, announcements, and events. There is an Announcements box on the Welcome tab of the portal that has information you need to see for things like registration schedules, class cancellations, and assignment submission deadlines. On your computer, however, the announcements box is always at the bottom of the page and you have to scroll down to see it. Last week you missed an important announcement because you forgot to scroll down and check the announcements.

Tasks

  1. Change the placement of the announcements box on the Welcome tab.
  2. Change the placement of other boxes on the Welcome tab.
  3. (Note to Test Coordinator: Only ask if user has not yet encountered a locked box.) On the Welcome tab, move (or say drag, if they have not yet used drag-and-drop) the announcements box as far to the upper left as you can.
  4. (Note to Test Coordinator: Only ask if user has not dragged & dropped a box. Explain how to drag-and-drop if necessary.) Drag the announcements box to a new position on the Welcome tab.

Instructor

Your university has a portal that you frequent regularly for managing the classes you teach and distributing assignments, as well as being informed of campus news, announcements, and events. There is an announcements box on the Home page of the portal that has information you need to see for things like faculty and staff meetings. On your computer, however, the announcements box is always at the bottom of the page and you have to scroll down to see it. Last week you missed an important announcement because you forgot to scroll down and check the announcements.

Tasks

  1. Change the placement of the announcements box on the Welcome tab.
  2. Change the placement of other boxes on the Welcome tab.
  3. (Note to Test Coordinator: Only ask if user has not yet encountered a locked box.) On the Welcome tab, move (or say drag, if they have not yet used drag-and-drop) the announcements as far to the upper left as you can.
  4. (Note to Test Coordinator: Only ask if user has not dragged & dropped a box. Explain how to drag-and-drop if necessary.) Drag the announcements box to a new position on the Welcome tab.

Notes for Test Coordinator

Offering help during the test

Don't offer help; let the user attempt to perform the task themselves. If they ask for help reply with:

You want to observe whether the user has trouble:

Post-test Questionnaire

Note: Have users fill out this questionnaire on paper.  

How easy or difficult was it for you to realize you could move boxes in the portal by dragging a box from one location and dropping it in a new location?

  1. Very difficult
  2. Difficult
  3. Neutral
  4. Easy
  5. Very easy

How easy or difficult was it for you to actually move a box in the portal to a specific location?

  1. Very difficult
  2. Difficult
  3. Neutral
  4. Easy
  5. Very easy

How easy or difficult was it for you to distinguish between boxes in the portal that could be moved and those that were locked in place?

  1. Very difficult
  2. Difficult
  3. Neutral
  4. Easy
  5. Very easy

Post-test questions (ask these verbally after the user has filled out the questionnaire)

1. Did you notice a visual cue for picking up one of these boxes? How helpful was it to you? (looking for cursor change to hand)

2. Did you notice a visual cue for dropping (and placing) a box? How helpful was it to you? (looking for green box drop target)

3. Further questions about drop target, which can be skipped if answered in question #2:

4. Did you see a visual cue that helped you determine whether a box could be picked up or not? How helpful was it to you? (looking for lock icon to show the portlet is locked.)