Child pages
  • Uploader User Testing - Round 1 Results

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 5.3

Summary

Excerpt

The user testing on the Uploader was carried out in a "quick-and-dirty" fashion. Ten users (5 in Toronto, 5 in Berkeley) were asked to spend less than 30 minutes with us to complete 4 upload tasks.  Below are the results thus far.

Toronto: 

  • All users were able to successfully upload files
  • 3 of 4 users asked if selecting multiple files was possible before being asked to add multiple files. Given the selected users were rather tech-savvy, this demonstrates the users' high expectations.
  • All users acknowledged the list of selected files and proceeded to click on Upload

Berkeley:

  • All users were "wowed" by being able to browse and select multiple files at once.  Even though they didn't expect it, all quickly selected multiple files.
  • While files were uploading, 4 of 4 users chose "browse collections" to return to the image gallery thumbnail view.  They expected the upload to continue "behind the scenes".  There was no feedback that the upload would stop (but that's what happens).  This happened on one to several tasks for each user.
  • One user mentioned it would be nice to see thumbnails in the upload queue.  He said the thumbnail could even get bigger as the user mouses over it.
  • 3 of 4 users said they would not stop the upload in progress to remove files from the queue.  They would wait until all files were uploaded and then remove them from the Gallery.  One user did mention that if she was doing it on her own she  probably would have played around with the pause.
  • 4 out of 4 users didn't immediately notice when the files were done uploading.  For a couple it took prompting for them to realize.  A couple users mentioned that it was pretty subtle.  They expected to be taken back to where they started from (browse images & collections) when the upload was complete.
    • Increase the visibility of the "File Uploaded" message.  How about "Upload Complete" in bold?  There is no reason to grey it out.  Users immediately think greyed out information is irrelevant.
  • The one user that used "remove" in the upload queue said it is too subtle.  She had trouble seeing that the file was actually gone (files name with default camera numbers).  She ended up removing extra files because she didn't think it was removing.

Demographics

The users were selected at random by the tester rather than through a formal recruiting process. Below table shows the demographics and previous experiences of each user we spoke to. We felt that such criteria as areas of study has little relevance to users' behaviour with file uploader, so we made conscious decision to leave them out.

Demographic Background

User

Gender

Age

Role

Type of Computer

UT User 1

Female

25-30

Grad Student

Windows laptop

UT User 2

Male

31-35

Grad Student

Windows laptop

UT User 3

Male

25-30

Grad Student

Windows laptop

UT User 4

Female

19-24

Student

Windows desktop

UCB User 1

Male

25-30

Staff

Mac laptop

UCB User 2

Female

35-40

Staff

Mac 

UCB User 3

Female

41-50

Staff

PC

UCB User 4

Female

31-35

Staff

PC at work; no home computer

Do you do any of the following online and if so how often?

User

Email

IM

Shop Online

Online Banking

Research Online

Online Classes

Social Networking

UT User 1

All the time

A few/week

A few/week

A few/week

All the time

A few/month

All the time

UT User 2

All the time

All the time

A few/month

A few/month

All the time

Hardly ever

A few/week

UT User 3

All the time

All the time

A few/month

A few/week

All the time

Hardly ever

A few/week

UT User 4

All the time

All the time

A few/month

A few/month

All the time

Hardly ever

A few/week

UCB User 1

All the time

A few/week

A few/week

Never

All the time

Hardly ever

Hardly ever

UCB User 2

All the time

All the time

A few/month

A few/week

All the time

Hardly Ever

Hardly Ever

UCB User 3

All the time

All the time

A few/month

Never

A few/month

Never

A few/month

UCB User 4

All the time

All the time

A few/moth

Never

Never

Never

Hardly Ever

Do you upload files on the web and if so how often?

User

Pictures

Media

Documents to course sites

Documents to social networking sites

attachment in email

UT User 1

A few/month

Hardly ever

A few/month

Hardly ever

All the time

UT User 2

A few/month

A few/month

A few/month

A few/month

All the time

UT User 3

A few/week

A few/week

A few/week

Hardly ever

All the time

UT User 4

A few/month

Never

Never

Never

A few/week

UCB User 1

A few/month

A few/month

A few/month

A few/month

A few/month

UCB User 2

A few/month

A few/month

A few/month

Hardly ever

All the time

UCB User 3

All the time

All the time

All the time

A few/month

All the time

UCB User 4

A few/month

Never

A few/week

Hardly ever

All the time

Code Block
Legend
All the time - A few/week - A few/month - Hardly ever - Never

Interaction Notes

Setup:

  • These tests were done in "hallway" style.  We grabbed people community areas and asked them to participate for a few minutes. 
  • All tests done on a Mac.
  • Removed emphasis placed on the key convention for selecting multiple files, since the users were PC users
  • Added a question at the end of Task 4 "Once you're done adding all the files you want, what would you do next?" to see how they exit the File Uploader

Although the PC users were asked to perform the tasks on Erin or Daphne's Macbook, they had little problems completing the tasks. If they were obviously looking for the ctrl key to choose non-consecutive multiples, they were prompted that on a Mac the apple key is the parallel.   We assume they would normally be completing this task on their own computer and we were not testing their knowledge of the keyboard shortcuts. 

Buttons the users clicked on are notated in [fluid:brackets].

Task 1:  Upload a file from the Photography101 folder on your desktop.

    Optimal Path:  Add Images > Browse > Select file > Upload > Browse Images & Collections

Task 2:  Upload the 4 most recent files in the Photography101 folder.

   Optimal Path:  Add Images > Browse > Select files > Upload > Browse Images & Collections

Task 3:  Upload all files in Photography101. <while upload progress is going> You realized that IMG_9409.jpg and IMG_9407.jpg are the wrong files and you didn't intend to upload those to the Photography class site. What would you do?

   Optimal Path:  Add Images > Browse > Select file(s) from desktop > Upload > Pause > Remove both files > Resume Upload > Browse Images & Collections

Task 4:  If you wanted to add a couple files from Photography101 and some files from your desktop, how would you go about doing that?

    Optimal Path:  Add Images > Browse > Select files from Photography 101 folder > Add more files > Select file(s) from desktop > Upload > Browse Images & Collections
 

User

Task 1

Task 2

Task 3

Task 4

General Observations

UT User 1

- [fluid:Browse]
- Double-clicked on file
- [fluid:Upload]
- "It's weird it doesn't show me the files uploaded"

- [fluid:Shift] to select multiple files
- "Nice that it tells me the progress"
- "This is pretty self-explanatory"

- [fluid:Pause]
- Sees that the files she wants removed have already been uploaded, "Here I can't do anything I guess."

- To exit, "I would click on the breadcrumb"
- Expects to go back to the "main page" to "reorder, add descriptions, pick cover photo, send it to people"

 

UT User 2

- [fluid:Browse]
- Picked the folder
- Picked an image
- [fluid:Select]
- acknowledges "Ready to Upload"
- acknowledges "file uploaded"

- "files are already sorted by dates modified"
- [fluid:Shift] to select multiple
- Four files are "ready"
- [fluid:Upload]

- "If it's just two wrong files, I would wait till the upload is completed. If it's more than half of the files that are wrong, I would pause and do the selection again."
- Looked for the two wrong files mentioned and said "I can't select these files" (expected to be able to remove even the already-uploaded files from this screen)
- Would go to "Browse Image Collection" to check to see where these files are

- "Remove is right here! for the previous task, I would've used this remove button."
- Feels that it should support two different kinds of remove: 1) remove from the "intention list" 2) if they are uploaded already "remove from collection"
- "It's surprising the rows can't be clicked on" referring to the rows in the table

 

UT User 3

- [fluid:Browse]
- "So you can upload all of these?" to which I answered Yes, and the user skipped Task 1

- [fluid:Shift] to select multiple
- [fluid:Upload]

- [fluid:Pause]
- [fluid:Remove] files he doesn't want
- [fluid:Upload] the rest

- Would click on "Browse Images"
- "Would be nice to see what the images look like on the actual site, like thumbnails of images, so I can see exactly what I've uploaded and remove some if needed"

 

UT User 4

- [fluid:Browse]
- "Can I upload all of these?" to which I answered Yes, and the user skipped Task 1

- Click and drag to select multiple
- [fluid:Upload]

- [fluid:Pause]
- When asked to take out the two images whose names start with "IMG_", the user observed the table and said "The ones I wanted to delete happened to be one of these ones (ones already uploaded)" and expressed she'd like to have the ability to remove uploaded files

- Added them separately. Clicked on [fluid:Browse], added files from Photography101, and clicked [fluid:Upload]. Then, clicked [fluid:Browse] again, added files from desktop, and clicked [fluid:Upload]
- Would like to see a "OK" or "Back to main" button for when she finishes uploading

 

UCB User 1

Task completed via optimal path


Task completed via optimal path



Task NOT completed

Path:
- [fluid:Add images]
- [fluid:Browse] 
- Selected all files from Photography101 folder using the [fluid:shift key]
- After being told there were a couple files in the the upload queue that he actually didn't want to upload, the user went to [fluid:browse all images] expecting the images to show up there as they were uploaded.  He said he would just remove them after they were uploaded.  He mentioned that he just really used to using breadcrumbs.

- User expected the upload to continue behind the scenes  while he went to [fluid:browse all images].
- User didn't notice the Pause button.  Not sure he would have used it if he had.

Task completed
Path via optimal path

Would like to see thumbnails of images in the upload queue.  Even if they were small, they could be "blown up" on rollover.

UCB User 2

Task completed via optimal path.
Other comments:

  • A second of hesitation before she realize she needed hit the upload button.  She thought she was done after browsing.
  • A few seconds after upload was done she said, "Am I done?"
  • Not enough indication that something has changed after the files are uploaded.  She wasn't sure it was done.  Then she said, "It's all about once you know, you know".

Task completed via optimal path.
Other comments:

    Task NOT completed because user went to "Browse Images & Collections" before upload was complete.  Had she not done that she would have completed (assuming entire system was working) but not as expected.  User said she would let all the files upload and then delete the ones she didn't want.
    Other comments:

    • Assumed she could just upload the entire folder.
    • Went to "browse

    Task completed.

    Other comments:

    • She tried to select files from folder and then from desktop on same browse.  As soon as she realized the first files weren't in the queue she realized what happened and completed as expected.

    Wanted to upload folder but could not

    UCB User 3

    Task completed via optimal path.

    Other Comments:

    • On upload page, she was looking for her desktop -- "didn't think of browse"
    • She asked "how do I get back" but pretty quickly noticed the link at the top for Browse images & colllecitons.
    • Image Gallery feedback:
      • Took a long time to find 'Add Images'.  * Tried to use "Find" to browse for files (to upload).* Throughout tasks she wanted to go to Add Collections for add files.  She said she was thinking "add to collection".  Perhaps we could use "Create new collection" to make it clearer.

    Followed optimal path but Task NOT completed because user went to Browse Images & Collections before upload was complete. 

    Other Comments:

    • She was trying to get into 'Recently Added" playlist since the task mentioned recently added files.  Task should be updated not to user terminology in the interface.
    • Immediately used keyboard to select multiple at once.
    • Went to "Browse Images & Collections" before upload was complete.  She expected it to continue.

    Task completed (assuming entire system was working) but not as expected.  User said she would let all files upload and then go back and delete the ones she didn't want.  She followed optimal path through the first Upload then let the system upload them all. 

    Other commments:

    • She mentioned that had she been on her own she would have probably tried "pause".

    Task completed via optimal path.

    Other Comments:

    • "I would upload them from the folder first and then go back and upload the ones on the desktop".  After choosing from the folder, she quickly noticed the "browse" below the queue and used it to choose additional files.
    •  Seems like a clean layout
    • Image Gallery - icons aren't clear.

    UCB User 4

    Task completed via optimal path although it took awhile to realize uploading was done.

    Other Comments:

    • Mentioned the progress message was very dark.
    • Status went away and she said, "and now it's gone and I'm still on the same page". 
    • She said she expected it to take her back to where she started once the upload was done (Browse Images & Collections).  She "felt stuck". She suggested that if not taken back then adding an additional button on the upload page to 'take be back'.
    • She had a slight hesitation before realizing she needed to select "Upload" after selecting the images.

    Task completed via optimal path. 

    Other Comments:

    • "Oooh, it worked, that is spiffy" she said after selected multiple images at once.
    • Chose "Browse Images & Collections" before upload was complete.

    Used optimal path but did NOT complete task.  She removed more images than expected because she wasn't seeing them get removed from the queue (too quick and subtle).

    Other Comments

    • She said, "easy to remove but visual cue that file has been removed is too subtle."
    • She noticed the countdown during this task. She said since she was focused on the remove column her eye was drawn to it (it is left aligned in the table like the remove). 

    Task completed via optimal path.

    Other Comments:

    •  She mentioned that that first thing she tries when uploading is selected multiples even though she doesn't expect it to work in many places.
    • Image Gallery Issues:
      • She said she found herself drawn to the "Add Collections" so it took a few minutes  before she found the "Add Images" button.

    Post-test Questionnaire Responses

    The users were asked to evaluate their experience on Likert scales after the test. The questions ask how easy or difficult it was to do each of the following:

    User

    Uploading single

    Uploading multiple

    Uploading additional

    Cancel upload

    From 2 different locations

    UT User 1

    Very Easy

    Easy

    Easy

    Very Easy

    Easy

    UT User 2

    Very Easy

    Very Easy

    Very Easy

    Neutral

    Easy

    UT User 3

    Very Easy

    Very Easy

    Very Easy

    Easy

    Easy

    UT User 4

    Very Easy

    Very Easy

    Very Easy

    Very Easy

    Neutral

    UCB User 1

    Easy

    Easy

    Very easy

    Easy

    Easy

    UCB User 2

    Very Easy (once noticed "Add Images")

    Very Easy

    Easy

    N/A

    Easy

    UCB User 3

    Between Difficult & Neutral.  She said it was easy after first time.

    Easy

    Easy

    N/A

    Easy

    UCB User 4

    Very easy

    Very Easy

    Very Easy

    Easy

    Very Easy

    Potential Design Improvements (based on testing)

    File Uploader Component

    • Users expected items (entire row) in upload queue to be selectable to remove them.
    • Users would like to be able to remove files from upload queue even if they have been uploaded.  Can we do this?
    • Display thumbnails in the queue.
    • Include a lightbox-window style for next round of testing. 
    • Allow users to upload a folder and automatically "unpack" it for them.
    • Make the pause button pop a bit more.
    • Increase the visibility of the "File Uploaded" message.  How about "Upload Complete" in bold?  There is not reason to grey it out.  Users immediately think greyed out information is irrelevant.
    • Removing from queue is too subtle.  Particularly when files are named with camera defaults.  Need to be more obvious that something happened and the file is gone.
    • Need to help users know what to do after the upload is complete OR take them back to where they came from once the upload is complete.  If we don't take them back and we use the lightbox overlay pattern, perhaps a prominent message at the top could help, "Upload complete, you may close the window to continue your work."
    • After upload is complete, take user back to where they came from.  The main use case is upload a set of files so it makes sense to first design for it.  In the case where users want to upload more than one batch, it only requires them 1 click to choose "add files" (or whatever took them to the upload).
    • Button pattern usually has the main action left of the cancel button (or right is buttons are right-aligned).  Change the order of buttons so Upload is far left.

    Design Pattern

    • Design Pattern - newly added files should be in clear view when return to page they were adding from.  Users should easily be able to remove files from there.
    •  

    User Test

    • Check for leading task language.  We really should not be using "upload" in task.  We also don't want to use any terminology in the interface.
    • Post-test questionnaire should ask about ease of uploading the 1st time and then repetitive times.  Users mentioned that it wasn't as easy the first time but once they have done it, they knew.