Skip to end of metadata
Go to start of metadata

Sakai User Experience Walkthrough Report 

(Accessibility and usability heuristic evaluations combined with cognitive walkthroughs)

Evaluation Completed by:    [Insert evaluators name]

URL:  http://bspace.berkeley.edu/portal/site/24cc0997-e57a-4f55-00ca-02eb0b35a6e7

Date: 22 Oct 07, 29 Oct 07

Heuristic reviews and cognitive walkthroughs were performed on a [Dell] at [screen resolution] resolution (most common resolution) and [color setting] colors (most common novice resolution; if graphics work at this resolution, they'll work in higher resolutions) using [Firefox  and Win XP Pro SP2] at a [broadband] connection rate.

See UX Walkthrough Heuristics for heuristics used in this evaluation.

Scope of Walkthrough

•    See UX Walkthrough Work Breakdown

User profile(s) and context of use:

Profiles will generally only be used for the cognitive walk-through, not the heuristic evaluation. However, users may be set up with a certain role in the system (e.g. student, instructor) for the heuristic evaluation.

•    See Sakai Personas

Scenarios

Scenarios will generally only be used for the cognitive walk-through, not the heuristic evaluation. However, it is fine to use these scenarios for the heuristic evaluation if it is helpful to evaluators in figuring out how to walk through the application. See Content Management Scenariosfor a list of all possible scenarios. Evaluators will likely only cover some of the scenarios, in one of the roles, in their individual evaluations.

•    General Overview (heuristic evaluation)
•    Scenario 2
•    Scenario 3
•    Scenario 4
•    Scenario 5

Assumptions for this evaluation:

•    [Performed based on an instructor]

 Positive Findings

All positive findings, from both the heuristic evaluation and the cognitive walk-through, should be placed in this section. 

Usability Positives

Tool

Evaluator

Clicking the help icon takes you directly to the help section for the tool it was called from.

all

JO

Sortable tables show an arrow icon beside a heading. This shows both the method of sort (ascending/descending) and the heading which for which the sort is based on.

all sortable tables (i.e. announcements and resources)

JO

Users are able to include only the tools they want for their site

 

JO

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Accessibility Positives

Tool

Evaluator

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Summary of Usability & Accessibility Issues Found 

All usability & accessibility issues found, in both the heuristic evaluation and the cognitive walk-through, should be placed in this section. 

Priority Legend:

High = Task cannot be completed

Medium = Task completed with significant effort and failed attempts

Low = Task completed with minor complications and/or annoyance 

Usability Issues

Principle

Link to screen shots

Priority

Suggestions for solution

Tool

Component Identified?

 General

 

 

 

 

 

 

Use of the term  "Reset" for the mechanism to  take the user back to the first page of the tool. Reset is often used to refer to clearing the information entered and not just returning to a previous location.

Consistency and standards

 

 

A term such as "Return" may be more appropriate

All

 

There are three methods to organize content. Method 1 uses up/down arrows seen in the Syllabus, Method 2 uses drag and drop seen in the Site Info, and Method 3 uses numbers in a combo box  seen in the Forums

Consistency and standards

 

 

Choose one or at most two methods of organization.

Syllabus, Site Info, Forums

Syllabus Items, Tool Order, Organize Forums and Topics

Should the instructor be able to set permissions. Is this more of an admin function?

Aesthetic and minimalist design

 

 

Remove from the instructor's view

Announcements,
Resources, Chat Room, Email Archive, Polls, Schedule, Assignments

Permissions

Both the Question Pools and Resources pages are used to manage a hierarchical system of information, but are designed with a different look and feel.

Consistency and standards

 

 

Choose one method to be used in all necessary areas

Resources, Quiz

Site Resources, Question Pools

The horizontal navigation bar used for navigation within a tool is different across the tools, some user lighter text to show selection, others use darker, some are have vertical bars to separate items, others do not, some have no underline for the selected item, and etc.

Consistency and standards

 

 

Choose one method to be used in all necessary areas

All

 

Syllabus

 

 

 

 

 

 

Complicated work flow if the instructor just wants to add a file as the Syllabus. (Syllabus > create/edit > add > add attachment > upload file

Flexibility and efficiency of use, Match between system and the real world

 

 

Add an option along side create/edit to add a file. (System should probably also be able to display the contents of the file and not just a link)

Syllabus


The Add Syllabus form and editor do not display the same as the actual output. The style of bullets changes from the editor to the output. Also the preview shows the Title in a different font size and colour than is actually outputed.

Consistency and standards

 

 

The preview should be identitcal in appearance to the output on the site. The user may also like to be able to control the style of the Title.

Syllabus

Add Syllabus

In the create/edit screen of the Syllabus tool, the "Preview" and "Reset" link perform the same actions.

Consistency and standards, Aesthetic and minimalistic design

 

 

If there are no other functions for the "Preview" link to that what is performed by the "Reset" link, "Preview" should be removed.

Syllabus

create/edit

Redirecting a syllabus prevents other posted syllabus from being displayed. This is the case even after adding a new syllabus and verifying it's output with preview.

Error prevention, visibility of system status

 

 

The system should inform the user that a redirected website will block all other syllabus, or display all of them sequentially.

Syllabus

Redirect

To remove a redirected syllabus (web site) the user has to erase the url and push the save button. For all other syllabus the user can remove after it by going to create/edit, which isn't even viewable if a redirect has been used.

User control and freedom, consistency and standards, error prevention

 

 

Provide an item in the syllabus list (create/edit) so that all of the syllabus can be managed in a like manor.

Syllabus

Redirect, create/edit

When re-ordering syllabus in the Syllabus Item list, the user does not need to click the update button for the changes to take effect. Update is only needed for removal.

Error prevention, User control and freedom, Visibility of system status

 

 

Require that the user perform an update before the reordering takes effect, or rename the button to "Remove". Also disablling the button until an update/removal is needed will signal to the user when it is needed.

Syllabus

Syllabus 

Home


 





On the "Home" page, clicking "Options" under Worksite Information allows you to add text. However, clicking on "Options" under Recent Announcements provides no ability to add text.

Consistency and standards

 

 

Provide a link to the announcements editing page.

Home

Recent Announcements

The "Title" and "URL" text fields are both smaller in width than the "Description" text area

Consistency and standards

 

 

The text fields do not limit the input to their size, so they should be equal in width to the Description text area, allowing for maximum visibility of information.

Home

Worksite Information > Options

Site Info

 

 

 

 

 

 

Should an instructor be able to add/remove users and change their role? Is this more of an admin function?

Aesthetic and minimalist design

 

 

Remove these features from the instructor, or put them under a subheading

Site Info

 

The Edit Tool and Tool Order links have the same function but different implementations of it (with the exception of the ordering option for tool order)

Consistency and standards, Aesthetic and minimalist design

 

 

They should probably be combined or have the titles changed so that one indicates for keyboard use and the other for mouse.

Site Info

Edit Tools, Tool Order

Slow to add/remove tools

Visibility of system status

 

 

There is a message displayed after the tool has been added/removed. There should also be a message indicating that the process is in progress.

Site Info

Tool Order

The user can drag to add tools but can't drag to remove tools

Consistency and standards

 

 

The user should be able to drag the tool off of the column and drop it either onto the available tools column or drop anywhere off the current column to remove.

Site Info

Tool Order

Removed tools do not show up in available tools until after a save is performed.

Visibility of system status

 

 

Tools should show up in the available tools list as soon as they are removed from the list of current tools.

Site Info

Tool Order  

The "Reset" button is used to put the order of the tools to the default ordering. The term  "Reset" is also used at the top as a means to return the tool to it's start page.

Consistency and standards

 

 

One or both of these items should be renamed. The "Reset" button could be renamed to "Default Order".

Site Info

Tool Order

Assignments 

 

 

 

 

 

 

The descriptive text is unclear as to whether is is describing the combo boxes above or below it.

Error prevention

 

 

Information in the Assignment form needs to be borken up by white space to better seperate the sections.

Assignments

Add

Polls

 

 

 

 

 

 

When adding a poll, the user has to add the question and all possible answers on different pages. The only item that persists across pages is the question. If the user forgets what answers were already entered, they have to save, leaven and exit.

Recognition rather than recall

 

 

The user should be able to add the question and all possible answers on the same page.

Polls

Add

Schedule

 

 

 

 

 

 

When setting the time for an event, the time zone is displayed after the time combo boxes. However, the user is unable to change the time zone.

Aesthetic and minimalist design

 

 

The time zone seems to be set by the server, if the user has no control over it, it should not be displayed here.

Schedule

New Event

The legend below the Calendar is quite large and contains many items that may never appear on any one calendar all at a time.

Aesthetic and minimalist design

 

 

It would be better to only include those items which are currently visible in the calendar. Another solution would be to have a tooltip for the icon rather than include the icon in the event's title.

Schedule

Calendar

To scroll through the hours on the calendar, the user has to click on the links for earlier and later.

Consistency and standards

 

 

Users are often more familiar with using a scrollbar.

Schedule

Calendar

Announcements

 

 

 

 

 

 

The update button in the Announcements section is only used for removing announcements.  This can cause confusion as the user can sort the list based on the table headings; however these changes don't affect the announcements posted on the site.

Error prevention, User control and freedom, Visibility of system status

 

 

Rename the button to "Remove". Also disabling the button until an announcement has been checked for removal. Another option would be to have a removal button next to each announcement instead of a check box.

Announcements

 

Resources

 

 

 

 

 

 

It is not clear what the resources are for. Items such as Quiz and Syllabus allow the user to upload the resource directly and don't seem to link to the resources for the user to select an item from there.

Consistency and standards, Aesthetic and minimalist design, Match between system and the real world

 

 

 

Resources

 

When viewing other site's resources the user is given the same access level to the files but does not have access to the sort methods that they would on resources from their own site. Also, the items are listed with all of the information that the items in the user's table but the headings aren't easily matched.

Consistency and standards

 

 

When selecting a folder from another site it should be broken off to show the table headings for the columns that are displayed. This will allow the user to sort, understand what the information is, and separate it from the other resource folders available.

Resources

Site Resources

When creating a new folder, uploading a file, or adding a web link there is an ' X ' at the far right. The tooltip for the ' X ' says "Remove this item". Clicking the ' X ' does nothing and there is already a cancel button at the bottom.

Consistency and standards, Aesthetic and minimalist design

 

 

remove the ' X '

Resources

Upload files, Create folders, Add web links (urls)

When adding a new folder there is an option to add another folder. This can be done when the first folder isn't created. Also it is unclear whether this other folder is meant to be a sub folder to the original or the one you are creating.

Error prevention

 

 

Only show the add another folder link once the original folder has been specified, or have an add multiple folders option. Also allow the user to add subfolders for the newly created folder, thereby distinguishing the levels of the folders created.

Resources

Create Folders

The text document and the HTML page result in similar documents. The exception being that the HTML page has more formating options

Consistency and standards

 

 

Rename the html page to something like rich text document and/or remove the simple text document.

Resources

Create HTML page, Create text document

The options section says " Select or unselect resources types to indicate whether resources of that type can be created in this site." Currently it is only a list of checked items which cannot be edited, except for Form item. Unchecking Form item and updating does not seem to have any effect.

Consistency and standards, Aesthetic and minimalist design

 

 

If these items are set by another user (i.e. an admin) this section shouldn't be called options, as all other instances of options contain editable options. It may be better to include it as a list somewhere on the main resources page or to omit it completely; as the user can easily determine the types of resources by hovering over add.

Resources

Options

When hovering over add or actions when they are close to the bottom of the browser window, the context sensitive menus are cut off.

Visibility of system status, Error prevention

 

 

The menus should open upwards when at the bottom of the screen instead of always opening downwards.

Resources

Site Resources

Can't collapse items into the top level folder.

Consistency and standards, Aesthetic and minimalist design

 

 

Allow the user to collapse contents of the top level folder.

Resources, Drop Box

 

Roster

 

 

 

 

 

 

There are two views of the roster, Overview and Pictures. The information provided for the picture view is cut off.

Aesthetic and minimalist design, Recognition rather than recall

 

 

Rather than having two separate pages, there should be one page with an option (e.g. a check box) to show pictures. When show pictures is selected the image should appear in a new Pictures column next to the name.

Roster

 

The term "Roster" may not be immediately relevant to all users

Consistency and standards, Match between system and the real world

 

 

A term such as course list, course users, or course participants may be more applicable.

Roster

 

Chat Room

 

 

 

 

 

 

Chat room does not show current user in the "users in chat" list, to the current user. This prevents the user from knowing if others can see they are there.

Visibility of system status

 

 

The users should see everyone available in the chat, even themselves

Chat Room

 

Email Archive

 

 

 

 

 

 

There are two separate tools, one for sending e-mail (Mailtool) and one for receiving e-mail (Email Archive).

Match between system and the real world, Consistency and standards, Aesthetic and minimalist design

 

 

The MailTool and Email Archive should be rolled into one Email tool. If the instructor is the only one who needs to send e-mails this portion should be blocked from the students. Users are used to having their email clients able to both send and receive e-mail.

Email Archive, Mailtool

 

Forums

 

 

 

 

 

 

When adding a new Topic the user selects "New Topic", when the page opens it is called "Topic Settings"

Consistency and standards

 

 

Keep the name the same, the page should also be called New Topic

Forums

Topic Settings

The buttons at the bottom of the Topic Settings page include "Save Settings" and "Save Settings & Add Topic". The first implies that only the settings are being saved and the second that the settings and the topic are being saved. In actuality "Save Settings" saves the topic and the other saves and opens a new page to add another topic.

Match between system and the real world, Consistency and standards

 

 

Change the names to something like "Save Topic" and "Save & Add Another Topic"

Forums

Topic Settings

Gradebook

 

 

 

 

 

 

When a category at the top is selected the text colour becomes a lighter shade to indicate that it is the currently selected category. However, the Upload/Import link is always a lighter colour

Visibility of system status, Consistency and standards

 

 

Make the text colour for Upload/Import follow the same conventions of the other links, being a darker shade when not selected.

Gradebook

 

News

 

 

 

 

 

 

Descriptive text between Page Title and URL text fields is not clear as to whether it refers to the field above or below.

Error prevention
       

 

 

Use whitespace to separate it from the lower field.
          

News
               

News Options

Not clear that the URL field requires the URL for an RSS feed

Error prevention

 

 

Indicate that an RSS feed is required. Change "URL" to "RSS feed URL"

News

News Options

Web Content

 

 

 

 

 

 

The web content does not show the browsers icons when opened in a new page. Many users may not know the keyboard shortcuts for those actions.

Recognition rather than recall

 

 

Allow the page to load with a normal browser view, or provided icons within the browser to fulfill those functions.

Web Content

 

Wiki

 

 

 

 

 

 

When editting a wiki, clicking on the modifier icons for bold, italic, and etc, repeatedly doesn't remove a modification but adds it again instead.

Consistency and standards, Error prevention

 

 

A second click should remove the modification

Wiki

Wiki Editor

Home and View appear to perform the same function. Both bring the user back to the home page of the wiki.

Aesthetic and minimalist design, Consistency and standards

 

 

If it performs the same function remove "view"

Wiki

 

Mailtool

 

 

 

 

 

 

There are two separate tools, one for sending e-mail (Mailtool) and one for receiving e-mail (Email Archive).

Match between system and the real world, Consistency and standards, Aesthetic and minimalist design

 

 

The MailTool and Email Archive should be rolled into one Email tool. If the instructor is the only one who needs to send e-mails this portion should be blocked from the students. Users are used to having thier email clients able to both send and recieve e-mail.

Email Archive, Mailtool

 

Drop Box

 

 

 

 

 

 

Can't collapse items into the top level folder.

Consistency and standards, Aesthetic and minimalist design

 

 

Allow the user to collapse contents of the top level folder.

Resources, Drop Box

 


Accessibility Issue

Principle

Link to screen shot

Priority

Suggestions for solution

Tool

Component identified?

General

 

 

 

 

 

 

It isn't always clear which item is selected when tabbing through a page. The selection bounding box is not always visible.

 

 

 

 

 

 

After selection of a tool from the left navigation bar, the tab traversal resets and cycles back from the start.

 

 

 

It would be more helpful to the user if the tab traversal starts inside of the tool's frame

all

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

  • No labels