Sakai User Experience Walkthrough Report 

(Accessibility and usability heuristic evaluations combined with cognitive walkthroughs)

*Evaluation Completed by:Kathy Moore

http://wiki.fluidproject.org/pages/editpage.action?pageId=1705336

9/10/07
Heuristic reviews and cognitive walkthroughs were performed on a Dell pc using an 1152px resolution and millions of colors using Win xp Firefox 2.0.0.6 on a T1 connection.

See Heuristic Evaluation for heuristics used in this evaluation.

Scope of Walkthrough

Announcements, Home, Assignment, Drop box, Email Archive, Forums, Mail tool, News, Resources, Syllabus, Web content,

*User profile(s) and context of use: Ed, student

Profiles will generally only be used for the cognitive walk-through, not the heuristic evaluation. However, users may be set up with a certain role in the system (e.g. student, instructor) for the heuristic evaluation.

•    See Sakai Personas

Scenarios

Scenarios will generally only be used for the cognitive walk-through, not the heuristic evaluation. However, it is fine to use these scenarios for the heuristic evaluation if it is helpful to evaluators in figuring out how to walk through the application. See Content Management Scenarios for a list of all possible scenarios. Evaluators will likely only cover some of the scenarios, in one of the roles, in their individual evaluations.

• General Overview (heuristic evaluation)
• Scenario: Your instructor has made next week's assignment available.  You are to write a 10 page paper reflecting on what you've learned half way through the term.  Complete and turn in the assignment (since this is a paper you likely do this over a number of sittings).  NOTE:  You must have created this assignment as an instructor first.
• Scenario: Each week you need to post a topic and comment on one other student's post.  Do your homework for this week's topic.  You can respond to your own comment.
• Scenario: Figure out what the instructor expects you to do this week for class.
  

Assumptions for this evaluation:

•    [List any additional assumptions used in evaluation]

 Positive Findings

All positive findings, from both the heuristic evaluation and the cognitive walk-through, should be placed in this section. 

Summary of Usability & Accessibility Issues Found 

All usability & accessibility issues found, in both the heuristic evaluation and the cognitive walk-through, should be placed in this section. 

Priority Legend:

High = Task cannot be completed

Medium = Task completed with significant effort and failed attempts

Low = Task completed with minor complications and/or annoyance 

Usability Issues	Principle	Link to screen shots	Priority	Suggestions for solution	Tool	Component Identified?
Tool menu difficult to scan, no obvious organizational principle	Flexibility and efficiency of use Is the order of information natural and logical?		Medium	-   Alphabetize, -   Provide alphabetize button -   Let user customize	Home	Drag and drop utility?
In Syllabus tool, document attachment doesn't offer immediate view of content	Flexibility and efficiency of use		Low	Is there a way to show word or pdf doc in own i-frame?	Syllabus
In Syllabus tool, print button appears when only an attachment is visible; it has no meaning since it vanishes when document opens and can be printed.	Error prevention Match between system and real world		Medium	Do not display print button if the only content is an attachment	Syllabus
In forums, page is blank if there's been no instructor input	Visibility of system status; Help and documentation		Medium	Display default text for students (and instructors?)	Forums	Automatic notification of last changes/ next step
In Worksite setup, By default, student does not have ability to do anything but edit/delete "My Workspace" Should student be allowed to delete "My Workspace?"What has to happen for student to create project site?	Visibility of system status; Help and documentation  Error prevention		Medium		Worksite setup	Default text for students (and instructors?) Automatic notification of last changes/ next step, customized for role
Default name for "Web Content" tells user nothing.	Does a page title accurately describe its purpose?		High	Do not allow user to add tool without custom name; eliminate default in "add tool" process for this tool.	Web Content
Initial News page and links come up in Sakai but  clicked links open in new tab	Visibility of system status; Consistency (and standards)		Medium	Continue to show content within Sakai?(Discuss accessibility of alternatives.)	News	Provide portal-like display widgets to give user control of link display. (uPortal/LMS convergence?)
In absence of clear direction from Instructor, student must look in many places to see what is new in a class. (Syllabus, Schedule, Announcements, Assignments, Gradebook, Forums, Resources	Flexibility and efficiency of use		High: not  insurmountable but a constant annoyance likely to impact success of pedagogy	A "What's New" tool /utility might be a good thing This could be useful in uPortal as well.	Home	A "What's New" tool /utility might be a good thing This could be useful in uPortal as well.
Permissions section in Forums is forbidding and intrudes itself into student attempt to post Topic	Aesthetic and minimalist design		Medium	Allow faculty to set defaults, subject to change only upon request; Design pattern: Details on request	Forums	New permissions utility
Does "Gradebook Assignment" belong under the "Permissions" heading?	Are headings and titles clear and distinct?		Medium	Put this elsewhere and label it meaningfully	Forums
Submit buttons in Forums bury "Go" among too many choices	Is navigation consistent and intuitive? Does the application follow platform and usability conventions?		Medium	Rethink integration of settings here Break buttons into two rows; one shows standard Save/Cancel (Post Topic/Cancel)	Forums
No confirmation after posting	System visibility: Does the system always keep users informed about what's going on?		Medium		Forums
There's more -- but is this too much detail on one tool, in the absence of component ideas?					Forums

Accessibility Issue	Principle	Link to screen shot	Priority	Suggestions for solution	Tool	Component identified?
False? User is forced to tab through sometimes-lengthy tool menu to reach tool controls within page  (Is this true?)	Are shortcuts or skip links provided?		Medium	Are there shortcuts that aren't obvious to me?	All	Shortcut utility/coding
I-frames constrain type awkwardly when user requires small screen/big type	Is text still visible? Does it overlap?		Medium		Home
WSYWYG editor doesn't seem keyboard-accessible; Can't tab to formatting choices Can move cursor to text entry box, but typing does nothing	Are there any areas where you get stuck and have to use a mouse?		High		Schedule, others
Action buttons in Resources (Add, Actions) do not seem accessible by tab			High		Resources

Cognitive Walkthrough Worksheet

This section should be used by evaluators to keep track of the steps & screens they followed through the application in the cognitive walk-through. Any positive results or issues found should be included in the two sections above.

• Scenario: Your instructor has made next week's assignment available.  You are to write a 10 page paper reflecting on what you've learned half way through the term.  Complete and turn in the assignment (since this is a paper you likely do this over a number of sittings).  NOTE:  You must have created this assignment as an instructor first.

Tool: Assignments

#	Step	Screen	Comments/issues	Principle	Suggestions for solution
	1		After opening an assignment, the student sees buttons above and below the assignment leading back to the full listing. Good navigation.	Flexibility and effieciency of use
	2		After beginning the assignment and saving a draft, the user who opens an assignment sees 'Assignment - In progress Fill out form. Make sure you click "Submit".	Clear indication of status
	3		Feedback that the assignment has been submitted is very clear, in the assignments listing and because the user can send himself a confirmation.	Clear indication of status

Scenario: Each week you need to post a topic and comment on one other student's post.  Do your homework for this week's topic.  You can respond to your own comment.

#	Step	Screen	Comments/issues	Principle	Suggestions for solution
1	After filling in other fields, scroll down to find submit button; whoops, there's more...		The permissions section is more than I need. Should students even see this?I don't know much about most of the custom roles, do I? Can't I do a blanket setting?	Aesthetic and minimalist design	Make permissions a section to be opened on demand rather than displaying all this by default
1	After filling in other fields, scroll down to find submit button; whoops, there's more...		Does "Gradebook Assignment" belong under the "Permissions" heading?	Are headings and titles clear and distinct?	This might make more sense somewhere else on the screen.
1	Still looking to submit new topic		"Post" button is buried in middle of four. Finding the right ("post message") button requires a larger mental load than necessary.	Is navigation consistent and intuitive? Does the application follow platform and usability conventions?	Rethink integration of settings here. Break buttons into two rows; one shows standard save/cancel
2	Add Topic (with third button above)		No confirmation	Visibility of system status	Show confirmation or list of topics
3	Try to "reply" (thinking that topic has begun the thread)		No obvious way to reply to post ("You can respond to your own comment." ??)	System Visibility
3	Try to reply, post new message		There's a lot of room for confusion between "forum." "topic," and "thread."	Match between system and real world;Is plain language used?

1.  
2.  
3. Scenario: Figure out what the instructor expects you to do this week for class.

Scenario 4:  Left in place just in case...