Formalizing our Usability Tests

 

On Deck: 

1. Review and Discuss our Assignment Prompt
2. Discuss Your Usability Test Plans 
3. Look at samples 
4. Use your Plan to Prepare:
1. your usability test (with "procedures" and "script")
2. your data collection artifact 
5. Recruit test readers and be recruited (you will need at least two tests, but both do not have to be users from this class)

 

Due Next Class:

• Your Usability Test and Data Collection Artifact

---

 

 

 

 

 

 

 

 

 

 

User Test Stage 1 - The Usability Test

    

1) Review our (slightly revised) Assignment Prompt:

 

For this assignment, you will conduct a user test of a nearly finished draft of your set of technical instructions from the previous assignment (see Research).  

 

In order to conduct a user test and write a report memo successfully (see 2 C), you will need to look carefully at the sample user test designs and user reports in Anderson, Chapter 18.  You will then proceed with the following:

 

Preparation:  Prepare a user test (involving a 'procedure' and 'script') and data collection artifact as described in your text and our class lessons.  This test should not take longer than 15 minutes for an average user to complete.

 

Design a method and instrument to collect usability data from your target readers.  If you can, arrange for a performance test, where readers will actually perform the process in your set of instructions You can design an observational instrument to collect data from a performance test.

 

If it is not feasible to arrange a performance test (e.g., if your instructions describe how to perform an autopsy), arrange for a location test or an understandability test.  You can design a questionnaire or interview instrument to collect data from a location or understandability test.  

 

Research:  Conducting a user test is primary research.  Identify 2-3 people to serve as your test readers and arrange for them to work independently.  Ideally, test readers should have varying degrees of experience.  

 

Your testing must produce some sort of data collection artifact which can be submitted as an appendix along with your Script, and User Test Memo (stage 2 C of this assignment).

---

Samples:

1. Sample Usability Test.docx   (from Wayne State Composition, me)
2. From The Web Editor's Handbook
3. ... 

---

Preparing the Formal Sections of your User Test's Procedure and Script

 

The following are the formal sections for Web site usability testing, including common or "Recommended procedures" and Tips for your "Script".  You will need need to adapt these sections to the goals of your test (which is detailed in section one).

 

Procedure and Script:

 

Section One:  Purpose of the Test (required section)

 

*Note: this will be part of your script, and you will need to read this to your tester

 

• First:  Tell them clearly what and why are you testing: is it a performance test, location test, understandabilitytest, or some combination of these?  Tell them briefly why you chose this type of test, or combination of tests.

 

• Second:  Use the following "Invention Questions" to determine what else you want to establish as your goals for the test, then include this in formal prose in this first section:
• With all tests, your PURPOSE is to discover whether the user:
• gets the point of the page(s)
• understands the navigation system
• can guess where to find things.
• In a general (location and understandability) tests you MAY want to know:
• how do users interact with the web site you are testing?
• what is difficult for people to do?
• where do they get lost?
• what makes sense to them?
• what makes them feel distrustful or insecure?
• what do they like and what do they dislike?
• In a specific (location or performance) tests you might want to know, for example:
• can the user perform or accomplish a key task?
• can the user find something specific?
• Third, include Additional Introductory Items:
• Introduce yourself
• Reassure, establish rapport: 'Please think aloud,you cannot make mistakes, we are testing the site (not you).'
• Clarify: the key purposes of a test, reiterate the goals of 'understandability', 'location', or 'performance'.

 

 

Section Two) Key Definitions and Roles (required section)

 

*Note:  this will be part of your script, and you will need to read this to your tester.  YOU WILL NOT HAVE TO INCLUDE ALL ROLES LISTED HERE.

 

 

• Facilitator:  the person who is guiding the user through the test, and taking notes.
• Observer:  someone else observing the test, if necessary
• Owner:  the owner of the web site.
• Web site development team:  everyone involved with developing the web site including the strategy group, designers, programmers, stakeholders, etc.  
• Usability:  this generally means the extent to which the intended user can meet his or her goals using the website being tested.
• Users:  different persons who use the web site in a usability test.  Note, you might mention here that you are looking to test a range of users, but it is required that you test those:
• not involved with drafting the web site in any way (including your revision workshop)
• new to the web site (so don't ask the same person twice to be a user)

 

 

 

Section Three) Length of Test (required section)

 

State how long your usability test runs:

• Tests range from 5 minutes (for a single page design) to 1 hour (for a general response to a whole site or new design).

 

 

 

Section Four) Confirmation (required section)

• use a sample from above or p. 390 to create your confirmation signature form

 

 

 

Section Five - ?)  Remaining Script (procedure and script components to work in to your Test)

A script is part of your own quality assurance system: it helps ensure that you follow procedures, and that you are asking each user to do the same task.

 

Potential Sections to Include in the Remaining Script:   User Background Questions  Questions about Initial Site Navigation  Tasks Scenarios Interview Questions Ending the Test

 

 

i. User Background Questions

• Ask relevant questions about your User
• What is your major and class ranking (sophomore, junior, etc)? 
• Have you taken microbiology before?
• If not are you planning on taking microbiology?
• Have you used/made X (Baklava, a bathroom, creative suite) before?
• Would you consider making/using it?
• ...

 

 

ii. Ask Open-ended questions about "Initial Site Navigation": 

• Use "think aloud protocols" to generate feedback
• tell your reader what a 'think aloud' protocol means (you can see p. 383, or use language from a sample above to describe this) 
• Use general questions to keep them thinking about your site, for example:
• "First, I’m going to ask you to look at this page and tell me what you make of it: what strikes you about it, whose site you think it is, what you can do here, and what it’s for. Just look around and try to explain your observations out loud."
• "On first glance do you think that you could perform this instruction set ..."
• "Tell me more about your thoughts on whether or not the site motivates you/appeals to you/seems professionally done..."
• ... 

 

 

 

iii. Create "Tasks":

• State a task or tasks.
• Remind user to continue to 'think aloud'
• Design either performance tasks, location tasks, or understandability tasks:
• e.g. "Now I want you to try and execute steps 1 and 2, talking me through your process as you read the steps and apply them" or "now I want you to discuss whether or not you could"  
• e.g. "Now I want you to locate, as quickly as possible, the alternate method for creating a .gif file with photoscape" 
• e.g. "Now I want you to paraphrase steps three, four and five for me, telling me what you think you have to do" (see strategies on Anderson p. 585)
• Allow users to try to accomplish the task in their own way.  Don't interrupt, just remind them to think aloud if necessary 

 

 

 

 

iv. Create "Scenarios": 

(Note: these are create solutions that asking users whether or not they could, in specific scenarios, perform tasks)

• State a scenario or state that you will walk the user through a several scenarios
• Remind user to continue to 'think aloud'
• Design either performance scenarios, location scenarios, or understandability scenarios:
• e.g. "Now I want you to imagine you are in the following scenario, you're planning a party and want to make baklava the night before the get-together, can you discuss whether or not you could use the instruction set to accomplish the following tasks..."
• e.g. "Now I want you to imagine that you're only interested in diagnosing the 'tonic phase' of a seizure, and I want you to locate, as quickly as possible, the diagnositic steps for this phase" 
• e.g. "Now I want you to imagine that you have to explain steps 3, 4 and 5 to a novice. Can you paraphrase steps three, four and five for me, teaching them how to to this?" (see strategies on Anderson p. 585)
• Allow users to try to accomplish the task in their own way.

 

 

 

 

v. Create follow up "Interview Questions" for your readers:

• Question your readers about usefulness/ease of use (especially if performance is an option) 
• See your PLANNING GUIDE, the sample tests above for ideas, and Guideline 2 (p. 385) for ideas 
• Question your readers about understanding, mainly by testing whether they understood key steps
• Question your readers about their attitudes towards: 

1. the subject matter (p. 386)
2. the quality of your communication (p. 387) (NOTE: YOU WANT TO REMIND THEM THAT THEY WILL NOT 'HURT YOUR FEELINGS")

• the persuasiveness of your communication
• Use QUANTITATIVE METRICS, such as the Likert-scale for some questions 

 

 

 

 

vi. Create a section called "Ending the test":

• Write a script to remind you to:
• say thank you,
• ask them if they have any questions,
• ask them if they have additional comments of suggestions
• reply to previously unanswered questions,
• give payment or a gift if appropriate.

 

---

 

Design your Data Collection Artifact:   how you will take notes

 

Note!!!

Facilitation style: bite your lip

• You have an agenda, certainly: but you are only an observer. Watch the user do what comes naturally. Don't help.
• The whole point of the test is to see what users do alone, without you helping.
• Much of the time you'll just be probing, and encouraging the user to say what they're doing and thinking.
• Use everyday language, not in-house jargon, unless you are testing an intranet.
• Keep calm: you want the user to find faults! Don't take it personally.
• Suspend judgement: don't even think about solving the problems at this stage.
• Keep encouraging the user to think aloud.
• Repeat: don't help! Don't think for the user. Explain that you can't answer questions during the test, because it's a test of how people use the site without you. But you'll answer them later if you can.
• Enjoy the job.
• Write everything down. If a task is part of the test, note the time it takes each user.
• Don't interfere or ask leading questions.
• Check your original task list as you go.

 

 

Now (before next class) Create a basic "Observation Sheet for User Tests" as the basic building block for your "Data Collection Artifact".

(the required basic format is on Anderson p.383)

 

Things to consider adding to your data collection artifact:

• you might add new sections or rows to the form that line up with your script,such as a row for tasks or scenarios
• you might add your likert-scale questions to your data-collection.
• you might include a blank paper or half sheet for rough notes or a mindmap
• you might include printouts of key pages
• you might include thumbnails of key images or graphics

 

Things to consider for your note-taking process:

•  To save time you can use abbreviations when noting occasions when the user:

1. hesitates,
2. looks worried,
3. misunderstands,
4. looks frustrated
5. or gives up.

---

Next class: