Email Usability Test on Mobile Devices

(Usability Testing at University of Washington)
Team Members: Amol Surve, Erika Dillman, Jenn Chan, and Kanika Bhargava

email USER RESEARCH

TARGET USERS

FullĀ­time undergraduate students ages 18-25.

RESEARCH GOAL

  • Learn which mobile phone email apps students use the most, and what influences their decisions to select those apps.
  • Learn how students currently communicate and collaborate with their peers using their phones.
  • Learn how students use Outlook on their phones through the usability study.

ONLINE SURVEY

The online survey was completed prior to the usability study to provide us with insights into how students use emails, and other communication and collaboration software. It contained open and closed questions to provide qualitative and quantitative data. In order to collect data on the motivations behind mobile phone use, the survey contained questions about both mobile phone and laptop/desktop use.

email STUDY PLAN

We performed our usability test on the Microsoft Outlook mobile application.

WORKFLOW

We created a workflow diagram to map out functionalities of Outlook, which would help us design our usability tasks.
App Workflow App Workflow App Workflow

PARTICIPANT SCREENER

These are participant criteria for the usability study:
  • FullĀ­time undergraduate students.
  • Between the ages of 18 and 25
  • Smartphone owners (iOS or Android), who can bring their own phones to use in the study.
  • Available to participate on the day of the study.
  • Download Outlook and sign up before arriving to the study.

SCHEDULE

Each member of the team rotated roles between moderator and note taker. We had 8 target users, and ran sessions concurrently by splitting the team into two. This was done because of the limited time available in the usability lab.

TIME

Maximum 60 minutes per session:
  • ~10 minutes for introduction and signing consent forms
  • ~25 minutes for interview on how they communicate and collaborate with their peers
  • ~25 minutes for usability test and post-test survey.

PRINTOUTS FOR TEST DAY

  • Consent forms
  • NDA forms
  • Gratuity forms
  • Introductory talking points
  • Usability task list
  • Interview questions
  • Post-test surveys

DELIVERABLES

  1. Presentation of the findings
  2. In-depth report of the user research and usability testing sessions

email USABILITY TASKS

In addition to qualitative notes from feedback and users thinking out loud, we also marked each task as complete or incomplete, and asked for the users' confidence level from 1 to 5.

Moderator Action User Task Success Criteria
Send an email to <team member email>. Email is sent.
Send user an email. Read this email and save the attachment. Attachment is saved.
Set a reminder to read this email later. Flag or mark as unread or archive or create a new filter.
Find the email you set the reminder for. Find the email by filtering or searching.
Mark this email as junk. Email is in junk folder.
If you did that by mistake, can you reverse it? Email is back in inbox.
Remove this email. Delete email.
If you did that by mistake, can you reverse it? Email is back in inbox.
Send user 3 emails. Remove these emails. Select all and delete, or go into each email and delete.
Send user an email Schedule a meeting with the sender. Reply and create calendar event, or go to banner and click on calendar and invite from calendar, or reply and state the meeting time.