Reclaiming Identity: Know Your Data
UX researcher
Overview
|
I worked on a team to develop and implement a usability study of the Alaska Airlines website booking process with screen reader users who identified as blind.
We tested six participants in two hour sessions in their homes, using their own devices. We designed a pre-test questionnaire, six to ten discrete booking tasks, and a post-test questionnaire. We used a modified think aloud protocol, observation, video and screen recording to collect data. We passed along 21 findings and corresponding recommendations to Alaska Airlines. We gave a presentation to employees of many Alaska Airlines teams and were asked back to speak to a larger group in the spring. |
Company: Alaska Airlines
Title: UX Researcher Type: Class Project Course: HCDE 517 Usability Studies Date: Jan 2018 - Mar 2018 Team: Nicole Alvarez, Anthony Ugas, Amy Chen, and Anting Chen Roles:
|
Purpose of the Study
The purpose of the study was to evaluate the accessibility of booking a flight on the Alaska Airlines website for users who are blind or have low vision and use screen readers to navigate desktop websites. Additionally, Alaska Airlines wished to go above and beyond the required criteria to meet web accessibility guidelines. They expressed that they wanted to make their website as easy to use as possible and make the experience of booking a ticket with Alaska Airlines delightful for people who are blind or have low vision. We conducted the study to find usability issues or pain points for which improvements could be made to achieve this goal.
Study Design
Overview
We tested the booking process from the homepage to seat selection with all participants. The tests were conducted using a modified think aloud protocol and several questionnaires in participants’ homes with their preferred devices. The tests took between one and two hours.
We tested the booking process from the homepage to seat selection with all participants. The tests were conducted using a modified think aloud protocol and several questionnaires in participants’ homes with their preferred devices. The tests took between one and two hours.
Recruitment
We recruited participants through a pilot study participant who had many friends and contacts in the local blind community and through our contact at Alaska Airlines, Liz Aderhold. We screened the recruits to find both JAWS and VoiceOver users, people who had booked their own travel before, and both people who had and people who had not previously used the Alaska Airlines website. We began with 11 participants from recruitment. After a screening questionnaire, we eliminated two participants. Of the nine remaining participants, three could not find time to participate.
We recruited participants through a pilot study participant who had many friends and contacts in the local blind community and through our contact at Alaska Airlines, Liz Aderhold. We screened the recruits to find both JAWS and VoiceOver users, people who had booked their own travel before, and both people who had and people who had not previously used the Alaska Airlines website. We began with 11 participants from recruitment. After a screening questionnaire, we eliminated two participants. Of the nine remaining participants, three could not find time to participate.
Participants
Test Environments and Equipment
Four tests were conducted in the participants’ homes. One test was conducted on campus, in a private room at the University of Washington, and one was conducted in a private room at Lighthouse for the Blind. All participants used their own preferred devices with their preferred accessibility tools. We made these environment and equipment decisions to elicit natural behaviors from participants.
Four tests were conducted in the participants’ homes. One test was conducted on campus, in a private room at the University of Washington, and one was conducted in a private room at Lighthouse for the Blind. All participants used their own preferred devices with their preferred accessibility tools. We made these environment and equipment decisions to elicit natural behaviors from participants.
Methods
We asked participants to imagine they were purchasing a flight from Seattle to Los Angeles in late March and to complete their flight booking process on the Alaska Airlines website. We asked participants to complete tasks while describing what they were hearing, expecting, and thinking as much as possible. After each task we had participants pause and answer a series of questions. After the test concluded, we interviewed participants about their experience and asked them questions about their demographic information. We screen recorded the sessions using Zoom and video recorded the participants’ facial expressions and gestures.
We asked participants to imagine they were purchasing a flight from Seattle to Los Angeles in late March and to complete their flight booking process on the Alaska Airlines website. We asked participants to complete tasks while describing what they were hearing, expecting, and thinking as much as possible. After each task we had participants pause and answer a series of questions. After the test concluded, we interviewed participants about their experience and asked them questions about their demographic information. We screen recorded the sessions using Zoom and video recorded the participants’ facial expressions and gestures.
Modified Think Aloud
Because the purpose of the study was to understand the experience of the participants, it was important to us to have participants express their candid thoughts aloud as they navigated the website. Traditional think aloud protocol would not work for us because participants need to listen to their screen reader to navigate the page and could not reasonably be expected to listen and think aloud at the same time. When not in the usability study, our participants typically listened to their screen reader using headphones and a very high speaking speed. We asked them to play their screen reader audio through the speakers so that we could avoid interrupting them. We also asked participants to retain their preferred speaking speed but set up visual text display of the screen reader’s output so we could pick it up with screen recording.
Because the purpose of the study was to understand the experience of the participants, it was important to us to have participants express their candid thoughts aloud as they navigated the website. Traditional think aloud protocol would not work for us because participants need to listen to their screen reader to navigate the page and could not reasonably be expected to listen and think aloud at the same time. When not in the usability study, our participants typically listened to their screen reader using headphones and a very high speaking speed. We asked them to play their screen reader audio through the speakers so that we could avoid interrupting them. We also asked participants to retain their preferred speaking speed but set up visual text display of the screen reader’s output so we could pick it up with screen recording.
Tasks
Findings and Recommendations
|
Data Analysis
After all of the sessions were complete, we gathered the data from our notes and created an affinity diagram to locate common occurrences and themes. We decided upon a severity rating scale and analyzed each finding in order to give it a rating. Because Alaska had approached us with the intention of improving the experience of using their website, our main focus was on qualitative data and participant feedback. We collected additional quantitative data, such as errors on task and task failures, but we focused on our qualitative data analysis to make our findings and recommendations. |
|
Severity Ratings
Each of the following findings include a severity rating intended to rank the finding in terms of its impact on the experience of booking a flight on Alaska Airlines’ website for a person who is blind or has low vision. Ratings are based on:
Each of the following findings include a severity rating intended to rank the finding in terms of its impact on the experience of booking a flight on Alaska Airlines’ website for a person who is blind or has low vision. Ratings are based on:
- Frequency - How many participants experienced the issue? How many times?
- Impact - How difficult was it for participants to overcome the issue? Did the issue cause them to fail the task?
- Persistence - How avoidable was the issue? Could the participants take an alternate route to avoid the issue?
- Experience - How did the issue affect the participant? Did it cause frustration, anger, confusion?
Summary
We presented 21 findings to Alaska Airlines based on our study. The vast majority of findings were not severe. Due to a non-disclosure agreement, I cannot share information about findings.
We presented 21 findings to Alaska Airlines based on our study. The vast majority of findings were not severe. Due to a non-disclosure agreement, I cannot share information about findings.
Presentation and Report
We gave an hour long presentation to Alaska Airlines employees from many different departments. After our presentation, we received much positive feedback and were asked to give the presentation again to a wider audience in Spring 2018.
The following presentation and report have been edited to remove information related to study results.
The following presentation and report have been edited to remove information related to study results.
|
|
|
