Designing a Voice Assistant for Mobile Screen Navigation

keywords: qualitative study, Wizard of Oz, web application, accessibility, aging and technology

Related publication: ASSETS 2020 Extended Abstract [pdf], CHI 2023 [pdf]

The Problem

The previous study revealed that older adults can articulate what they were looking for either by specifying the feature's name or describing it. Additionally, our team found that when we ask older users to ask a question when they got stuck, they use question words like 'where' and 'how', which led us to envision a voice assistant that helps users navigate a mobile interface.

With two design requirements, we began designing a voice assistant.

  1. Allow users to ask interface feature-related questions verbally.
  2. Offer visual cues on the mobile interface to narrow down the search space.



Approach & Method

Prototyping

We gathered additional requirements through interviews and brief testing sessions with older users, iterating on the design based on their feedback.

  • Operate like Google Assistant and Siri, activated with a simple press of the home button, ensuring user ease without intrusion.
  • Operate at the platform level rather than as an in-app feature of specific applications. While it requires device permissions, this approach allows it to be usable across any app.
  • Identify up to three UI components similar to the user's query and highlight them on the screen.
  • Limit UI-feature search to the current page to prevent user confusion caused by automatic page navigation.





Clickable Prototype





Wizard-of-Oz Experiment

Before getting the design right, we wanted to investigate if our solution addresses the users’ requirement of quickly locating a UI feature in mobile interfaces.

To that end, we decided to conduct a Wizard-of-Oz study to examine how older adults would use our design solution on two websites.

We wanted to examine if users find the design intervention useful.
We investigated the following three research questions:
  1. Do users use the voice assistant?
  2. When do users use the voice assistant?
  3. How does the voice assistant intervention influence subsequent user interactions?


Tasks
  1. You are visiting Chicago in March. Find an event that you would like to go to.
  2. You want to go to a museum with your kids. Find a kid-friendly museum to go to.
  3. Check the Purple Line's Saturday operating hours (not Purple Line Express).
  4. You are at the Davis Station. You are waiting to take the Purple line heading toward Linden. Please find out when the next train is due.
  5. Find out how much it costs to take the 'L' train from O'Hare airport. Is there a senior discount?



Wiz-of-Oz App Implementation

  • We simulated a local version of these two websites which could be modified in real-time by the wizard using the wizard’s interface.
  • Participants can access the experiment app via any web browser.
  • The Facilitator can watch how participants interact with a website in real time.
  • When a participant asks a question, the facilitator can listen to their voice
  • The Facilitator could quickly add visual cues and then make them appear on the participant's screen.
Implementation
  • Web interface: HTML, JavaScript, jQuery
  • Server: javaScript
  • Hosting: Amazon AWS


User Study & Data analysis

Participants
  • 15 participants.
  • 55 years old or above.
  • Own a smartphone and/or tablet.
  • Their MDPQ-14 (Mobile Device Proficiency Questionnaire) scores were not significantly different from scores reported in other literature.


Settings
  • Given the COVID-19 pandemic situation, the experiment was conducted remotely via a video-calling application (e.g., WhatsApp, Skype, or Zoom).
  • Participants used their own mobile devices.
  • No time limit was imposed.
  • No one app version or OS was prescribed.

Data Analysis
  • Atlas.ti was used for video coding.
  • Our video corpus was tagged and all interactions were classified using an existing codebook.


Result Highlight

88% of participants used the voice assistant.




  • 13 out of 15 participants asked at least one question, and 33 questions were recorded.
  • A total of 41 getting lost were identified. 63% of getting lost were followed by the question (26 out of 41).


The voice assistant helped them recover and complete the task.




  • 77% of getting lost with a question was recovered immediately after the intervention (20 out of 26).
  • The voice assistant was used significantly more frequently than self-exploration when recovering after getting lost.


Resources