Abstract
Abstract
We explore a voice assistant that helps older adults locate UI features by verbal query. Through three studies — a think-aloud study with 19 older adults, a baseline with 12 younger adults, and a Wizard-of-Oz experiment (n=15) — we contribute a taxonomy of 5 query types and show that 87% of older adults used the voice assistant when lost, recovering immediately 77% of the time.
Keywords: voice assistant, older adults, mobile interface, Wizard-of-Oz, query taxonomy, accessibility, CHI 2023
1. Query Taxonomy (Studies 1 & 2)
How Older Adults Verbalize Interaction Problems
In an exploratory think-aloud study (19 older adults, 16 unique apps), we asked participants to speak their thoughts aloud whenever the interface caused confusion. 111 questions were collected. Using inductive open coding, we identified five query types:
| Type | Description | Example | Wh? |
|---|---|---|---|
| Validation | Verify whether doing the right thing | "Should I click the 'View Cart' option?" | Y/N |
| Informational–Directed | Clear next action, unclear which feature | "How do I find an address on this map?" | how |
| Informational–Undirected | No clear intuition about next feature | "Which one to choose?" | what/which |
| Navigational | Where is a particular feature? | "Where is history?" | where |
| Conceptual | About general app workflow, not UI location | "Why is it just processing?" | why |
Compared to younger adults (n=12), older adults asked significantly more validation questions (U=46, p=0.004, r=0.6) and more directed informational questions (U=53, p=0.007, r=0.53). The most frequent question types in order: validation (37.8%), directed informational (22.5%), navigational (18%).
Design Insight
"Where" and "how" questions — the most common UI-related types — map directly onto a voice query input: users already phrase their problem as a question the assistant can answer.
2. Design Probe
Just-in-Time, Just-in-Place Voice Assistant
The proposed system works as follows: at any point during app exploration, the user invokes the voice assistant. It processes the speech query using a Transformer-based keyword extractor (APE model, 85% accuracy), then reads the current screen's UI structure via Android's Assist API, extracts keywords using TF-IDF, and highlights up to 3 matching UI elements using ConceptNet similarity (F1 = .8).
Key design constraints: limited to the current screen only (no automatic navigation to other pages); highlights up to 3 features as a balance between guidance and visual clutter; operates at the platform level, not as an in-app feature.
Wizard's Interface
The wizard used a custom GUI to add highlights and arrows in real-time via drag-and-drop. A pre-generated question list per task allowed rapid responses without perceptible delays.
Audio tones confirmed query receipt and highlight rendering. Highlights disappeared as soon as participants resumed interaction.
3. Wizard-of-Oz Study
5 Tasks · 2 Websites · 15 Older Adults
Study tasks used two feature-rich mobile websites: choosechicago.com (tourism) and transitchicago.com (public transit). Conducted remotely via video call (COVID-19).
Interactions were coded using 9 event types including: unique/non-unique/successful/off-task selection, revert, cycle, retry, slip, and questioning. Participants were marked "getting lost" when making consecutive non-unique or off-task selections.
4. Results
88% Used the Assistant · 77% Recovered Immediately
Bonus Finding
Participants indirectly learned from VA outputs. Features highlighted in one task were reused in subsequent tasks — P11: "VA made me realize that the search function is quite useful." This was not a design goal but emerged organically.
VA use was significantly more frequent after getting lost than when not lost (Z=2.3, p=0.01, r=0.71). Among those who used the VA, it was used significantly more than self-exploration when recovering (Z=2.4, p=0.009, r=0.86).