Nav Nudge: On-Demand Feature Search for Any App

Abstract

Keywords: Nav Nudge, voice assistant, LLM, Android, feature search space, older adults, mobile navigation, CHI 2024

1. Design & Pipeline

From Verbal Query to Visual Highlight in Three Steps

Nav Nudge's pipeline processes a spoken query and the current app screen to identify and visually emphasize the 3 most relevant UI features:

• Step 1 — Keyword extraction from query Voice input transcribed; OpenAI text-davinci-002 extracts keywords from the spoken phrase (94% accuracy on in-house dataset of 240 queries).
• Step 2 — UI feature extraction Android's Assist API retrieves the view hierarchy of the current screen — labels, descriptions, and coordinates of all UI elements, including partially visible ones. APE model extracts keywords (93.4% accuracy).
• Step 3 — Semantic matching Universal Sentence Encoder v4 computes similarity between query keywords and UI element keywords. Elements scoring above .3 are selected; top 3 by score are visually emphasized using Highlight with Context or Weighted Zoom.

A dedicated REST API server handles Steps 1 and 3 to offload computation from the mobile device. The result — coordinate-tagged UI elements — is returned to the app and rendered in real time.

2. Implementation

Android 13 · Platform-Level · Works Across Any App

Nav Nudge is implemented as a platform-level Android 13 service. It requires Assist API permissions but works across any installed app without modification — users invoke it the same way they would invoke Google Assistant or Siri.

The visual output uses the two best-performing cue types from the controlled study: Highlight with Context (HC) and Weighted Zoom (WZ). Users can configure which they prefer. Content labels from AssistStructure are shown alongside spatial emphasis.

3. Evaluation Study

10 Older Adults · Organic Maps · Face-to-Face

Nav Nudge was evaluated using Organic Maps (open-source, no prior experience for participants). 4 tasks covered lookup, hidden feature discovery, location sharing, and bookmark management — designed to require finding non-obvious features.

4. Results

8/10 Used It · 15/15 Recoveries After Use

Figure 8. Three examples of Nav Nudge in use: (a) P7 "I want to add a starting point" — route/bookmark icons highlighted; (b) P2 "How can I see the bookmark list?" — bookmarks panel shown; (c) P1 "Send the current location" — position + menu emphasized. (CHI 2024)

• 8/10 used Nav Nudge at least once. 2 participants completed all tasks without invoking it.
• 19 total invocations — 18/19 contained the correct feature in the reduced set (1 failure due to voice recognition timeout, immediately resolved on retry).
• Getting lost: 20 events — Nav Nudge was used in 15 of 20. In all 15 cases, the participant recovered immediately.
• Unexpected learning: When multiple features were highlighted, participants explored all of them and often correctly chose the second or third option without getting confused.