Background
I joined Adept AI Labs as the Lead Product Designer and worked on designing how humans can best utilize computer use agents. Two months later the team got acquired* by Amazon.
At Amazon, my team of two was initially tasked with aligning the Alexa org with the Adept team. Then we decided to design and prototype a solution for what we think is the right interaction model for Adept's workflows.
Breaking away from turn-by-turn chat
The challenge
It's cumbersome to have a back-and-forth conversation with a voice agent to get real-life tasks done. For most of the time it's faster to do it yourself.
How do we make a case for a voice interaction to be useful?
Inventing a new way to use agents
The first prototype
My small team built a working protoype of a UI component embedded into the chat interface. The user can talk naturally, and the interface instantly updates as you speak.
The prototype asks you follow-up questions, but as a subtle visual nudge instead of interrupting the user—taking into account that visual recognition only takes milliseconds.
I named this pattern "Live Notes."
Building groceries cart
The interface evolved into a full-screen experience outside of the chat view.
This is me talking naturally to my groceries cart to add items.
Ordering food
A new component for when the user asks questions about the menu
Alexa+ Launch Demo
A few months later—our prototype, built by three people including myself, got on the stage for the Alexa+ launch demo.