Inspiration

We all have seen how wearable tech like the Apple Watch or the FitBit pushed the boundaries of personal computing. Knowing how fast technology is moving, we wanted to look towards the near future. We realized that AR glasses offer a unique opportunity to tackle challenges directly from the perspective of the user. We focused heavily on accessibility challenges and health-centric tasks.

What it does

Cleo is an always-on, AI-powered assistant that lives in a pair of AR glasses, designed with accessibility and memory retention challenges in mind. We built a real-time narrator for the visually impaired, a color blind correction tool, object tracking for remembering where you placed things, facial recognition for users with Alzheimer's, plus note-taking, food tracking, and weather integration. All skills are voice-activated and designed to be responsive.

How we built it

We designed Cleo as a distributed AR runtime: a process orchestrator boots a graph of Python microservices in dependency order (sensor, data, assistant, frontend, transcription, video, tools), health-checks each gRPC endpoint, and handles graceful shutdown. At the contract layer, the system is defined with Protocol Buffers + gRPC across all services. Media is transported with explicit chunking semantics and streaming RPCs.

For hardware, we built a Rust + PyO3 bridge to expose VITURE device primitives to Python. The speech pipeline uses Amazon Transcribe Streaming with a custom TriggerRouter for wake phrase detection. The assistant uses Amazon Bedrock (Claude Sonnet) with dynamic function-calling style routing via a ToolRegistry. Memory uses SQLite + FAISS + object storage for multimodal semantic retrieval. The UI is a React + TypeScript HUD inside Tauri with a Rust gRPC client.

My Role

I worked to create the custom AI orchestrator with function tool calling, setting up gRPC/protocol buffer architecture, and building many of the tools.