Skip to content
All posts
·2 min read

Building AgentX: An AI Life Copilot on Azure

Architecture and tradeoffs behind a context-aware copilot that helps neurodivergent users plan their day.

aiazureproduct

Why I Wanted a Life Copilot

Most assistants answer questions. My neurodivergent friends needed something that tracked focus, moods, and actionable plans. AgentX pairs persistent context with tailored nudges so users feel supported, not lectured.

System Architecture Overview

Next.js App (web + mobile shell)
   |
   v
Gateway API (Fastify) --> Orchestrator (FastAPI)
   |                         |--> Azure OpenAI (GPT-4 Turbo)
   |                         |--> Azure Cognitive Services (speech + sentiment)
   |                         |--> Task & Calendar Service (Cosmos DB + Functions)
   |                         |--> Notification queue (Azure Service Bus)
Analytics pipeline (Event Hubs -> Data Explorer)
  • Frontend: React Native Web + Next.js for shared components, offline-first caching, and push notifications.
  • Backend: Fastify handles auth/token refresh; FastAPI orchestrates tool calls and conversational state.
  • Memory: User state lives in Cosmos DB with TTL backed snapshots for recovery.
  • Queues: Service Bus manages scheduled nudges and retry logic.
  • Analytics: Event Hubs captures interactions, with dashboards in Data Explorer for product experiments.

Prompting and Tooling Strategy

System prompts encode AgentX's persona: empathetic, actionable, never intrusive. Tools are invoked via a planner that matches intents to capabilities: task creation, focus sessions, mood tracking, and calendar sync. Memory is summarized after each session to maintain tone without ballooning tokens.

Streaming UX

We stream responses via Server-Sent Events so users see the copilot "thinking" within 200 milliseconds. The orchestrator pushes interim state updates (e.g., "scheduling focus block") so the UI animates context switchers in real time.

Cost and Safety Considerations

  • Cached embeddings for frequently referenced goals.
  • Per-user token budgets with redlines in dashboards.
  • Input filters catch crisis language and escalate to human support resources.
  • Rate limiting prevents runaway tool loops.

What Worked and What Did Not

Users loved the daily focus storyboard and actionable mood prompts. They got confused when AgentX suggested tasks without checking calendars first. Next steps include:

  • Fine-tuning on consented transcripts for better summarization.
  • Expanding the toolset with habit loops.
  • Experimenting with multi-agent planning for complex projects.

Building AgentX forced me to balance latency, empathy, and governance. The architecture keeps context grounded while staying affordable and safe.