A high-performance C++ firmware, lightweight Go MCP service & command proxy, and reactive web control studio combined into a single real-time voice-assistant media player ecosystem.
Nexus is split into decoupled components that communicate seamlessly using WebSockets, JSON-RPC, and Model Context Protocol (MCP).
Ultra-low latency, real-time edge processing
Runs on the Waveshare Audio Development Board. It processes audio input and output locally, maintains a secure WebSocket session directly to Google's Gemini Live API, coordinates hardware states, and responds to local physical buttons.
// Lock-free state modification void updateVolume(int new_vol) { EmbeddedSysDb::getInstance().mutate([new_vol](SystemState& s) { s.audio.speaker_volume = new_vol; }); // Tasks registered to COMP::AUDIO receive // synchronous FreeRTOS task notifications }
Robust media coordination & Model Context Protocol skills
Acts as the central cloud capability provider and media agent. It hosts a high-performance Model Context Protocol (MCP) server over SSE providing external tools (news headlines, web search, weather forecasts) directly to the assistant without needing firmware updates. Also handles stream resolution and audio playback queues.
// Register remote MCP tools over SSE s.RegisterTool("news_headlines", handleNews) s.RegisterTool("web_search", handleSearch) s.RegisterTool("get_weather", handleWeather) // ESP32 queries tools dynamically without OTA
High-performance reactive control dashboard
A lightweight, single-bundle control center served directly from ESP32 flash memory. Provides live telemetry over WebSockets, dynamic Gemini Live model & VAD parameter selection, a 4-source alarm manager with YouTube and Opus upload, reminder & action scheduling, and dual-slot OTA firmware management.
// 4-source tone picker & morning briefing const saved = await saveAlarm({ tone: "builtin:chime", briefing: true, snooze_min: 9, kind: "alarm" });
See how the entire system reacts when you speak a voice command, streaming audio, instructions, and remote skills seamlessly in milliseconds.
WakeNet wakes the board. AudioService captures mic stream, runs local VAD, and streams PCM to the WebSocket.
Decodes voice stream with chosen model, applies search grounding, and returns structured function call JSON.
Core hardware skills (15 tools) execute locally on ESP32; remote tools (news, search, weather) query Go MCP over SSE.
Audio plays, actions execute via the scheduler, and responses stream back to the speaker in real time.
Target routing hands off playback to PC/desktop satellites with decentralized stream resolution and auto-failover.
Advanced engineering implementations resolving audio streaming, synchronization, and intelligent edge-cloud capabilities.
Direct bidirectional voice streaming with runtime Gemini Live model selection, Google Search grounding where supported, adaptive voice-activity detection (VAD), and conversational barge-in sensitivity.
Powered by a dedicated Go Model Context Protocol (MCP) server over Server-Sent Events (SSE). Seamlessly extends assistant intelligence with real-time news headlines, web search, and weather forecasts without requiring firmware updates.
An autonomous cron-like task scheduler that handles both natural language spoken reminders and scheduled time-based device actions ("play lofi at 7", automated routine triggers from notes).
Wake-up tones selectable from synthesized hardware patterns, custom uploaded Opus audio files, YouTube downloads, or the offline music library with classic fallback. Delivers a spoken morning briefing upon stopping.
Spoken note-taking stored persistently on SD card storage, protected with spoken verbal confirmations before any destructive deletion action.
A trimmed and deterministic embedded toolset executing direct hardware control—alarms, timers, volume, display brightness, WiFi diagnostics, and audio recording—with low-latency edge execution.
The ESP32 firmware dynamically handles linear upsampling (24kHz assistant to 32kHz native DAC) and downsampling (32kHz mic to 16kHz VAD/WakeNet), using a leaky bucket algorithm to prevent buffer starvation.
The media subsystem logs played network stream packets directly to disk (using stream-record). This caches music transparently as you listen, ensuring no bandwidth is wasted on repeat tracks.
Complete browser management studio served directly by the ESP32 featuring real-time WebSocket telemetry, tone manager, scheduler controls, and failsafe dual-slot flash OTA updates with instant rollback.
Extends the ESP32-S3 into a distributed multi-device audio constellation. While Nexus Core remains the brain, queue master, and voice interface, audio rendering seamlessly hands off to PC and workstation satellites with decentralized stream resolution, MPRIS media keys, and instant local failover.