A high-performance C++ firmware, lightweight Go command proxy, and Svelte 5 control interface combined into a single real-time voice-assistant media player ecosystem.
Nexus is split into three decoupled components that communicate seamlessly using WebSockets, JSON-RPC, and MQTT.
Ultra-low latency, real-time edge processing
Runs on the Waveshare Audio Development Board. It processes audio input and output locally, maintains a secure WebSocket session directly to Google's Gemini Live API, coordinates hardware states, and responds to local physical buttons.
// Lock-free state modification void updateVolume(int new_vol) { EmbeddedSysDb::getInstance().mutate([new_vol](SystemState& s) { s.audio.speaker_volume = new_vol; }); // Tasks registered to COMP::AUDIO receive // synchronous FreeRTOS task notifications }
Robust media coordination & stream resolution
Acts as the central coordination agent. It runs on a local machine or home server, receiving MQTT messages from the ESP32 or WebSocket packages from the web dashboard. It dynamically fetches media links from YouTube via `yt-dlp`, plays them via `mpv`, tracks local play queues, and stores offline tracks in an SQLite database.
// Intercept voice events to duck music func (q *Queue) HandleAssistantState(state string) { switch state { case "pause": q.wasPlaying = q.mpv.IsPlaying() q.mpv.Pause() case "play": if q.wasPlaying { q.mpv.Resume() } } }
Beautiful, reactive user dashboard
A fast, reactive web interface built with Svelte 5 and Tailwind CSS v4. It monitors the MPV server state in real time, displays playback information, hosts a debug WebSocket message console, and provides UI controls to remotely update the ESP32 configuration.
// Svelte 5 reactive stores using Runes export class MpvStore { playbackState = $state({ title: "", volume: 100 }); update(data) { this.playbackState.title = data.title; this.playbackState.volume = data.volume; } }
See how the entire system reacts when you say a voice command, streaming audio and instructions seamlessly in milliseconds.
WakeNet wakes up board. AudioService captures mic stream and pumps it to the WebSocket.
Decodes voice stream, matches tool calling schemas, and returns structured function call JSON.
Receives tool call over MQTT/CMD, queries yt-dlp, and loads resolved stream URL to mpv socket.
Go server publishes play state over WebSockets, updating the user dashboard instantly.
Advanced engineering implementations resolving audio streaming, synchronization, and configuration problems across the projects.
The ESP32 firmware dynamically handles linear upsampling (24kHz assistant to 32kHz native DAC) and downsampling (32kHz mic to 16kHz VAD/WakeNet), using a leaky bucket algorithm to prevent buffer starvation.
The Go Backend instructs `mpv` to log played network stream packets directly to disk (using `stream-record`). This caches music transparently as you listen, ensuring no bandwidth is wasted on repeat tracks.
Enables absolute administrative control. Svelte clients edit and write the ESP32 SD card's settings via custom JSON WebSocket commands that the Go backend translates to targeted MQTT packets.