Overview - Naxodev/mar

naxodev/mar

7 days30 days

Latest CI Pipeline Executions

Fuzzy

Failed
refactoring/client/voice
Fix ready→
49aefd70 nx suggestion
3 days ago by Jacarte
Failed
refactoring/client/voice
b125d31b fix
3 days ago by Jacarte
Failed
refactoring/client/voice
Fix applied→
01a32b10 linting
3 days ago by Jacarte
Failed
refactoring/client/voice
d3a7ea28 no more Set<Function>
3 days ago by Jacarte
Failed
refactoring/client/voice
Fix ready→
18e4c209 private methods solution
3 days ago by Jacarte
Failed
refactoring/client/voice
Fix ready→
a64b11e4 concurrency fixes
3 days ago by Jacarte
Failed
refactoring/client/voice
Fix ready→
bad8b003 fine tunning
3 days ago by Jacarte
Failed
refactoring/client/voice
Fix ready→
911e42f7 merging
5 days ago by Jacarte
Succeeded
main
d9b89d27 fix: forward client VAD events to voice-engine and use them to finalize transcripts (#15) Description: --- ## Summary - Forward client-side VAD (speech_start/speech_end) events into the voice-engine via a new handleVADEvent method, so the engine is aware of speech boundaries detected in the browser - Use VAD speech_end to mark the next interim transcript as final, with a 200ms debounce fallback if no new interim arrives — prevents Deepgram from hanging on open-ended streams - Fix STT-provider VAD events (from Deepgram) that were logged but never propagated to listeners/callbacks ## Changes engine/src/lib/voice/types.ts - Added handleVADEvent(event: VADEvent) to the VoiceEngine interface engine/src/lib/voice/voice-engine.ts - Implemented handleVADEvent: propagates to listeners/callbacks, and on speech_end sets a flag so the next interim is promoted to final (or debounces at 200ms to finalize the last known transcript) - Fixed handleSTTEvent vad case: now propagates to vadListeners and callbacks.onVADEvent instead of just logging - Added hasEmittedFinal guard to prevent duplicate final transcripts between the VAD path and stopListening() mar/maelstrom/server/api/src/lib/maelstrom-ws.ts - Added voiceEngine.handleVADEvent() calls in both vad.speech_start and vad.speech_end handlers, alongside the existing inline barge-in + start/stop listening logic (preserved for recording and Deepgram stream management)
5 days ago by Jacarte
Failed
refactoring/client/voice
Fix ready→
77d138e1 linting
7 days ago by Jacarte
Failed
refactoring/client/voice
Fix ready→
0acff7d6 adding tests for audio capture package
7 days ago by Jacarte
Failed
refactoring/client/voice
Fix ready→
52f1e1e4 adding tests for audio playback
8 days ago by Jacarte
Succeeded
fix/deepgram
a84245c2 final transcript also depends on the vad event
8 days ago by Jacarte
Succeeded
feat/barge-in
f0a739cb update tests
10 days ago by Jacarte
Failed
feat/barge-in
Fix ready→
e01e9b7e fix the hanging
10 days ago by Jacarte
Succeeded
feat/barge-in
cd1d30b7 faster vad loading
11 days ago by Jacarte
Succeeded
feat/barge-in
4dcb23ef interrupting sending tts audio if VAD event is received from client linting
11 days ago by Jacarte
Succeeded
main
1260d4dd feat: Voice Activity Detection (VAD) Integration & Audio Synchronization (#12) ## Summary This PR implements client-side Voice Activity Detection (VAD) with server synchronization and adds audio packet delay checking for real-time voice interactions. The changes move VAD event handling from the server (Deepgram) to the client (Silero), enabling faster speech detection and better control over the voice conversation flow. ## Key Changes ### 1. **Client-Side VAD Event Handling** (`use-maelstrom-voice.ts`) - **Added VAD callbacks**: `onSpeechStart` and `onSpeechEnd` handlers that send events to the server - **Pre-buffered audio**: When speech is detected, the client now sends audio that was captured *before* VAD detection (via ring buffer), reducing latency - **Speech state tracking**: Introduced `userIsTalkingRef` to track whether the user is currently speaking - **Audio chunk filtering**: Only send audio chunks to the server when `userIsTalkingRef` is true (during active speech) ### 2. **Audio Packet Delay Checking** (`use-maelstrom-voice.ts`) - **Timestamp tracking**: Server-sent TTS audio packets are now timestamped when received - **Delay threshold**: Packets older than 200ms are discarded to maintain real-time playback sync - **Queue management**: Replaced simple audio queue with timestamped queue (`receivedAudioWithTimestampsRef`) - **Logging**: Added debug logs showing packet age and queue status ### 3. **VAD Ring Buffer** (`vad-processor.ts`) - **Ring buffer implementation**: Maintains a sliding window of audio samples matching the speech threshold duration - **Pre-buffered audio export**: When speech is detected, the ring buffer is converted to PCM16 format and sent to the client - **Automatic cleanup**: Ring buffer is cleared after speech start event is emitted ### 4. **Protocol Updates** (`voice-messages.ts`) - **Removed**: `SpeechStartedMessage` and `SpeechStoppedMessage` (server-side TTS notifications) - **Added**: `VADSpeechStartMessage` and `VADSpeechEndMessage` (client-side VAD events) - **Added**: `cancellable` flag to `AudioOutputMessage` for future barge-in support - **Updated union types**: Reflect new message types in `VoiceMessage` and `VoiceServerMessage` ### 5. **Server-Side VAD Event Handling** (`maelstrom-ws.ts`) - **New handlers**: Process `vad.speech_start` and `vad.speech_end` messages from client - **Listening control**: Calls `voiceEngine.startListening()` on speech start and `stopListening()` on speech end - **Error handling**: Graceful error logging for listening state transitions ### 6. **Deepgram STT Cleanup** (`deepgram-stt.ts`) - **Removed VAD configuration**: Deepgram VAD events are no longer used - **Removed VAD event handlers**: Speech start/utterance end event handling removed - **Simplified connect**: Removed VAD-related URL parameters ### 7. **Voice Engine Simplification** (`voice-engine.ts`) - **Removed VAD event handling**: No longer processes VAD events from STT provider - **Removed VAD-based transcript finalization**: Utterance end detection moved to client - **Added debug logging**: Better visibility into audio chunk flow and STT connection state ## Benefits ✅ **Lower latency**: Pre-buffered audio reduces delay between speech detection and server processing ✅ **Better UX**: Client-side VAD provides immediate visual feedback (orb glow) ✅ **Real-time sync**: Packet delay checking prevents stale audio playback ✅ **Cleaner architecture**: VAD responsibility moved to client where it belongs ✅ **Future-ready**: `cancellable` flag enables barge-in (interrupt TTS) support ## Tests to be added
12 days ago by Jacarte
Succeeded
feat/ws
ede5e145 fixing deepgram fix tests linting fix test fix tests fix types fix tests fix linting cleaning rollback removal rollback
12 days ago by Jacarte
Succeeded
feat/ws
8a88a928 rollback
12 days ago by Jacarte