Overview - Naxodev/mar

naxodev/mar

7 days30 days

Latest CI Pipeline Executions

Fuzzy

Succeeded
main
ddbd47eb refactor: client/voice (audio playback TTS) (#16) ## PR Type What kind of change does this PR introduce?  ``` [ ] Bugfix [ ] Feature [ ] Code style update (formatting, local variables) [x] Refactoring (no functional changes, no api changes) [ ] Build related changes [ ] CI related changes [ ] Documentation content changes [ ] Other... Please describe: ``` ## What is the current behavior? The voice functionality is currently tightly coupled in the React layer with: - Audio capture logic embedded in `use-maelstrom-voice.ts` (using inline AudioWorklet code) - Audio playback handling mixed with state management - VAD logic partially in React hooks - Complex orchestration spread across multiple files - Limited test coverage for audio operations ## What is the new behavior? This PR refactors the voice architecture by extracting concerns into dedicated packages: 1. **`audio-capture`**: New standalone package with `AudioCapture` class handling microphone access, AudioWorklet management, and audio streaming 2. **`audio-playback`**: New standalone package with `AudioPlayback` class for TTS audio queue management and scheduling 3. **`voice-session`**: New package containing `VoiceSession` class that orchestrates STT/TTS flow, VAD integration, and state management 4. **Simplified React hooks**: `use-maelstrom-voice.ts` reduced from ~800 lines to ~200 lines by delegating to `VoiceSession` 5. **Improved VAD**: Better Silero VAD integration with configurable timeout options 6. **Enhanced testing**: Added comprehensive test suites for audio-capture (328 lines), audio-playback (194 lines), and voice-session (1238 lines) 7. **Better concurrency**: Fixed race conditions in audio playback and capture 8. **Code organization**: Moved all audio processing logic from React layer to client library API remains unchanged for consumers of the React hook. ## Does this PR introduce a breaking change? ``` [ ] Yes [x] No ```  ## Other information - Removed ~700 lines from `use-maelstrom-voice.ts` and ~777 lines from `voice-connector.ts` - Total net addition of ~1,300 lines (mostly new tests and documentation) - All existing voice functionality preserved - Improves maintainability by separating audio concerns from React UI logic - Enables reuse of voice logic in non-React contexts
18 hours ago by Javier Cab...
Succeeded
main
467d7eb7 fix: zero executor not found
5 days ago by NachoVazquez
Succeeded
main
d9b89d27 fix: forward client VAD events to voice-engine and use them to finalize transcripts (#15) Description: --- ## Summary - Forward client-side VAD (speech_start/speech_end) events into the voice-engine via a new handleVADEvent method, so the engine is aware of speech boundaries detected in the browser - Use VAD speech_end to mark the next interim transcript as final, with a 200ms debounce fallback if no new interim arrives — prevents Deepgram from hanging on open-ended streams - Fix STT-provider VAD events (from Deepgram) that were logged but never propagated to listeners/callbacks ## Changes engine/src/lib/voice/types.ts - Added handleVADEvent(event: VADEvent) to the VoiceEngine interface engine/src/lib/voice/voice-engine.ts - Implemented handleVADEvent: propagates to listeners/callbacks, and on speech_end sets a flag so the next interim is promoted to final (or debounces at 200ms to finalize the last known transcript) - Fixed handleSTTEvent vad case: now propagates to vadListeners and callbacks.onVADEvent instead of just logging - Added hasEmittedFinal guard to prevent duplicate final transcripts between the VAD path and stopListening() mar/maelstrom/server/api/src/lib/maelstrom-ws.ts - Added voiceEngine.handleVADEvent() calls in both vad.speech_start and vad.speech_end handlers, alongside the existing inline barge-in + start/stop listening logic (preserved for recording and Deepgram stream management)
5 days ago by Jacarte
Succeeded
main
a4b09c2a refactor: create message processor event loop in client v2 (#11) ## PR Type What kind of change does this PR introduce?  - [ ] Bugfix - [x] Feature - [ ] Code style update (formatting, local variables) - [ ] Refactoring (no functional changes, no api changes) - [ ] Build related changes - [ ] CI related changes - [ ] Documentation content changes - [ ] Other... Please describe: ## What is the current behavior?  **Client Architecture Issues:** - The orchestrator was a monolithic module with all logic in a single file - Legacy action execution infrastructure (runner-registry, uow) was difficult to maintain - Message processing lacked proper state management and type safety - No separation between voice detection and orchestration concerns **Voice Orb:** - Audio level visualization was based on FFT frequency data (`AnalyserNode.getByteFrequencyData()`) - Responded to any sound including TTS playback, background noise, and non-speech audio - No client-side Voice Activity Detection (VAD) Issue Number: N/A ## What is the new behavior? ### Client v2 Architecture **New Package: `packages/maelstrom/client-v2/`** - `MessageProcessor` - Modern message processing with event listeners and state management - `ActionManager` - Full action lifecycle management with waiting capabilities - `LayoutManager` - View/layout state management - Comprehensive test suite with 400+ tests **Orchestrator Refactor:** - Split monolithic orchestrator into modular components: - `engine-communication/` - Engine processor, lifecycle, update queue - `orchestrator/` - Main orchestration logic with API contract tests - `registry/` - Vessel registry management - `state-management/` - Store manager, event bus, typed store - `vad/` - Voice Activity Detection (Silero VAD integration) - `types/` - Comprehensive type definitions (split from single file) **Strict Type System:** - `ActionRunner<V>` with full parameter inference from vessel definitions - `defineVessel()` helper to preserve literal types - Support for partial action subsets per view - Type-safe action runners in React hooks ### Voice Activity Detection (VAD) **New VAD Module (`packages/maelstrom/react/src/lib/vad/`):** - Uses **Silero VAD v5 + ONNX Runtime Web (WASM)** - `silero-vad.ts` - Low-level ONNX Runtime wrapper - `vad-processor.ts` - Speech segment detection with configurable thresholds - `use-vad.ts` - React hook for VAD integration - Processes 512-sample windows (32ms @ 16kHz) **Integration:** - VAD probability drives voice orb visualization (0-1 scale) - Playback mode raises threshold to 0.9 during TTS for barge-in support - Browser Acoustic Echo Cancellation (AEC) prevents feedback **Assets:** - Silero VAD model: `apps/mar/public/models/silero_vad.v5.onnx` (2.3MB) - ONNX Runtime WASM loaded from CDN ## Does this PR introduce a breaking change? - [x] Yes - [ ] No  **Breaking Changes:** - `registerRunners()` changed to `registerRunner()` with object-based API - `ActionRunner` now requires Vessel type parameter for full inference - Old `message-queue.ts` removed (use `MessageProcessor` from client-v2) - Legacy action execution infrastructure removed **Migration:** - Update vessel definitions to use `defineVessel()` helper - Replace `registerRunners()` calls with `registerRunner()` - Update React components to use new typed `useVesselActions` hook ## Other information **Stats:** 111 files changed, +7107/-3835 lines (net +3272 lines) **Commits:** 22 commits including: - VAD implementation with Silero - Orchestrator refactor into subdomains - Client-v2 with MessageProcessor and ActionManager - Strict ActionRunner type inference - Full integration between client v1 and v2 **New Dependencies:** - `onnxruntime-web` - ONNX Runtime for browser-based VAD **Key Architectural Improvements:** - Modular client architecture with clear separation of concerns - Type-safe action system with full inference - Comprehensive test coverage (400+ tests in client-v2) - Client-side VAD for better voice UX --------- Co-authored-by: nx-cloud[bot] <71083854+nx-cloud[bot]@users.noreply.github.com> Co-authored-by: NachoVazquez <NachoVazquez@users.noreply.github.com>
7 days ago by NachoVazquez
Succeeded
main
96d68173 chore: lock zero version
9 days ago by NachoVazquez
Succeeded
main
57e88171 feat: implement effective barge-in for voice interruption (#14) Description Implement client and server-side barge-in to enable users to interrupt TTS playback by speaking. ### Changes - **Client**: Re-chunk incoming TTS into 512-sample chunks (~32ms) for responsive interruption - **Client**: Add bargeIn handler to stop playback when VAD detects speech - **Server**: Handle vad.speech_start by interrupting TTS provider if currently speaking ### Result Users can now interrupt AI speech with sub-50ms latency, providing a natural conversational experience.
9 days ago by NachoVazquez
Succeeded
main
1260d4dd feat: Voice Activity Detection (VAD) Integration & Audio Synchronization (#12) ## Summary This PR implements client-side Voice Activity Detection (VAD) with server synchronization and adds audio packet delay checking for real-time voice interactions. The changes move VAD event handling from the server (Deepgram) to the client (Silero), enabling faster speech detection and better control over the voice conversation flow. ## Key Changes ### 1. **Client-Side VAD Event Handling** (`use-maelstrom-voice.ts`) - **Added VAD callbacks**: `onSpeechStart` and `onSpeechEnd` handlers that send events to the server - **Pre-buffered audio**: When speech is detected, the client now sends audio that was captured *before* VAD detection (via ring buffer), reducing latency - **Speech state tracking**: Introduced `userIsTalkingRef` to track whether the user is currently speaking - **Audio chunk filtering**: Only send audio chunks to the server when `userIsTalkingRef` is true (during active speech) ### 2. **Audio Packet Delay Checking** (`use-maelstrom-voice.ts`) - **Timestamp tracking**: Server-sent TTS audio packets are now timestamped when received - **Delay threshold**: Packets older than 200ms are discarded to maintain real-time playback sync - **Queue management**: Replaced simple audio queue with timestamped queue (`receivedAudioWithTimestampsRef`) - **Logging**: Added debug logs showing packet age and queue status ### 3. **VAD Ring Buffer** (`vad-processor.ts`) - **Ring buffer implementation**: Maintains a sliding window of audio samples matching the speech threshold duration - **Pre-buffered audio export**: When speech is detected, the ring buffer is converted to PCM16 format and sent to the client - **Automatic cleanup**: Ring buffer is cleared after speech start event is emitted ### 4. **Protocol Updates** (`voice-messages.ts`) - **Removed**: `SpeechStartedMessage` and `SpeechStoppedMessage` (server-side TTS notifications) - **Added**: `VADSpeechStartMessage` and `VADSpeechEndMessage` (client-side VAD events) - **Added**: `cancellable` flag to `AudioOutputMessage` for future barge-in support - **Updated union types**: Reflect new message types in `VoiceMessage` and `VoiceServerMessage` ### 5. **Server-Side VAD Event Handling** (`maelstrom-ws.ts`) - **New handlers**: Process `vad.speech_start` and `vad.speech_end` messages from client - **Listening control**: Calls `voiceEngine.startListening()` on speech start and `stopListening()` on speech end - **Error handling**: Graceful error logging for listening state transitions ### 6. **Deepgram STT Cleanup** (`deepgram-stt.ts`) - **Removed VAD configuration**: Deepgram VAD events are no longer used - **Removed VAD event handlers**: Speech start/utterance end event handling removed - **Simplified connect**: Removed VAD-related URL parameters ### 7. **Voice Engine Simplification** (`voice-engine.ts`) - **Removed VAD event handling**: No longer processes VAD events from STT provider - **Removed VAD-based transcript finalization**: Utterance end detection moved to client - **Added debug logging**: Better visibility into audio chunk flow and STT connection state ## Benefits ✅ **Lower latency**: Pre-buffered audio reduces delay between speech detection and server processing ✅ **Better UX**: Client-side VAD provides immediate visual feedback (orb glow) ✅ **Real-time sync**: Packet delay checking prevents stale audio playback ✅ **Cleaner architecture**: VAD responsibility moved to client where it belongs ✅ **Future-ready**: `cancellable` flag enables barge-in (interrupt TTS) support ## Tests to be added
12 days ago by Jacarte
Succeeded
main
ea3e8eed VAD in the client using silero and onnxruntime web (#10) ## PR Type What kind of change does this PR introduce? - [ ] Bugfix - [x] Feature - [ ] Code style update (formatting, local variables) - [ ] Refactoring (no functional changes, no api changes) - [ ] Build related changes - [ ] CI related changes - [ ] Documentation content changes - [ ] Other... Please describe: ## What is the current behavior? The voice orb has no visual feedback based on actual speech detection. Previously, audio level visualization was based on FFT frequency data (`AnalyserNode.getByteFrequencyData()`), which responded to any sound including TTS playback, background noise, and non-speech audio. Yet, the most important part of the PR is to introduce the use of a client based VAD to open the door for a better audio sending. ## What is the new behavior? This PR introduces client-side Voice Activity Detection (VAD) using **Silero VAD + ONNX Runtime Web (WASM)** to make the voice orb glow based on actual speech probability. ### VAD Module (`packages/maelstrom/react/src/lib/vad/`) - **`types.ts`**: Type definitions for VAD configuration, events, and state - **`silero-vad.ts`**: Low-level ONNX Runtime wrapper that loads and runs the Silero VAD v5 model - **`vad-processor.ts`**: Higher-level state machine for speech segment detection with configurable thresholds: - `minSpeechDurationSeconds`: Minimum continuous speech before START_OF_SPEECH (default: 0.25s) - `minSilenceDurationSeconds`: Minimum silence before END_OF_SPEECH (default: 0.5s) - `activationThreshold`: Speech probability threshold (default: 0.5, raised to 0.9 during TTS playback) - **`use-vad.ts`**: React hook exposing `probability`, `start()`, `stop()`, `processAudio()`, `setPlaybackMode()` - **`index.ts`**: Module exports ### Integration in Voice System - **Audio Pipeline**: VAD processes 512-sample windows (32ms @ 16kHz) sliced from AudioWorklet's 4096-sample chunks - **Probability Callback**: `onSpeechProbability` callback updates `audioLevel` state in `useMaelstromVoice` - **Playback Mode**: During TTS playback, VAD threshold is raised to 0.9 to support barge-in while avoiding false positives from speaker audio - **Browser AEC**: Relies on browser's Acoustic Echo Cancellation (`echoCancellation: true` in `getUserMedia`) ### Voice Orb Visualization - Orb scale now driven by VAD speech probability (0-1) - Visual feedback specifically responds to human speech, not arbitrary audio Assets - **Model**: `apps/mar/public/models/silero_vad.v5.onnx` - Silero VAD ONNX model - **WASM**: ONNX Runtime Web WASM files loaded from CDN (`https://cdn.jsdelivr.net/npm/onnxruntime-web@1.24.1/dist/`) - ### ⚠️ Debug Artifacts - TO BE REMOVED BEFORE MERGE This PR contains debugging code that should be removed before merging to production: 1. **`voice-orb.tsx`**: Scale formula `1 * audioLevel` makes orb disappear at 0 probability (intentional for testing AEC (Audio echo cacellation, i.e. MAR voice should not activate the VAD) behavior) 2. **`use-maelstrom-voice.ts`**: Debug logging in `onSpeechProbability` callback (when `debug` prop is true) 4. **Probability label display**: Visual probability indicator shown on the voice button for real-time debugging These artifacts are intentionally left in to verify: - VAD model loads and runs correctly in browser - Browser AEC is working (orb shouldn't glow during TTS playback) - Speech probability values are reasonable ## Does this PR introduce a breaking change? - [ ] Yes - [x] No The `audioLevel` property returned by `useMaelstromVoice` maintains the same interface (0-1 number), only the underlying source changed from FFT-based to VAD-based. ## Other information Dependencies Added - **`onnxruntime-web`**: ONNX Runtime for running Silero VAD model in browser (WASM backend) Technical Details - **Input Format**: Float32Array from AudioWorklet (browser native) - **VAD Window**: 512 samples = 32ms @ 16kHz (Silero requirement) - **Model Input**: 576 samples (64 context + 512 audio window) - **State Management**: RNN state (2×1×128 tensor) maintained across inferences for temporal context ## Known Limitations - WASM cold start adds ~100-200ms on first VAD inference - Browser AEC quality varies by browser/OS - may need threshold tuning
14 days ago by Jacarte
Succeeded
main
9fd0dfe0 fix vite.config.ts to vite.config.mts (#9) ## PR Type What kind of change does this PR introduce? [x] Bugfix [ ] Feature [ ] Code style update (formatting, local variables) [ ] Refactoring (no functional changes, no api changes) [x] Build related changes [ ] CI related changes [ ] Documentation content changes [ ] Other... Please describe: ## What is the current behavior? vite.config.ts was making nx to fail in building ## What is the new behavior? we force vite.config as a module `mts` ## Does this PR introduce a breaking change? [] Yes [x] No
17 days ago by NachoVazquez
Succeeded
main
37c44235 feat: use state management for orchestrator (#8) ## PR Type What kind of change does this PR introduce? [ ] Bugfix [x] Feature [ ] Code style update (formatting, local variables) [x] Refactoring (no functional changes, no api changes) [ ] Build related changes [ ] CI related changes [ ] Documentation content changes [ ] Other... Please describe: ## What is the current behavior? The current implementation uses a manual state management pattern with `dispatch` functions and reducer logic scattered across the codebase. Action handling lacks proper testing infrastructure and relies on external orchestrator implementations. ## What is the new behavior? This PR introduces a comprehensive architectural overhaul: ### State Management Migration - **Migrated to `@xstate/store`**: Replaced manual reducer pattern with `xstate/store` for more robust state management - **New Store API**: All components now use `store.send()` instead of `dispatch()` with improved type safety - **Consistent Event Pattern**: Standardized on event-based state transitions across the codebase ### Action Runner System - **New `useVesselActions` Hook**: Provides stable action runner registration that: - Registers action handlers on component mount - Uses stable wrapper functions that always call the latest implementations - Automatically cleans up on unmount - Prevents re-registration on re-renders (performance optimization) - Supports synchronous and asynchronous action handlers - **Pending Actions Queue**: Robust queuing system for actions across multiple instances - **Instance-Level Isolation**: Each vessel instance maintains independent action runners ### Testing Infrastructure - **Added `@testing-library/react`**: Comprehensive testing setup for React components - **Happy DOM Integration**: Full DOM testing environment setup - **Hook Unit Tests**: 40+ test cases covering: - Runner registration/unregistration lifecycle - Stable wrapper behavior on re-renders - Action execution and error handling - Multiple instance isolation - Async action handling ### Dependency Updates - **Added `@xstate/store ^3.15.0`**: Primary state management library - **Added `@testing-library/react ^16.3.2`**: React component testing - **Added `@testing-library/dom ^10.4.1`**: DOM testing utilities - **Added `happy-dom ^20.5.0`**: DOM environment for testing - **Updated `bun.lock`**: Dependency tree refreshed to include all new packages ### Code Quality Improvements - **Type Safety**: Enhanced TypeScript definitions for all new patterns - **Consistent API**: Standardized nomenclature (`executeAction` → `useVesselActions`) - **Removed Legacy Code**: Deleted deprecated `useActionParams` hook - **Better Separation**: Clear boundaries between orchestrator, state, and UI layers ### Impact on Existing Features - **Labs Component**: Migrated to new action handling pattern with improved state management - **Maelstrom Client**: Complete refactor of orchestrator and state management - **React Integration**: Updated all React hooks to use new store pattern - **Component Isolation**: Better instance management and cleanup ## Does this PR introduce a breaking change? [x] Yes [ ] No ### Breaking Changes 1. **API Change**: `dispatch()` method changed to `send()` on orchestrator 2. **Hook Signature**: `usePendingActions()` signature changed - now requires `runners` parameter 3. **Internal State Structure**: `MaelstromState` now uses `xstate/store` internal structure 4. **Action Registration**: Custom action runners must now use `useVesselActions` hook ### Migration Guide ```typescript // Before const { dispatch } = useMaelstromContext(); dispatch({ type: 'ACTION_TYPE', payload }); // After const { orchestrator } = useMaelstromContext(); orchestrator.send({ type: 'actionType', payload }); // Before usePendingActions(instanceId, onActionHandler); // After useVesselActions(instanceId, { actionName: async (params) => { /* handler */ } }); ``` ## Other information - Performance: New pattern prevents unnecessary re-registrations, improving performance especially with media/audio components - Test Coverage: 95%+ coverage on action handling logic - Type Safety: Full TypeScript support with compile-time guarantees - Documentation: Added comprehensive JSDoc comments for all new APIs - Compatibility: Maintains backward compatibility at the feature level - all user-facing functionality preserved --------- Co-authored-by: nx-cloud[bot] <71083854+nx-cloud[bot]@users.noreply.github.com> Co-authored-by: NachoVazquez <NachoVazquez@users.noreply.github.com>
18 days ago by NachoVazquez
Succeeded
main
1a29264f feat: make action events params more generic (#7) [ ] Bugfix [x] Feature [ ] Code style update (formatting, local variables) [ ] Refactoring (no functional changes, no api changes) [ ] Build related changes [ ] CI related changes [ ] Documentation content changes [ ] Other... Please describe: ActionExecution and ActionStartedEvent use non-generic params typing, making it harder to strongly-type action parameters per action. There isn’t a shared alias representing the “default” action params shape. Added a reusable ActionParams alias: export type ActionParams = Record<string, unknown>; ActionExecution is now generic: ActionExecution<T extends ActionParams = ActionParams> ActionStartedEvent is now generic: ActionStartedEvent<T extends ActionParams = ActionParams> Updated packages/protocol/README.md to document the new generic signatures for both types. [ ] Yes [x] No The default type parameter (= ActionParams) preserves full backward compatibility: existing code that references ActionExecution or ActionStartedEvent without type arguments continues to compile and behave exactly as before. Consumers can now optionally provide a concrete params type (e.g. ActionExecution<{ foo: string }>), improving type safety without requiring any migrations. Closes MAE-12
20 days ago by NachoVazquez
Succeeded
main
dc7e78d6 refactor: organize imports
26 days ago by NachoVazquez
Succeeded
main
0da42449 Feature/create binary tree tilling window layout (#6) ## PR Type What kind of change does this PR introduce? - [ ] Bugfix - [x] Feature - [ ] Code style update (formatting, local variables) - [ ] Refactoring (no functional changes, no api changes) - [ ] Build related changes - [ ] CI related changes - [ ] Documentation content changes - [ ] Other... Please describe: ## What is the current behavior? Maelstrom uses a grid-based layout system where vessel instances are positioned in a fixed grid. Actions from the engine are executed but existing instances don't update their variants when the engine requests changes. Issue Number: N/A ## What is the new behavior? - **Binary tree tiling layout**: New layout mode that arranges vessel instances using a binary tree structure with horizontal/vertical splits, providing more flexible and dynamic layouts - **Action execution fixes**: - Added `useConsumeAndExecutePendingActions` to vessel renderers to properly consume queued actions - Fixed stale closure issue in `useExecuteAction` by using fresh state from orchestrator - Added `emitEvent` to orchestrator interface for action lifecycle events - **Variant updates for existing instances**: `applyTilingInstanceDiff` now updates variants for existing instances when they differ from the layout response - **Improved prompt builder**: Enhanced prompts to guarantee actions are sent with new instances - **Tiling chart component**: New `TilingChart` component with `TilingVesselRenderer` for rendering instances in tiling layout mode ## Does this PR introduce a breaking change? [ ] Yes [x] No ## Other information Related Linear tickets for follow-up improvements: - MAE-36: Redundant Variant State Management - MAE-37: Fragile Action Params Delivery - MAE-38: Missing Action Lifecycle Events - MAE-39: No Action Validation - MAE-40: Scattered Debug Logging - MAE-41: Inconsistent Hook APIs - MAE-42: Missing Error Boundaries Integration - MAE-43: No Tests for Action Flow - MAE-44: Grid vs Tiling Layout Divergence
26 days ago by NachoVazquez
Succeeded
main
0b96e444 feat: unify audio capabilities into maelstrom (#5) ## PR Type What kind of change does this PR introduce?  - [ ] Bugfix - [x] Feature - [ ] Code style update (formatting, local variables) - [x] Refactoring (no functional changes, no api changes) - [ ] Build related changes - [ ] CI related changes - [ ] Documentation content changes - [ ] Other... Please describe: ## What is the current behavior?  The audio/voice capabilities were implemented in a separate maelstrom/audio package with its own isolated implementation. The dashboard component had a complex, monolithic structure that was difficult to maintain. Completes MAE-9 ## What is the new behavior? - Unified voice/audio capabilities into the maelstrom core engine, protocol, and react packages - Removed the standalone maelstrom/audio package in favor of integrated voice functionality - Added comprehensive voice engine with STT (Deepgram) and TTS (Cartesia) provider implementations - Added voice protocol messages for client-server communication - Added React hooks (useMaelstromVoice) and components (VoiceButton, VoiceOrb) for voice integration - Integrated voice WebSocket handling into the maelstrom server API - Added interruption support for voice interactions - Added greeting message functionality - Refactored and simplified the dashboard component structure - Added comprehensive test coverage for: - Voice engine (voice-engine.test.ts) - Voice types (types.test.ts) - Cartesia TTS provider (cartesia-tts.test.ts) - Deepgram STT provider (deepgram-stt.test.ts) - Voice messages protocol (voice-messages.test.ts) - useMaelstromVoice hook (use-maelstrom-voice.test.ts) - Voice connector (voice-connector.test.ts) ## Does this PR introduce a breaking change? - [ ] Yes - [x] No ## Other information Files changed: 56 files with ~9,400 insertions and ~900 deletions Key packages modified: - packages/maelstrom/engine - Voice engine core with STT/TTS providers - packages/maelstrom/protocol - Voice message protocol definitions - packages/maelstrom/react - React hooks and components for voice - packages/maelstrom/client - Client types for voice integration - packages/mar/maelstrom/server/api - WebSocket handling for voice - packages/mar/shell/client/feature - Dashboard refactoring
27 days ago by NachoVazquez
Succeeded
main
abca2d09 feat: use structured outputs on engine instead of asking the agent to produce json as text. (#4) ## PR Type What kind of change does this PR introduce? ``` [ ] Bugfix [x] Feature [ ] Code style update (formatting, local variables) [x] Refactoring (no functional changes, no api changes) [ ] Build related changes [ ] CI related changes [ ] Documentation content changes [ ] Other... Please describe: ``` ## What is the current behavior? The Maelstrom engine uses Vercel AI SDK (`ai` and `@ai-sdk/google`) to communicate with LLMs. The LLM returns free-form text that is then parsed and validated manually in `response-parser.ts`, requiring JSON extraction from code blocks or raw text, with extensive manual type checking. Issue Number: N/A ## What is the new behavior? - **Migrated from Vercel AI SDK to TanStack AI** (`@tanstack/ai` and `@tanstack/ai-gemini`) - **Added structured output support via Zod schemas** - The LLM now returns typed responses directly via `outputSchema`, eliminating manual JSON parsing - **Created `@maelstrom/protocol` schemas** - New `schemas.ts` file with Zod schemas for `MaelstromResponseSchema`, `ActionExecutionSchema`, `LayoutDirectiveSchema`, etc. - **Simplified `response-parser.ts`** - Renamed to `buildResponse()` since parsing is no longer needed; reduced from ~120 lines to ~60 lines - **Removed JSON response format examples from system prompt** - No longer needed since the schema defines the structure - **Updated `LLMProvider` interface** - Now returns `MaelstromResponseFromSchema` instead of raw `string` - **Removed `PARSE_ERROR` and `VALIDATION_ERROR` handling in retry logic** - Structured output eliminates these error types - **Added `paramsJson` field** - Action parameters are now encoded as JSON strings and parsed at runtime ## Does this PR introduce a breaking change? ``` [ ] Yes [x] No ``` ## Other information The migration to TanStack AI with structured outputs improves reliability by letting the LLM provider handle response formatting and validation, reducing the likelihood of malformed responses and simplifying the engine codebase. ****
30 days ago by NachoVazquez
Succeeded
main
7cd407a2 feat: reduce desync between audio and text
1 month ago by NachoVazquez
Succeeded
main
f6284cd3 feat: bring maelstrom to the repo (#3) ## PR Type What kind of change does this PR introduce?  ``` [ ] Bugfix [x] Feature [ ] Code style update (formatting, local variables) [ ] Refactoring (no functional changes, no api changes) [ ] Build related changes [ ] CI related changes [ ] Documentation content changes [ ] Other... Please describe: ``` ## What is the current behavior? We are using Maelstrom via a bun link, which makes it impossible to bundle and deploy. ## What is the new behavior? We have brought Maelstrom into this monorepo so we can more easily develop it until is ready. ## Does this PR introduce a breaking change? ``` [ ] Yes [x] No ```
1 month ago by NachoVazquez
Succeeded
main
101fa876 docs: update readme
1 month ago by NachoVazquez
Succeeded
main
85561b90 ci: cleanup extra stuff
1 month ago by NachoVazquez
Succeeded
main
5933e558 ci: add cloudflare api token
1 month ago by NachoVazquez