Loading workspace insights... Statistics interval
7 days30 daysLatest CI Pipeline Executions
cb6be753 feat(ai-grok): audio, speech, and realtime adapters + example wiring (#506)
* feat(ai-grok): add audio and speech adapters for xAI
Add `grokSpeech` (TTS via /v1/tts), `grokTranscription` (STT via /v1/stt),
and `grokRealtime` + `grokRealtimeToken` (Voice Agent via /v1/realtime)
because xAI's standalone audio APIs were shipped publicly and the
adapter previously exposed only text/image/summarize. The TTS/STT
endpoints are not OpenAI-compatible so these adapters use direct fetch
rather than the OpenAI SDK; the realtime API mirrors OpenAI's shape with
URL/provider swaps. E2E coverage is wired via mock.mount('/v1/tts'...)
on aimock.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Merge from upstream
* feat(ai-grok): wire shared debug logger into audio and realtime adapters
Adopt the @tanstack/ai/adapter-internals logger across grokSpeech,
grokTranscription, grokRealtimeToken, and grokRealtime so users can toggle
debug output the same way they do on other adapters — `debug: true` for full
tracing, `debug: false` to silence, or a DebugConfig for per-category control
and a custom Logger. Replaces the remaining console.error / console.warn
calls in the realtime adapter with logger.errors so nothing is lost when
debugging is off.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: apply automated fixes
* fix(ai-grok): correct super() arg order in audio adapters
The transcription and TTS adapters were calling super(config, model),
but BaseTranscriptionAdapter/BaseTTSAdapter expect (model, config),
causing TS2345 build errors.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ai-grok): pass logger to audio adapter tests
After the logger was wired into the audio adapters, the unit tests
need to provide one when calling transcribe/generateSpeech directly
(activities normally inject it via resolveDebugOption).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(ai-grok): route audio adapter tests through core functions
Per project convention, tests should not invoke adapter methods
directly — they call generateSpeech()/generateTranscription() with
the adapter instance, so the core function injects logger, emits
events, and exercises the real public surface.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: apply automated fixes
* fix(ai-grok): address cr-loop round 1 findings
ai-grok realtime adapter:
- cleanup pc/localStream/audioContext/dataChannel on connect() failure
- dataChannelReady rejects on error/close/ICE-failed/timeout
- RTCErrorEvent extracted properly instead of [object Event]
- onmessage parse errors emit to consumers
- input_audio_transcription no longer overrides caller on every update
- response.done preserves idle mode after stopAudioCapture
- setupOutputAudioAnalysis disposes prior audioElement, surfaces autoplay blocks
- audioContext.resume failures emit error instead of silent swallow
- currentMessageId reset on response.created (tool-only turns)
- pc.onconnectionstatechange / oniceconnectionstatechange emit status_change
- sendImage uses object image_url for OpenAI-realtime compatibility
- unknown server events logged via default branch
ai-grok TTS/STT:
- getContentType returns audio/L16 for pcm (valid IANA MIME)
- toAudioFile requires explicit audio_format for bare base64
- transcription option renamed format -> inverse_text_normalization
ai-grok realtime token:
- expires_at unit-safety guard (seconds vs ms)
ai-grok types:
- single source of truth for GrokRealtimeModel (model-meta)
ai-grok tests:
- cover aac/flac in pickCodec test
- normalize header assertions via Headers()
- add realtime-token unit-safety tests
examples/ts-react-chat:
- resolveModel fails loud via InvalidModelOverrideError (no silent fallback)
- audio/speech/transcribe routes return 400 with structured body
testing/e2e:
- media-providers uses valid grok-2-image-1212 model
- test-matrix imports from feature-support (dedupe)
* fix(ai-grok): address cr-loop round 2 confirmation findings
ai-grok realtime adapter:
- shared teardownConnection() helper runs on SDP and post-SDP failure paths, disposing input/output analysers, audio element, mic, data channel, pc, and audio context
- pre-open dataChannelReady rejection on failed/closed/disconnected pc states
- pc.onconnectionstatechange is sole source of status_change (ice handler only rejects)
- sendImage detects data: prefix (no more double-wrap)
ai-grok audio utils:
- malformed data: URI MIME parse throws instead of silently defaulting to audio/mpeg
- empty/missing base64 payload throws
- explicit audioFormat argument wins over URI-embedded MIME
ai-grok TTS:
- audio/L16 content-type includes required rate= parameter from modelOptions.sample_rate
ai-grok tests:
- realtime-token afterEach restores original XAI_API_KEY
- new coverage for malformed data URIs, audioFormat precedence, and rate= in audio/L16
examples/ts-react-chat:
- new UnknownProviderError typed class, 400 mapping in audio/speech/transcribe routes
- server-fns ServerFnError wraps typed adapter errors with stable code/details
* fix(ai-grok): address cr-loop round 3 confirmation findings
examples/ts-react-chat:
- generateSpeechFn/transcribeFn/generateSpeechStreamFn/transcribeStreamFn now wrap adapter construction with rethrowAudioAdapterError for consistent typed-error responses
- realtime image display guards against data:/http(s): double-wrap
ai-grok realtime adapter:
- teardownConnection drains pendingEvents; sendEvent logs and skips after teardown
ai-grok TTS:
- sample_rate always forwarded in output_format so body and contentType rate agree
* fix(ai-grok): address cr-loop round 4 confirmation findings
ai-grok realtime adapter:
- teardownConnection on getUserMedia failure (mic/pc/dataChannel leak on mic denial)
- response.function_call_arguments.done drops event if call_id absent (no item_id fallback)
- isTornDown set at top of teardown to guard handlers firing during close() awaits
- setupInputAudioAnalysis/setupOutputAudioAnalysis skip when torn down
- onconnectionstatechange no longer double-emits status_change during disconnect()
ai-grok audio utils:
- toAudioFile Blob/File branch prefers explicit audioFormat over Blob.type
ai-grok TTS:
- sample_rate forwarded only when caller provides one or codec is pcm (don't override server defaults for container codecs)
Tests updated to cover new audioFormat precedence paths and adjusted sample_rate assertions.
* fix(ai-grok): address cr-loop round 5 confirmation findings
ai-grok realtime adapter:
- pc.connectionState=failed triggers automatic teardownConnection (mic/pc/audioContext no longer leak on spontaneous failure)
- flushPendingEvents wraps send in try/catch; emits error on failure instead of hanging caller
- handleServerEvent case 'error' validates shape of event.error; preserves code/type/param; safe against null/missing fields
- autoplay and audioContext.resume failures log without emitting fatal error events (routine browser-policy outcomes)
- dataChannel.onerror/onclose gated behind isTornDown to suppress post-disconnect error events
examples/ts-react-chat:
- realtime.tsx handleImageUpload validates FileReader result, file.type, and base64 extraction; surfaces errors visibly
* fix(ai-grok): extensionFor maps mulaw/alaw MIME types to sensible filenames
utils/audio.ts produced 'audio.basic' and 'audio.x-alaw-basic' for mulaw/alaw
via the default-branch MIME split. Servers using filename as a format hint
now see 'audio.mulaw' / 'audio.alaw', matching the reverse toMimeType mapping.
* ci: apply automated fixes
* refactor(ai-grok): extract form/body builders, adopt ModelMeta convention, fix xAI realtime event names
Refactors from user review:
adapters:
- tts.ts: extract buildTTSRequestBody helper (codec/sample_rate/voice default
resolution + body assembly). Export getContentType for consumer use.
- transcription.ts: extract buildTranscriptionFormData helper (wire-field
mapping including xAI's named 'format' boolean toggle for inverse text
normalization).
model-meta.ts: audio and realtime models now use the same
`as const satisfies ModelMeta` convention as chat/image models
(GROK_TTS, GROK_STT, GROK_VOICE_FAST_1, GROK_VOICE_THINK_FAST_1) with
input/output modalities and tool_calling / reasoning capabilities.
realtime adapter:
- Replace drive-by 'as' casts on untyped server events with runtime-checked
readers (readString, readObject, readObjectArray); malformed frames return
undefined instead of throwing a TypeError.
- Accept both legacy OpenAI-realtime event names and current xAI voice-agent
names per docs.x.ai: response.output_audio.* / response.output_audio_transcript.* /
response.text.* (plus existing response.audio.* / response.audio_transcript.* /
response.output_text.* aliases for compatibility).
- RealtimeServerError type replaces repeated 'as Error & { code?: string }' casts.
realtime token:
- Wrap request body with { session: { model } } per xAI /v1/realtime/client_secrets
schema (was bare { model } before).
* test(ai-grok): cover realtime token body { session: { model } } shape
* ci: apply automated fixes
* refactor(ai-grok): drop @tanstack/ai-client peer dep by inlining realtime contract
The RealtimeAdapter / RealtimeConnection interfaces are duplicated locally
in src/realtime/realtime-contract.ts. The adapter imports them from there
instead of @tanstack/ai-client, so consumers of @tanstack/ai-grok no longer
have to install @tanstack/ai-client unless they also want to construct a
RealtimeClient from it (structural typing covers that use case).
@tanstack/ai-client stays as a devDependency to run a type-level drift check
(tests/realtime-contract.drift.test-d.ts) that asserts our inlined contract
is bidirectionally assignable to the canonical one. If ai-client ever changes
the interface, that file will fail to compile and we update both in lockstep.
publint --strict: clean.
* ci: apply automated fixes
* fix(ai-grok): address CodeRabbit PR review
- tts.ts / transcription.ts: spread `defaultHeaders` BEFORE Authorization /
Content-Type so a caller-supplied header can't silently clobber the bearer
token or auth content-type.
- utils/audio.ts: new `arrayBufferToBase64` helper — Buffer fast path on
Node, chunked btoa fallback everywhere else (browser, Workers, Bun). Replaces
the Node-only `Buffer.from(arrayBuffer).toString('base64')` in tts.ts.
- transcription.ts: new `GrokTranscriptionWord` interface extends the core
`TranscriptionWord` with optional `confidence` and `speaker`. The adapter
now preserves both fields when xAI returns them, so callers that narrow via
`as Array<GrokTranscriptionWord>` get the diarization output they asked
for. Test expectations updated.
- tts.ts: mulaw/alaw `contentType` now includes a `;rate=…` parameter (as
`audio/PCMU` / `audio/PCMA` per RFC 3551) when the caller requests a
non-default sample rate, instead of the 8 kHz-implying `audio/basic` /
`audio/x-alaw-basic`.
- realtime/adapter.ts: `conversation.item.truncated` flips mode back to
`listening` so the visualiser can't get stuck on `speaking` after an
interrupt. `sendEvent` wraps `dataChannel.send` in try/catch consistent
with `flushPendingEvents`. The shared `emptyFrequencyData` /
`emptyTimeDomainData` buffers are gone — `getAudioVisualization`
returns a fresh `Uint8Array` per call so consumers can't mutate a
module-level instance.
- realtime/token.ts: adds a 15s `AbortController` timeout on the
client_secrets request so a dead endpoint can't hang the caller forever.
Validates `client_secret.value` / `expires_at` shape at runtime before
dereferencing so a malformed response throws a descriptive error.
- realtime/realtime-contract.ts: JSDoc filename ref updated.
- examples/ts-react-chat audio/speech/transcribe routes: unify the 400
unknown_provider payload under the `provider` key (was `providerId`)
to match the invalid_model_override branch and the request body.
* ci: apply automated fixes
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Alem Tuzlak <t.zlak@hotmail.com> cb6be753 feat(ai-grok): audio, speech, and realtime adapters + example wiring (#506)
* feat(ai-grok): add audio and speech adapters for xAI
Add `grokSpeech` (TTS via /v1/tts), `grokTranscription` (STT via /v1/stt),
and `grokRealtime` + `grokRealtimeToken` (Voice Agent via /v1/realtime)
because xAI's standalone audio APIs were shipped publicly and the
adapter previously exposed only text/image/summarize. The TTS/STT
endpoints are not OpenAI-compatible so these adapters use direct fetch
rather than the OpenAI SDK; the realtime API mirrors OpenAI's shape with
URL/provider swaps. E2E coverage is wired via mock.mount('/v1/tts'...)
on aimock.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Merge from upstream
* feat(ai-grok): wire shared debug logger into audio and realtime adapters
Adopt the @tanstack/ai/adapter-internals logger across grokSpeech,
grokTranscription, grokRealtimeToken, and grokRealtime so users can toggle
debug output the same way they do on other adapters — `debug: true` for full
tracing, `debug: false` to silence, or a DebugConfig for per-category control
and a custom Logger. Replaces the remaining console.error / console.warn
calls in the realtime adapter with logger.errors so nothing is lost when
debugging is off.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: apply automated fixes
* fix(ai-grok): correct super() arg order in audio adapters
The transcription and TTS adapters were calling super(config, model),
but BaseTranscriptionAdapter/BaseTTSAdapter expect (model, config),
causing TS2345 build errors.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ai-grok): pass logger to audio adapter tests
After the logger was wired into the audio adapters, the unit tests
need to provide one when calling transcribe/generateSpeech directly
(activities normally inject it via resolveDebugOption).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(ai-grok): route audio adapter tests through core functions
Per project convention, tests should not invoke adapter methods
directly — they call generateSpeech()/generateTranscription() with
the adapter instance, so the core function injects logger, emits
events, and exercises the real public surface.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: apply automated fixes
* fix(ai-grok): address cr-loop round 1 findings
ai-grok realtime adapter:
- cleanup pc/localStream/audioContext/dataChannel on connect() failure
- dataChannelReady rejects on error/close/ICE-failed/timeout
- RTCErrorEvent extracted properly instead of [object Event]
- onmessage parse errors emit to consumers
- input_audio_transcription no longer overrides caller on every update
- response.done preserves idle mode after stopAudioCapture
- setupOutputAudioAnalysis disposes prior audioElement, surfaces autoplay blocks
- audioContext.resume failures emit error instead of silent swallow
- currentMessageId reset on response.created (tool-only turns)
- pc.onconnectionstatechange / oniceconnectionstatechange emit status_change
- sendImage uses object image_url for OpenAI-realtime compatibility
- unknown server events logged via default branch
ai-grok TTS/STT:
- getContentType returns audio/L16 for pcm (valid IANA MIME)
- toAudioFile requires explicit audio_format for bare base64
- transcription option renamed format -> inverse_text_normalization
ai-grok realtime token:
- expires_at unit-safety guard (seconds vs ms)
ai-grok types:
- single source of truth for GrokRealtimeModel (model-meta)
ai-grok tests:
- cover aac/flac in pickCodec test
- normalize header assertions via Headers()
- add realtime-token unit-safety tests
examples/ts-react-chat:
- resolveModel fails loud via InvalidModelOverrideError (no silent fallback)
- audio/speech/transcribe routes return 400 with structured body
testing/e2e:
- media-providers uses valid grok-2-image-1212 model
- test-matrix imports from feature-support (dedupe)
* fix(ai-grok): address cr-loop round 2 confirmation findings
ai-grok realtime adapter:
- shared teardownConnection() helper runs on SDP and post-SDP failure paths, disposing input/output analysers, audio element, mic, data channel, pc, and audio context
- pre-open dataChannelReady rejection on failed/closed/disconnected pc states
- pc.onconnectionstatechange is sole source of status_change (ice handler only rejects)
- sendImage detects data: prefix (no more double-wrap)
ai-grok audio utils:
- malformed data: URI MIME parse throws instead of silently defaulting to audio/mpeg
- empty/missing base64 payload throws
- explicit audioFormat argument wins over URI-embedded MIME
ai-grok TTS:
- audio/L16 content-type includes required rate= parameter from modelOptions.sample_rate
ai-grok tests:
- realtime-token afterEach restores original XAI_API_KEY
- new coverage for malformed data URIs, audioFormat precedence, and rate= in audio/L16
examples/ts-react-chat:
- new UnknownProviderError typed class, 400 mapping in audio/speech/transcribe routes
- server-fns ServerFnError wraps typed adapter errors with stable code/details
* fix(ai-grok): address cr-loop round 3 confirmation findings
examples/ts-react-chat:
- generateSpeechFn/transcribeFn/generateSpeechStreamFn/transcribeStreamFn now wrap adapter construction with rethrowAudioAdapterError for consistent typed-error responses
- realtime image display guards against data:/http(s): double-wrap
ai-grok realtime adapter:
- teardownConnection drains pendingEvents; sendEvent logs and skips after teardown
ai-grok TTS:
- sample_rate always forwarded in output_format so body and contentType rate agree
* fix(ai-grok): address cr-loop round 4 confirmation findings
ai-grok realtime adapter:
- teardownConnection on getUserMedia failure (mic/pc/dataChannel leak on mic denial)
- response.function_call_arguments.done drops event if call_id absent (no item_id fallback)
- isTornDown set at top of teardown to guard handlers firing during close() awaits
- setupInputAudioAnalysis/setupOutputAudioAnalysis skip when torn down
- onconnectionstatechange no longer double-emits status_change during disconnect()
ai-grok audio utils:
- toAudioFile Blob/File branch prefers explicit audioFormat over Blob.type
ai-grok TTS:
- sample_rate forwarded only when caller provides one or codec is pcm (don't override server defaults for container codecs)
Tests updated to cover new audioFormat precedence paths and adjusted sample_rate assertions.
* fix(ai-grok): address cr-loop round 5 confirmation findings
ai-grok realtime adapter:
- pc.connectionState=failed triggers automatic teardownConnection (mic/pc/audioContext no longer leak on spontaneous failure)
- flushPendingEvents wraps send in try/catch; emits error on failure instead of hanging caller
- handleServerEvent case 'error' validates shape of event.error; preserves code/type/param; safe against null/missing fields
- autoplay and audioContext.resume failures log without emitting fatal error events (routine browser-policy outcomes)
- dataChannel.onerror/onclose gated behind isTornDown to suppress post-disconnect error events
examples/ts-react-chat:
- realtime.tsx handleImageUpload validates FileReader result, file.type, and base64 extraction; surfaces errors visibly
* fix(ai-grok): extensionFor maps mulaw/alaw MIME types to sensible filenames
utils/audio.ts produced 'audio.basic' and 'audio.x-alaw-basic' for mulaw/alaw
via the default-branch MIME split. Servers using filename as a format hint
now see 'audio.mulaw' / 'audio.alaw', matching the reverse toMimeType mapping.
* ci: apply automated fixes
* refactor(ai-grok): extract form/body builders, adopt ModelMeta convention, fix xAI realtime event names
Refactors from user review:
adapters:
- tts.ts: extract buildTTSRequestBody helper (codec/sample_rate/voice default
resolution + body assembly). Export getContentType for consumer use.
- transcription.ts: extract buildTranscriptionFormData helper (wire-field
mapping including xAI's named 'format' boolean toggle for inverse text
normalization).
model-meta.ts: audio and realtime models now use the same
`as const satisfies ModelMeta` convention as chat/image models
(GROK_TTS, GROK_STT, GROK_VOICE_FAST_1, GROK_VOICE_THINK_FAST_1) with
input/output modalities and tool_calling / reasoning capabilities.
realtime adapter:
- Replace drive-by 'as' casts on untyped server events with runtime-checked
readers (readString, readObject, readObjectArray); malformed frames return
undefined instead of throwing a TypeError.
- Accept both legacy OpenAI-realtime event names and current xAI voice-agent
names per docs.x.ai: response.output_audio.* / response.output_audio_transcript.* /
response.text.* (plus existing response.audio.* / response.audio_transcript.* /
response.output_text.* aliases for compatibility).
- RealtimeServerError type replaces repeated 'as Error & { code?: string }' casts.
realtime token:
- Wrap request body with { session: { model } } per xAI /v1/realtime/client_secrets
schema (was bare { model } before).
* test(ai-grok): cover realtime token body { session: { model } } shape
* ci: apply automated fixes
* refactor(ai-grok): drop @tanstack/ai-client peer dep by inlining realtime contract
The RealtimeAdapter / RealtimeConnection interfaces are duplicated locally
in src/realtime/realtime-contract.ts. The adapter imports them from there
instead of @tanstack/ai-client, so consumers of @tanstack/ai-grok no longer
have to install @tanstack/ai-client unless they also want to construct a
RealtimeClient from it (structural typing covers that use case).
@tanstack/ai-client stays as a devDependency to run a type-level drift check
(tests/realtime-contract.drift.test-d.ts) that asserts our inlined contract
is bidirectionally assignable to the canonical one. If ai-client ever changes
the interface, that file will fail to compile and we update both in lockstep.
publint --strict: clean.
* ci: apply automated fixes
* fix(ai-grok): address CodeRabbit PR review
- tts.ts / transcription.ts: spread `defaultHeaders` BEFORE Authorization /
Content-Type so a caller-supplied header can't silently clobber the bearer
token or auth content-type.
- utils/audio.ts: new `arrayBufferToBase64` helper — Buffer fast path on
Node, chunked btoa fallback everywhere else (browser, Workers, Bun). Replaces
the Node-only `Buffer.from(arrayBuffer).toString('base64')` in tts.ts.
- transcription.ts: new `GrokTranscriptionWord` interface extends the core
`TranscriptionWord` with optional `confidence` and `speaker`. The adapter
now preserves both fields when xAI returns them, so callers that narrow via
`as Array<GrokTranscriptionWord>` get the diarization output they asked
for. Test expectations updated.
- tts.ts: mulaw/alaw `contentType` now includes a `;rate=…` parameter (as
`audio/PCMU` / `audio/PCMA` per RFC 3551) when the caller requests a
non-default sample rate, instead of the 8 kHz-implying `audio/basic` /
`audio/x-alaw-basic`.
- realtime/adapter.ts: `conversation.item.truncated` flips mode back to
`listening` so the visualiser can't get stuck on `speaking` after an
interrupt. `sendEvent` wraps `dataChannel.send` in try/catch consistent
with `flushPendingEvents`. The shared `emptyFrequencyData` /
`emptyTimeDomainData` buffers are gone — `getAudioVisualization`
returns a fresh `Uint8Array` per call so consumers can't mutate a
module-level instance.
- realtime/token.ts: adds a 15s `AbortController` timeout on the
client_secrets request so a dead endpoint can't hang the caller forever.
Validates `client_secret.value` / `expires_at` shape at runtime before
dereferencing so a malformed response throws a descriptive error.
- realtime/realtime-contract.ts: JSDoc filename ref updated.
- examples/ts-react-chat audio/speech/transcribe routes: unify the 400
unknown_provider payload under the `provider` key (was `providerId`)
to match the invalid_model_override branch and the request body.
* ci: apply automated fixes
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Alem Tuzlak <t.zlak@hotmail.com>