Loading workspace insights... Statistics interval
7 days30 daysLatest CI Pipeline Executions
26650859 Feat(ai-gemini): Gemini Realtime Adapter (#405)
* Added gemini ephemeral realtime token
* added initial realtime adapter
* ci: apply automated fixes
* Added gemini adapter, refactored realtime client
* Added gemini live option to example
* Fixed refactoring of realtime client
* Bug fixes, added go_away event, more gemini live provider config options
* refactored event emitter and media handling in gemini adapter
* added go_away & usage event handlers in ai-client
* bug fixes
* refactor: Gemini realtime adapter, Gemini realtime client and audio capture & playback, add support for dynamic token updates and session resumption
* renamed providerOptions to modelOptions
* minor changes
* fixed liveConnectConstraints in geminiRealtimeToken overriding GeminiLiveClient setup message
* fixed message handling and bugs
* consolidated all session updates under updateSession method in the realtime client, added more config
* removed old model
* updated changeset
* fix(ai-gemini): address review findings on the realtime adapter
Correctness / error surfacing:
- token.ts: compute token expiry inside generateToken() so a reused adapter no
longer mints tokens with a fixed (eventually past) expiry.
- client.ts: surface an error instead of silently dropping messages when the
socket isn't open; route JSON parse failures to onError instead of swallowing
them; stop the WebSocket close handler from overwriting a surfaced error with
a benign "idle"; detach socket handlers on disconnect so deliberate reconnects
don't emit a spurious idle/disconnect.
- adapter.ts: emit input transcripts with the real `finished` flag instead of
always-final; tear down socket + audio graph if mic/worklet setup fails
(e.g. denied mic) rather than leaking; route playback errors to onError.
- utils.ts: clean up mic/AudioContext on failed start()/init(); drop the empty
resume() catch; revoke worklet blob URLs.
Types / correctness:
- Fix tool double-conversion: build FunctionDeclarations once from the correct
source (Zod tools vs serialized configs) and use genai v2's responseJsonSchema
(the old outputJsonSchema was silently dropped). Removes the `as any` casts and
the dead functionsMap.
- event-emitter: replace Set<RealtimeEventHandler<any>> with a never-param store
(contravariance) — no `any`, no non-null assertions.
- ai-client updateSession: replace the `(this.options as any)` loop with explicit
typed field assignment.
- Flesh out GeminiRealtimeProviderOptions to match what's consumed (googleGrounding,
proactiveAudio) and export it so it's a usable, non-dead type.
- Rename the isSetupCompelete typo to isSetupComplete.
Cleanup:
- Remove leftover debug console.logs (incl. the full-setup-JSON dump); decode
audio base64 once instead of twice.
Tests:
- Add focused tests for parseResponseMessages (server-message -> event mapping)
and per-call token expiry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: apply automated fixes
* fix(ai-client): include tool outputSchema in applySessionConfig
Mirror the outputSchema through the session-update tool mapping the same way
inputSchema is handled, so tool output schemas reach providers (e.g. Gemini's
responseJsonSchema) on mid-session updateSession(), not just at connect.
Addresses a CodeRabbit review comment on PR #405.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(realtime): use updateSession instead of removed setVADMode
This PR replaced the useRealtimeChat `vadMode`/`setVADMode` return values with
a generic `updateSession`. Update the realtime-chat doc snippet to destructure
`updateSession` and switch VAD at runtime via `updateSession({ vadMode })`, so
the (non-ignored) snippet type-checks again under kiira.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Tom Beckenham <34339192+tombeckenham@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> 26650859 Feat(ai-gemini): Gemini Realtime Adapter (#405)
* Added gemini ephemeral realtime token
* added initial realtime adapter
* ci: apply automated fixes
* Added gemini adapter, refactored realtime client
* Added gemini live option to example
* Fixed refactoring of realtime client
* Bug fixes, added go_away event, more gemini live provider config options
* refactored event emitter and media handling in gemini adapter
* added go_away & usage event handlers in ai-client
* bug fixes
* refactor: Gemini realtime adapter, Gemini realtime client and audio capture & playback, add support for dynamic token updates and session resumption
* renamed providerOptions to modelOptions
* minor changes
* fixed liveConnectConstraints in geminiRealtimeToken overriding GeminiLiveClient setup message
* fixed message handling and bugs
* consolidated all session updates under updateSession method in the realtime client, added more config
* removed old model
* updated changeset
* fix(ai-gemini): address review findings on the realtime adapter
Correctness / error surfacing:
- token.ts: compute token expiry inside generateToken() so a reused adapter no
longer mints tokens with a fixed (eventually past) expiry.
- client.ts: surface an error instead of silently dropping messages when the
socket isn't open; route JSON parse failures to onError instead of swallowing
them; stop the WebSocket close handler from overwriting a surfaced error with
a benign "idle"; detach socket handlers on disconnect so deliberate reconnects
don't emit a spurious idle/disconnect.
- adapter.ts: emit input transcripts with the real `finished` flag instead of
always-final; tear down socket + audio graph if mic/worklet setup fails
(e.g. denied mic) rather than leaking; route playback errors to onError.
- utils.ts: clean up mic/AudioContext on failed start()/init(); drop the empty
resume() catch; revoke worklet blob URLs.
Types / correctness:
- Fix tool double-conversion: build FunctionDeclarations once from the correct
source (Zod tools vs serialized configs) and use genai v2's responseJsonSchema
(the old outputJsonSchema was silently dropped). Removes the `as any` casts and
the dead functionsMap.
- event-emitter: replace Set<RealtimeEventHandler<any>> with a never-param store
(contravariance) — no `any`, no non-null assertions.
- ai-client updateSession: replace the `(this.options as any)` loop with explicit
typed field assignment.
- Flesh out GeminiRealtimeProviderOptions to match what's consumed (googleGrounding,
proactiveAudio) and export it so it's a usable, non-dead type.
- Rename the isSetupCompelete typo to isSetupComplete.
Cleanup:
- Remove leftover debug console.logs (incl. the full-setup-JSON dump); decode
audio base64 once instead of twice.
Tests:
- Add focused tests for parseResponseMessages (server-message -> event mapping)
and per-call token expiry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: apply automated fixes
* fix(ai-client): include tool outputSchema in applySessionConfig
Mirror the outputSchema through the session-update tool mapping the same way
inputSchema is handled, so tool output schemas reach providers (e.g. Gemini's
responseJsonSchema) on mid-session updateSession(), not just at connect.
Addresses a CodeRabbit review comment on PR #405.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(realtime): use updateSession instead of removed setVADMode
This PR replaced the useRealtimeChat `vadMode`/`setVADMode` return values with
a generic `updateSession`. Update the realtime-chat doc snippet to destructure
`updateSession` and switch VAD at runtime via `updateSession({ vadMode })`, so
the (non-ignored) snippet type-checks again under kiira.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Tom Beckenham <34339192+tombeckenham@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> 26650859 Feat(ai-gemini): Gemini Realtime Adapter (#405)
* Added gemini ephemeral realtime token
* added initial realtime adapter
* ci: apply automated fixes
* Added gemini adapter, refactored realtime client
* Added gemini live option to example
* Fixed refactoring of realtime client
* Bug fixes, added go_away event, more gemini live provider config options
* refactored event emitter and media handling in gemini adapter
* added go_away & usage event handlers in ai-client
* bug fixes
* refactor: Gemini realtime adapter, Gemini realtime client and audio capture & playback, add support for dynamic token updates and session resumption
* renamed providerOptions to modelOptions
* minor changes
* fixed liveConnectConstraints in geminiRealtimeToken overriding GeminiLiveClient setup message
* fixed message handling and bugs
* consolidated all session updates under updateSession method in the realtime client, added more config
* removed old model
* updated changeset
* fix(ai-gemini): address review findings on the realtime adapter
Correctness / error surfacing:
- token.ts: compute token expiry inside generateToken() so a reused adapter no
longer mints tokens with a fixed (eventually past) expiry.
- client.ts: surface an error instead of silently dropping messages when the
socket isn't open; route JSON parse failures to onError instead of swallowing
them; stop the WebSocket close handler from overwriting a surfaced error with
a benign "idle"; detach socket handlers on disconnect so deliberate reconnects
don't emit a spurious idle/disconnect.
- adapter.ts: emit input transcripts with the real `finished` flag instead of
always-final; tear down socket + audio graph if mic/worklet setup fails
(e.g. denied mic) rather than leaking; route playback errors to onError.
- utils.ts: clean up mic/AudioContext on failed start()/init(); drop the empty
resume() catch; revoke worklet blob URLs.
Types / correctness:
- Fix tool double-conversion: build FunctionDeclarations once from the correct
source (Zod tools vs serialized configs) and use genai v2's responseJsonSchema
(the old outputJsonSchema was silently dropped). Removes the `as any` casts and
the dead functionsMap.
- event-emitter: replace Set<RealtimeEventHandler<any>> with a never-param store
(contravariance) — no `any`, no non-null assertions.
- ai-client updateSession: replace the `(this.options as any)` loop with explicit
typed field assignment.
- Flesh out GeminiRealtimeProviderOptions to match what's consumed (googleGrounding,
proactiveAudio) and export it so it's a usable, non-dead type.
- Rename the isSetupCompelete typo to isSetupComplete.
Cleanup:
- Remove leftover debug console.logs (incl. the full-setup-JSON dump); decode
audio base64 once instead of twice.
Tests:
- Add focused tests for parseResponseMessages (server-message -> event mapping)
and per-call token expiry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: apply automated fixes
* fix(ai-client): include tool outputSchema in applySessionConfig
Mirror the outputSchema through the session-update tool mapping the same way
inputSchema is handled, so tool output schemas reach providers (e.g. Gemini's
responseJsonSchema) on mid-session updateSession(), not just at connect.
Addresses a CodeRabbit review comment on PR #405.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(realtime): use updateSession instead of removed setVADMode
This PR replaced the useRealtimeChat `vadMode`/`setVADMode` return values with
a generic `updateSession`. Update the realtime-chat doc snippet to destructure
`updateSession` and switch VAD at runtime via `updateSession({ vadMode })`, so
the (non-ignored) snippet type-checks again under kiira.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Tom Beckenham <34339192+tombeckenham@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>