ta
TanStack
OSS
ta
Pipelines
Nx runs
Insights
Compare tasks
Analytics
Sign in
Toggle Sidebar
ai
ai
Pipelines
fix(ai): report evaluate usage when answer mapping fails (#1742) * fix(ai): report evaluate usage when answer mapping fails - Wrap only mapAnswers in decide() so a mapping failure reports usage (evaluate:usage event and onUsage) before the error is rethrown - A provider answer that is missing or has the wrong type made decide() throw before usage was reported, so billed tokens were lost - The success path is unchanged, and onFinish still runs only on success - Add tests for the mapping failure, the success event order, a throwing onUsage, and an adapter that throws * docs(evaluate): note that usage is reported when answer mapping fails - Say that onUsage and evaluate:usage still get the usage when an answer is missing or has the wrong type - Bump the evaluate page updatedAt * chore(changeset): say which error decide() throws when onUsage fails - A throwing onUsage hook propagates on the mapping-failure path, same as on the success path - The changeset said decide() always throws the original error, which was not true in that case * test(e2e): cover decide() usage reporting when answer mapping fails - Add an aimock mount at /evaluate-usage-on-failure that serves an OpenAI Decisions response with valid usage and a choice outside the question's options - Add the /api/evaluate-usage-on-failure route, which runs decide() through the OpenAI evaluate adapter and returns the error name and message plus the usage seen by the onUsage hook - Add evaluate-usage-on-failure.spec.ts asserting decide() rejects and the onUsage hook received the usage * fix(ai): log the decide() mapping error before onUsage runs A throwing onUsage hook replaces the mapping error, so log it first. Also correct the e2e comments (the choice has no probability, it is not checked against the options) and widen the docs and changeset wording. --------- Co-authored-by: Tom Beckenham <34339192+tombeckenham@users.noreply.github.com>
nx run-many --targets=build --exclude=examples/**,testing/**
⌘K
fix(ai): report evaluate usage when answer mapping fails (#1742)
Succeeded
in
2m 20s
Started
Linux
4 CPU cores
read-write
access token used
Copy command
nx run-many --targets=build --exclude=examples/**,testing/**
1776
7358801
GitHub
Open on GitHub
Tasks
Resource Usage
ta
TanStack
OSS