feat: add resp format, timeout, prompt caching, & usage to conversation api #4129
+1,050
−383
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds:
llms.WIthPromptCaching(true)that you can set, but setting that and testing against the conformance tests yields an err bc then it sets a bool but openai (and therefore all the providers using openai under the hood in contrib) fail bc it expects this to be a time duration string.API PR: dapr/dapr#9241
I need this PR merged first to then merge my dapr/dapr one.
I tested conformance test on: mistral, openai, anthropic.
I have a few fixes left for ollama, and aws (this one was broken to begin with...).
Issue reference
We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.
Please reference the issue this PR will close: #[issue number]
Checklist
Please make sure you've completed the relevant tasks for this PR, out of the following list:
Note: We expect contributors to open a corresponding documentation PR in the dapr/docs repository. As the implementer, you are the best person to document your work! Implementation PRs will not be merged until the documentation PR is opened and ready for review.