Skip to content

Conversation

@sicoyle
Copy link
Contributor

@sicoyle sicoyle commented Dec 20, 2025

Description

This PR adds:

  • timeout support for llm providers. As long as the context timeout is set, langchain supports it.
  • prompt caching using metadata passed to the llm providers. This is a direct work around since langchain has a field of llms.WIthPromptCaching(true) that you can set, but setting that and testing against the conformance tests yields an err bc then it sets a bool but openai (and therefore all the providers using openai under the hood in contrib) fail bc it expects this to be a time duration string.
  • I then unfortunately had to add usage metrics in order to actually test my prompt cache field addition workaround. This required a lot of data type translations for langchain.
  • response format field - this is something that users rely on. In dapr agents we do a workaround for this that I want to rm with this addition.
  • I also renamed CacheTTL to ResponseCacheTTL since it is not prompt caching and should be explicitly showing that. I support the old json tag still.
  • i also had to bump langchain versions for some of the new options to be supported (even with the workarounds)

API PR: dapr/dapr#9241

I need this PR merged first to then merge my dapr/dapr one.

I tested conformance test on: mistral, openai, anthropic.
I have a few fixes left for ollama, and aws (this one was broken to begin with...).

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #[issue number]

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

  • Code compiles correctly
  • Created/updated tests
  • Extended the documentation
    • Created the dapr/docs PR:

Note: We expect contributors to open a corresponding documentation PR in the dapr/docs repository. As the implementer, you are the best person to document your work! Implementation PRs will not be merged until the documentation PR is opened and ready for review.

…che works) for convo api

Signed-off-by: Samantha Coyle <[email protected]>
@sicoyle sicoyle requested review from a team as code owners December 20, 2025 00:02
Copy link
Contributor

@CasperGN CasperGN left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sicoyle does this (or the linked pr) also carry bubbling out the model, tokens etc into the metadata api like we discussed?

if textLength == 0 {
return 0
}
return int64((textLength + 3) / 4)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking over the OpenAI example, would it be better to iterate over the words in the input instead?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants