Skip to content

Conversation

@jwlee64
Copy link
Contributor

@jwlee64 jwlee64 commented Dec 18, 2025

Description

  • Enhances LLMAsAJudgeScorer to support MessagesPrompt objects in addition to string prompts
  • Adds field validator to handle prompt references and object deserialization
  • Improves the score() method to format prompts differently based on type
  • Updates test cases with new object digests to reflect changes

Testing

  • Added comprehensive test suite in tests/trace/test_llm_as_a_judge_scorer.py that tests:
    • Publishing and loading a scorer with a prompt reference
    • Scoring with a simple string prompt
    • Scoring with a MessagesPrompt and template variables

Copy link
Contributor Author

jwlee64 commented Dec 18, 2025

This stack of pull requests is managed by Graphite. Learn more about stacking.

@jwlee64 jwlee64 changed the title init Chore Dec 18, 2025
@github-actions
Copy link
Contributor

❌ Documentation Reference Check Failed

No documentation reference found in the PR description. Please add either:

This check is required for all PRs except those that start with "chore(weave)" or explicitly state "docs are not required". Please update your PR description and this check will run again automatically.

@jwlee64 jwlee64 changed the title Chore chore(weave): Adds prompt to LLMAsAJudgeScorer Dec 18, 2025
@jwlee64 jwlee64 marked this pull request as ready for review December 18, 2025 17:11
@jwlee64 jwlee64 requested a review from a team as a code owner December 18, 2025 17:11
@wandbot-3000
Copy link

wandbot-3000 bot commented Dec 18, 2025

@codecov
Copy link

codecov bot commented Dec 18, 2025

Codecov Report

❌ Patch coverage is 82.60870% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
weave/scorers/llm_as_a_judge_scorer.py 82.60% 2 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

@neutralino1 neutralino1 added the merge-post-freeze-12-25 PRs to merge after December '25 freeze ends. label Jan 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-post-freeze-12-25 PRs to merge after December '25 freeze ends.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants