Skip to content

Conversation

@jamesbraza
Copy link
Collaborator

#6 forgot to use test arg, and when adding it just now, I realized:

  1. accuracy_reward should reuse extract_answer_loose for test
    • This was due to the organic nature of our internal code base as many scripts, arising without a DRY'ing out
  2. The baselines demo could use accuracy_reward directly

@jamesbraza jamesbraza self-assigned this Jun 6, 2025
@jamesbraza jamesbraza added the enhancement New feature or request label Jun 6, 2025
Copilot AI review requested due to automatic review settings June 6, 2025 21:50
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR refactors how the test flag is handled in reward extraction by reusing extract_answer_loose in the accuracy_reward function and updates the README to demonstrate its usage.

  • Replaces duplicated logic with a concise ternary expression in rewards.py.
  • Updates the README to leverage the new accuracy_reward function for baseline demos.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
src/ether0/rewards.py Refactors answer extraction to use a ternary expression based on test.
README.md Removes old extraction logic and shows usage of accuracy_reward.
Comments suppressed due to low confidence (2)

src/ether0/rewards.py:706

  • [nitpick] Consider adding parentheses around the ternary expression to improve readability and clarify the execution order.
extract_answer_loose(content) if test else extract_thought_answer_strict(content, reasoning=reasoning)[1]

README.md:191

  • [nitpick] Consider adding a brief comment explaining the use of strict=True in zip to clarify intent for future maintainers.
for prob_type, reward in zip(test_ds["problem_type"], rewards, strict=True):

@jamesbraza jamesbraza merged commit 97042fd into main Jun 6, 2025
3 checks passed
@jamesbraza jamesbraza deleted the test-in-baseline branch June 6, 2025 22:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants