Difficulty Reproducing HumanEval Results for Instruct Model

[AndreHe02]

Hi, I’ve been trying to replicate the HumanEval results reported in the paper for the Instruct model, but I’m only achieving around 35% accuracy.

A large fraction of responses (~40%, depending on inference settings) come out empty, consisting entirely of <end_of_text> tokens.

Here are some of the inference settings I’ve tried:

temperature=0.1, top_p=0.9, alg=entropy
temperature=0.2, top_p=0.95, alg=entropy
temperature=0.1, top_p=0.9, alg=origin
temperature=0.5, top_p=0.9, alg=origin
temperature=1.0, top_p=0.9, alg=origin

Could you provide guidance on:

• How to avoid the empty-response issue, and
• What inference settings were used for the reported HumanEval results?

Thanks!

[zhxieml]

Hi there! The inference settings we used are available in eval.sh. Such a high rate of empty responses is not expected. Were you able to reproduce our results for the other benchmarks?

[xijia-tao]

Hi there! Changing transformers from 4.46.2 to a newer release (I used 4.53.3) worked for me.