Skip to content

Fixes for tasks 211 and 215 #49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 23, 2025
Merged

Conversation

hvaara
Copy link
Contributor

@hvaara hvaara commented Sep 14, 2024

This PR adds 200 OK status code for the mocks in task 211 and 215. For 215 it also adds the option to use response.json() and fixes a spelling error in the description.

Test results for starcoder2-15b:

  • BigCodeBench/211: Progressed past the HTTP error, but saves the zip file to the current directory as opposed to the destination directory. It is actually not specified in the description of the task where to save the zip file. This might be addressed with and update to the task specification or the tests. I'll leave it as future work. Another, bigger, issue is that the zip file is removed with os.remove('temp.zip') at the end, which is against the task description, so the model should fail this task based on my understanding.

  • BigCodeBench/215: Issue resolved through updates to the tests. starcoder2-15b now passes this test in BigCodeBench.

Fixes #33

@terryyz
Copy link
Collaborator

terryyz commented Oct 6, 2024

@hvaara Let me know how it goes with the model verification :) If it works, please put it into tools/fix_0.2.1 and point to the data to v0.1.3.

@hvaara
Copy link
Contributor Author

hvaara commented Feb 19, 2025

@terryyz I'm working on testing resolution with starcoder2-15b where the bug was detected. Which fix file, BIGCODEBENCH_VERSION and BIGCODEBENCH_NEW_VERSION should I be targeting?

@hvaara hvaara changed the title Add status code to mock response for BigCodeBench tasks 211 and 215 Fixes for tasks 211 and 215 Feb 19, 2025
@hvaara
Copy link
Contributor Author

hvaara commented Feb 19, 2025

@terryyz This is ready to be merged assuming I'm making changes to the correct fix file.

@terryyz
Copy link
Collaborator

terryyz commented Feb 20, 2025

Thanks @hvaara! Could you move the part to a file called fix_v023.py, which will be a patch based on the v0.2.3 release.

@hvaara
Copy link
Contributor Author

hvaara commented Feb 20, 2025

@terryyz Thanks for the review! I moved the changes to a new file and updated the patch versions of BIGCODEBENCH_VERSION and BIGCODEBENCH_NEW_VERSION.

@terryyz terryyz merged commit 72ac919 into bigcode-project:main Feb 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

🐛 [TestRemoval/TestRepair] - 211, 215- include status code in mock response
2 participants