Skip to content

Issues: bigcode-project/bigcodebench

[Roadmap] BigCodeBench Q3 2024 Roadmap
#45 opened Sep 10, 2024 by terryyz
Open
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
10 Open 29 Closed
10 Open 29 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

[REQUEST] Remove Reflection-Llama-3.1-70B
#51 by wakamex was closed Oct 15, 2024
1
Pass@100 not supported?
#50 by JohnLins was closed Oct 6, 2024
2
No instructions for evaluating for pass@k where k is greator than 1
#47 by JohnLins was closed Sep 14, 2024
Batch generation (to be released)
#46 by terryyz was closed Oct 6, 2024
@terryyz
1
Upcoming Leaderboard Model Update
#44 by terryyz was closed Sep 13, 2024
@terryyz
1
🤗 [REQUEST] - Reflection-Llama-3.1-70B
#43 by sammcj was closed Sep 13, 2024
2 tasks done
2
[bug] Lingering processes after eval run completion
#42 by hvaara was closed Oct 3, 2024
1
1
Add option to report general issues
#39 by hvaara was closed Sep 4, 2024
2
Batch size configuration when using greedy decoding
#36 by YSLIU627 was closed Aug 15, 2024
4
🐛 [TaskRepair] - 760 Imports
#35 by terryyz was closed Aug 8, 2024
@terryyz
1
[Question] Evaluation by docker does not work bug Something isn't working
#34 by VichyTong was closed Jul 29, 2024
@terryyz
5
🐛 [TaskRemoval/TaskRepair] - 16 Extra period in tests vs prompt bug Something isn't working
#30 by dmelcer9 was closed Sep 10, 2024
1
@terryyz
2
🤗 [REQUEST] - Mathstral
#29 by ethanc8 was closed Jul 28, 2024
2 tasks done
3
🤗 [REQUEST] - Mistral NeMO
#28 by ethanc8 was closed Aug 6, 2024
2 tasks done
1
🤗 [REQUEST] - Codestral Mamba
#27 by ethanc8 was closed Jul 21, 2024
2 tasks done
4
🤗 [REQUEST] - InverseCoder
#26 by terryyz was closed Aug 6, 2024
2 tasks done
@terryyz
4
🐛 [TaskRemoval/TaskRepair] - 178 import unused library bug Something isn't working
#24 by imamnurby was closed Aug 8, 2024
@terryyz
4
🤗 [REQUEST] - NTQAI/Nxcode-CQ-7B-orpo
#22 by bino282 was closed Jul 21, 2024
2 tasks done
@terryyz
4
(Tokenization) Performance Degradation Starting from Transformers v4.42.*
#21 by takkyu2 was closed Jul 17, 2024
@terryyz
17
🤗 [REQUEST] - CodeGeeX-4-9B
#19 by ethanc8 was closed Jul 9, 2024
2 tasks done
6
🤗 [REQUEST] - Gemma 2
#18 by ethanc8 was closed Jul 9, 2024
2 tasks done
@terryyz
4
🤗 [REQUEST] - Artigenz-Coder-DS-6.7B
#15 by ethanc8 was closed Jul 9, 2024
2 tasks done
@terryyz
1
[Feature] BigCodeBench-Hard is coming...
#14 by terryyz was closed Jul 17, 2024
@terryyz
Very low pass@1
#13 by marianna13 was closed Jul 3, 2024
@terryyz
15
🤗 [REQUEST] - ReflectionCoder
#11 by ethanc8 was closed Jun 30, 2024
2 tasks done
@terryyz
6
ProTip! no:milestone will show everything without a milestone.