docs: add DeepSeek V3.1 benchmark results to leaderboard #4464
Conversation
Don't ask me why it needs to be defined also with deepseek/ in order to apply.
|
Updated with extended context window, though given that there's still exactly one exhaustion the better score is likely just a bit more luck with sampling. |
|
It seems that deepseek v3.1 is a mixed thinking model and it is mainly a reasoner. Maybe you shall also test deepseek-reasoner. |
I already ran this test but after 50 or so cases the score was also ~70%. It's not clear to me that API endpoint is updated. |
|
They just announced 10mins ago on X the new models in there API maybe I was to soon |
|
...and they also announced a significant price hike for |
|
what are the results for 3.2 reasoner? |
74.2% on the official API, and 70.2% for the chat model See #4551 |
Actual cost was only 0.60$, i.e. I was only charged this much in my account.