Add benchmarks endpoint support

[xiaohua-scale]

Summary

Support benchmark creation API for self-serve projects.

E2E Test:

API request	UI on selfserve

Next

• Update the self-serve API demo to demonstrate the usage: https://colab.research.google.com/drive/1XZjbVtA2oGOdMp0-v0E80ujMB8tbeh7x?usp=sharing
• I'm not going to update the public API doc on scale.com just yet since this is for self-serve projects only at this point. Customers should refer to the API demo for the usage.
• Essentially this API is very similar to the task creation API, except that we have 2 more required params: project and expected_response, and one more optional param: initial_response.

[shaun-scale]

@xiaohua-scale can you please send a PRD or something like a PRD over api-reviews@scaleapi.com before we get too far?

[xiaohua-scale]

@xiaohua-scale can you please send a PRD or something like a PRD over api-reviews@scaleapi.com before we get too far?

As discussed in the email thread. We want to launch this API for self-serve customers first since we have a full-blown UI for self-serve customers to manage the benchmarks.

[shaun-scale]

2 minor spelling things and a comment to address, but otherwise this looks good to me. Approved.

[shaun-scale]

spelling - task types

[shaun-scale]

Missing a space in thefunction

[fatihkurtoglu]

@xiaohua-scale @shaun-scale If we're not adding it to the API docs, should we also skip SDK Readme? Otherwise, it would be good to document it under Readme with an example, with disclaimers for Self-Serve customers.

[fatihkurtoglu]

@xiaohua-scale Could you please add a new test case for the new create_benchmark method under tests/test_client.py?

[shaun-scale]

If we're not adding it to the API docs, should we also skip SDK Readme? Otherwise, it would be good to document it under Readme with an example, with disclaimers for Self-Serve customers.

I think we're good to add it to the SDK Readme - and candidly, I think I want this in our docs with the same disclaimer - people at least know about it then and have one consistent place to access our API. @xiaohua-scale how against adding this to our docs are you?

[xiaohua-scale]

I think we're good to add it to the SDK Readme - and candidly, I think I want this in our docs with the same disclaimer - people at least know about it then and have one consistent place to access our API. @xiaohua-scale how against adding this to our docs are you?

I can add this to Scale doc and SDK readme and just make it clear that this is for self-serve only then.

[xiaohua-scale]

TODO: benchmark endpoint is going to be renamed. Will update the PR later.