Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a Python SDK #2387

Open
epicfaace opened this issue Jun 11, 2020 · 1 comment
Open

Create a Python SDK #2387

epicfaace opened this issue Jun 11, 2020 · 1 comment
Labels
backend feature For new featues p3 Do it some day.

Comments

@epicfaace
Copy link
Member

It might be useful to create a Python SDK for CodaLab. This would make it easier to run jobs in a matrix, or automate repeatable tasks, or even just interact with CodaLab in general -- and it would probably work better than having to create a CLI wrapper as mentioned in #2350.

Perhaps usage of it would look something like this:

import codalab

# run a bundle
uuid = codalab.run("echo hello world")
codalab.wait(uuid)
print(codalab.cat(uuid))

# upload a file
with open("input.txt") as f:
    codalab.upload(f)

# run matrix jobs and wait for all bundles to finish
uuids = []
for i in range(0, 100):
    uuids.push(codalab.run("echo hello world {}".format(i)))
codalab.wait(uuids)

Other benefits of such an integration:

  • This would make it really easy to call CodaLab from a Jupyter notebook -- essentially, it would be an interactive job runner at that point.
  • Once the Python SDK is ready, we could in fact refactor the CLI to use the CodaLab Python SDK. This would not only reduce code duplication, but would also make our codalab tests faster / easier to create / more readable, because we could move our tests to directly use the Python SDK instead of always using the CLI (and keep just a base set of tests to test CLI integration).
  • In the long run, the SDK model is good if we ever were to expand to other languages.
@Levilian
Copy link
Contributor

This does look extremely useful.

@percyliang percyliang added p3 Do it some day. feature For new featues labels Aug 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend feature For new featues p3 Do it some day.
Projects
None yet
Development

No branches or pull requests

4 participants