Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up task submissions a bit #5992

Merged
merged 3 commits into from
Oct 25, 2019
Merged

Speed up task submissions a bit #5992

merged 3 commits into from
Oct 25, 2019

Conversation

ericl
Copy link
Contributor

@ericl ericl commented Oct 24, 2019

Why are these changes needed?

  • port the function descriptor caching from @pcmoritz
  • avoid referencing os.environ when possible
  • avoid setting GPU ids when possible

@@ -149,8 +151,8 @@ def invocation(args, kwargs):
num_return_vals)
else:
object_ids = worker.core_worker.submit_task(
self._function_descriptor.get_function_descriptor_list(),
list_args, num_return_vals, resources)
self._function_descriptor_list, list_args, num_return_vals,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pcmoritz was this the only optimization? I thought there were a couple more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was a little more in #4522 concerning the argument parsing, I'll see if that's still relevant.

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/17852/
Test FAILed.

@@ -294,6 +294,8 @@ def __init__(self, worker):
# to a FunctionExecutionInfo object. This should only be used on
# workers that execute remote functions.
self._function_execution_info = defaultdict(lambda: {})
self._function_descriptors = defaultdict(lambda: {})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two are not needed

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/17861/
Test FAILed.

@ericl
Copy link
Contributor Author

ericl commented Oct 24, 2019

jenkins retest this please

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/17868/
Test FAILed.

@ericl ericl merged commit 4edae7e into ray-project:master Oct 25, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants