Retry and exception for hang on memory store full #5143

richardliaw · 2019-07-08T09:14:16Z

What do these changes do?

Aims to address #4878.

TODO:

Merge ARROW-5560: [C++][Plasma] Cannot create Plasma object after OutOfMemory error apache/arrow#4850.
Add test.

Related issue number

Linter

I've run scripts/format.sh to lint the changes in this PR.

AmplabJenkins · 2019-07-08T12:00:45Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15193/
Test PASSed.

ericl · 2019-07-08T20:56:03Z

Shouldn't this be done in plasma?

richardliaw · 2019-07-08T21:43:45Z

Probably, this is just an initial investigation..

AmplabJenkins · 2019-07-10T22:48:24Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15287/
Test FAILed.

richardliaw · 2019-07-10T22:56:42Z

@ericl what should be done in plasma, the retries?

AmplabJenkins · 2019-07-10T23:28:44Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15286/
Test FAILed.

AmplabJenkins · 2019-07-10T23:36:33Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1596/
Test FAILed.

AmplabJenkins · 2019-07-10T23:59:03Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1597/
Test FAILed.

AmplabJenkins · 2019-07-11T00:23:00Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1598/
Test FAILed.

AmplabJenkins · 2019-07-11T02:05:12Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15288/
Test PASSed.

python/ray/tests/test_actor.py

AmplabJenkins · 2019-07-13T03:23:02Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15358/
Test PASSed.

AmplabJenkins · 2019-07-14T01:33:38Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1662/
Test FAILed.

stephanie-wang · 2019-07-15T17:38:33Z

python/ray/worker.py

@@ -931,16 +931,33 @@ def _process_task(self, task, function_execution_info):
        finally:
            self._current_task = None

-        # Store the outputs in the local object store.
+        retries_left = 5


Let's make this and delay constants in ray_constants.py.

stephanie-wang · 2019-07-15T17:51:03Z

python/ray/worker.py

-                if num_returns == 1:
-                    outputs = (outputs, )
-                self._store_outputs_in_object_store(return_object_ids, outputs)
+            while retries_left:


I think we should actually put this retry loop in a lower-level function, either Worker.put_object or Worker.store_and_register. The current fix might work for the return value problem, but we'll probably run into the same issue with ray.put. Also, it seems like this might not work if there are multiple return values and only one of them fails, since then we'll try to put all of the return values into the store again.

ericl · 2019-07-25T00:23:12Z

Is this fixed?

AmplabJenkins · 2019-07-25T01:25:58Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15628/
Test FAILed.

AmplabJenkins · 2019-07-25T02:05:26Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15629/
Test FAILed.

AmplabJenkins · 2019-07-25T03:23:05Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15626/
Test PASSed.

AmplabJenkins · 2019-07-25T04:09:38Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15630/
Test PASSed.

AmplabJenkins · 2019-07-25T04:21:34Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15632/
Test PASSed.

updated

AmplabJenkins · 2019-07-25T08:40:26Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1799/
Test FAILed.

AmplabJenkins · 2019-07-25T08:41:07Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1801/
Test FAILed.

AmplabJenkins · 2019-07-25T08:41:31Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1802/
Test FAILed.

AmplabJenkins · 2019-07-25T08:41:51Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1803/
Test FAILed.

AmplabJenkins · 2019-07-25T08:42:33Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1805/
Test FAILed.

stephanie-wang · 2019-07-25T17:47:09Z

python/ray/tests/test_actor.py

+            return np.zeros(10**7 + 2, dtype=np.uint8)
+
+    actor = LargeMemoryActor.remote()
+    with pytest.raises(ray.exceptions.RayActorError):


Shouldn't this exception also be plasma.PlasmaStoreFull?

python/ray/tests/test_actor.py

stephanie-wang · 2019-07-25T18:01:41Z

python/ray/worker.py

-                "so are are falling back to cloudpickle.".format(type(value)))
-            logger.warning(warning_message)
-            self.store_and_register(object_id, value)
+        def try_store_and_register():


Can you make this a separate method on Worker instead of a closure?

stephanie-wang · 2019-07-25T18:02:11Z

python/ray/worker.py

+                logger.warning(warning_message)
+                self.store_and_register(object_id, value)
+
+        delay = ray_constants.DEFAULT_PUT_OBJECT_DELAY


Can you add a comment here explaining what happens in the loop?

AmplabJenkins · 2019-07-25T19:00:54Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15658/
Test FAILed.

AmplabJenkins · 2019-07-26T00:21:41Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15665/
Test PASSed.

…gmem

AmplabJenkins · 2019-07-26T22:30:05Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15687/
Test PASSed.

AmplabJenkins · 2019-07-27T02:21:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15692/
Test PASSed.

richardliaw · 2019-07-27T08:19:46Z

Looks like fork_consistency fails quite often (but also does so in other PRs), but not in a way that seems to be relevant to this change.

a poor attempt

7afeca5

richardliaw changed the title ~~Retry and exception for hang on memory store full~~ [do not merge, wip] Retry and exception for hang on memory store full Jul 8, 2019

richardliaw added 3 commits July 10, 2019 15:21

Merge branch 'master' into hangmem

48a4782

plasmafix

d463628

Merge branch 'master' into hangmem

bee0cb2

richardliaw changed the title ~~[do not merge, wip] Retry and exception for hang on memory store full~~ Retry and exception for hang on memory store full Jul 10, 2019

formatting

f83feaf

richardliaw added 4 commits July 12, 2019 17:01

Merge branch 'master' into hangmem

8521549

fix up a test

dab9881

fix up test

55e57a3

test

06c445f

richardliaw requested review from stephanie-wang, ericl and raulchen July 13, 2019 00:34

ericl reviewed Jul 13, 2019

View reviewed changes

python/ray/tests/test_actor.py Outdated Show resolved Hide resolved

stephanie-wang previously requested changes Jul 15, 2019

View reviewed changes

stephanie-wang self-assigned this Jul 15, 2019

worker

6255552

put

49e1899

simon-mo added the stability-blocker label Jul 25, 2019

stephanie-wang approved these changes Jul 25, 2019

View reviewed changes

plasma

091e82a

fixtests

b9a5069

richardliaw added 2 commits July 26, 2019 12:16

arrow

674a0bb

Merge branch 'hangmem' of https://github.com/richardliaw/ray into han…

f39b6cb

…gmem

update

93e5abf

richardliaw merged commit 9c00616 into ray-project:master Jul 27, 2019

richardliaw deleted the hangmem branch July 27, 2019 08:20

edoakes pushed a commit to edoakes/ray that referenced this pull request Aug 9, 2019

Retry and exception for hang on memory store full (ray-project#5143)

2a0e66d

dwayne10 mentioned this pull request Dec 25, 2019

Hang when object store full #4878

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Retry and exception for hang on memory store full #5143

Retry and exception for hang on memory store full #5143

richardliaw commented Jul 8, 2019 •

edited

Loading

AmplabJenkins commented Jul 8, 2019

ericl commented Jul 8, 2019

richardliaw commented Jul 8, 2019

AmplabJenkins commented Jul 10, 2019

richardliaw commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 11, 2019

AmplabJenkins commented Jul 11, 2019

AmplabJenkins commented Jul 13, 2019

AmplabJenkins commented Jul 14, 2019

stephanie-wang Jul 15, 2019

stephanie-wang Jul 15, 2019

ericl commented Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

stephanie-wang Jul 25, 2019

stephanie-wang Jul 25, 2019

stephanie-wang Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

AmplabJenkins commented Jul 26, 2019

AmplabJenkins commented Jul 26, 2019

AmplabJenkins commented Jul 27, 2019

richardliaw commented Jul 27, 2019 •

edited

Loading

Retry and exception for hang on memory store full #5143

Retry and exception for hang on memory store full #5143

Conversation

richardliaw commented Jul 8, 2019 • edited Loading

What do these changes do?

Related issue number

Linter

AmplabJenkins commented Jul 8, 2019

ericl commented Jul 8, 2019

richardliaw commented Jul 8, 2019

AmplabJenkins commented Jul 10, 2019

richardliaw commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 11, 2019

AmplabJenkins commented Jul 11, 2019

AmplabJenkins commented Jul 13, 2019

AmplabJenkins commented Jul 14, 2019

stephanie-wang Jul 15, 2019

Choose a reason for hiding this comment

stephanie-wang Jul 15, 2019

Choose a reason for hiding this comment

ericl commented Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

AmplabJenkins commented Jul 25, 2019

stephanie-wang Jul 25, 2019

Choose a reason for hiding this comment

stephanie-wang Jul 25, 2019

Choose a reason for hiding this comment

stephanie-wang Jul 25, 2019

Choose a reason for hiding this comment

AmplabJenkins commented Jul 25, 2019

AmplabJenkins commented Jul 26, 2019

AmplabJenkins commented Jul 26, 2019

AmplabJenkins commented Jul 27, 2019

richardliaw commented Jul 27, 2019 • edited Loading

richardliaw commented Jul 8, 2019 •

edited

Loading

richardliaw commented Jul 27, 2019 •

edited

Loading