Hang when object store full

[ericl]

System information

Ray 0.7.0-dev

Describe the problem

The following workload will hang on test_get_parallel(n=5) with console messages like

pyarrow.lib.PlasmaStoreFull: object does not fit in the plasma store

(pid=23447) 2019-05-27 17:58:37,324	INFO worker.py:392 --
The object with ID ObjectID(c3e2fb3725ff2634d366070bb2e4689c01000000)
already exists in the object store.

That's not expected -- it should raise an error. Ideally it would run successfully, since the working set actually can fit in memory, but that's outside of the scope of this issue.

Source code / logs

import numpy as np
import time
import ray


@ray.remote
class Actor(object):
    def some_expensive_task(self):
        return np.zeros(25 * 1024 * 1024, dtype=np.uint8)


def test_get_serial():
    a = Actor.remote()
    i = 0
    start = time.time()
    for _ in range(100):
        ray.get(a.some_expensive_task.remote())
        i += 1
        if i % 10 == 0:
            print("Calls per second", i / (time.time() - start))
    a.__ray_terminate__.remote()


def test_get_parallel(n=10):
    i = 0
    actors = [Actor.remote() for _ in range(n)]
    start = time.time()
    for _ in range(100):
        pending = [a.some_expensive_task.remote() for a in actors]
        while pending:
            [done], pending = ray.wait(pending, num_returns=1)
            i += 1
            if i % 10 == 0:
                print("Calls per second", i / (time.time() - start))
    for a in actors:
        a.__ray_terminate__.remote()


if __name__ == "__main__":
    ray.init(object_store_memory=100 * 1024 * 1024)
    print("Test serial")
    test_get_serial()
    print("Test parallel 1")
    test_get_parallel(n=1)
    print("Test parallel 5")
    test_get_parallel(n=5)

[ericl]

cc @jovany-wang

[jovany-wang]

@ericl Do you have any thoughts about the log of plasma.err:

The number of bytes in use (before this eviction) is 78645122.
external/plasma/cpp/src/plasma/eviction_policy.cc:81: There is not enough space to create this object, so evicting 1 objects to free up 26215040 bytes. The number of bytes in use (before this eviction) is 78645122.
external/plasma/cpp/src/plasma/eviction_policy.cc:81: There is not enough space to create this object, so evicting 1 objects to free up 26215040 bytes. The number of bytes in use (before this eviction) is 78645122.
external/plasma/cpp/src/plasma/eviction_policy.cc:81: There is not enough space to create this object, so evicting 2 objects to free up 2 bytes. The number of bytes in use (before this eviction) is 78645122.
external/plasma/cpp/src/plasma/store.cc:244: Not enough memory to create the object 98b90b6f16d33d75835fe927b211f18701000000, data_size=26215040, metadata_size=0, will send a reply of PlasmaError::OutOfMemory

I can't see why only freed up 2 bytes at this line https://github.com/apache/arrow/blob/master/cpp/src/plasma/eviction_policy.cc#L80?

[ericl]

Hm, maybe there were only trivial objects freeable, and the rest were somehow pinned by python ref counts?

[stephanie-wang]

The reason that this job is hanging right now seems to be a combination of:

1. All of the tasks finish simultaneously, so they try to put objects into the object store at the same time. This makes it look like there are no freeable objects, even though there soon will be (as soon as one of the tasks succeeds and releases its object).
2. This plasma bug that prevents a client from creating the same object after an initial attempt returns OutOfMemory. This is what's preventing us from putting the exception into the object store, causing the hang.

I confirmed that if we add a random timeout in some_expensive_task, then test_get_parallel(n=5) also succeeds. So we should add a retry attempt when storing task outputs in the object store if it fails on OutOfMemory to fix the first issue.

There is also a separate issue that we're probably not seeing here but that could in theory come up: we use the object store to store the exception if there is no more memory. This is usually a really small object, but of course it might not fit.

[dwayne10]

@stephanie-wang @richardliaw I'm coming across this issue as well, and I'm on Ray 0.7.2. Unfortunately due to dependency conflicts I cannot upgrade to 0.7.3, where the fix for this seems to have gone in, #5143, and I was wondering if y'all had thoughts on how I can get myself out of such seemingly stuck situations.

Here's what I see: