[sgd] Plasma store stops evicting objects

[richardliaw]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux 16.04
• Ray installed from (source or binary): Latest wheels
• Python version: 3.6
• Exact command to reproduce:
  Using 4 nodes, 4 GPUs each - I placed 2 workers on each node with 2 GPUs each.

Then, using the following command, you can reproduce this issue.
Using this: https://github.com/ray-project/ray/blob/master/python/ray/experimental/sgd/mnist_example.py
python mnist_example.py --num-workers=8 --devices-per-worker=2 --redis-address="localhost:6379" --gpu

Below is the YAML for launching this cluster.

# An unique identifier for the head node and workers of this cluster.
cluster_name: sgd

# The maximum number of workers nodes to launch in addition to the head
# node. This takes precedence over min_workers. min_workers default to 0.
min_workers: 3
max_workers: 3

# Cloud-provider specific configuration.
provider:
    type: aws
    region: us-east-1
    availability_zone: us-east-1c

# How Ray will authenticate with newly launched nodes.
auth:
    ssh_user: ubuntu

head_node:
    InstanceType: p3.8xlarge
    ImageId: ami-3b6bce43


worker_nodes:
    InstanceType: p3.8xlarge
    ImageId: ami-3b6bce43

setup_commands:
    - echo "source activate tensorflow_p36;" >> ~/.bashrc
    - pip uninstall -y ray
    - pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.5.3-cp36-cp36m-manylinux1_x86_64.whl


# Custom commands that will be run on the head node after common setup.
head_setup_commands: []

# Custom commands that will be run on worker nodes after common setup.
worker_setup_commands: []

# # Command to start ray on the head node. You don't need to change this.
head_start_ray_commands:
    - ray stop
    - ray start --head --redis-port=6379 --object-manager-port=8076 --autoscaling-config=~/ray_bootstrap_config.yaml

# Command to start ray on worker nodes. You don't need to change this.
worker_start_ray_commands:
    - ray stop
    - ray start --redis-address=$RAY_HEAD_IP:6379 --object-manager-port=8076

Describe the problem

Plasma store on one node of the 4 runs out of memory, although it shouldn't?

Source code / logs

There is only eviction notifications on one of the nodes, (rather than all 4). I don't have information on where the PSes were placed.

The driver output looks like:

Time per iteration 0.2629866600036621
Iter 340 loss 0.1523076295852661 accuracy [0.9448999762535095, 0.9448999762535095, 0.9448999762535095, 0.9448999762535095, 0.9448999762535095, 0.9448999762535095, 0.9448999762535095, 0.9448999762535095, 0.9448999762535095, 0.9448999762535095, 0.9448999762535095, 0.9448999762535095, 0.9448999762535095, 0.9448999762535095, 0.9448999762535095, 0.9448999762535095]
Time per iteration 0.2672464847564697
Iter 350 loss 0.4333576261997223 accuracy [0.946399986743927, 0.946399986743927, 0.946399986743927, 0.946399986743927, 0.946399986743927, 0.946399986743927, 0.946399986743927, 0.946399986743927, 0.946399986743927, 0.946399986743927, 0.946399986743927, 0.946399986743927, 0.946399986743927, 0.946399986743927, 0.946399986743927, 0.946399986743927]
Time per iteration 0.2713921070098877
Iter 360 loss 0.2528766393661499 accuracy [0.9491000175476074, 0.9491000175476074, 0.9491000175476074, 0.9491000175476074, 0.9491000175476074, 0.9491000175476074, 0.9491000175476074, 0.9491000175476074, 0.9491000175476074, 0.9491000175476074, 0.9491000175476074, 0.9491000175476074, 0.9491000175476074, 0.9491000175476074, 0.9491000175476074, 0.9491000175476074]
Time per iteration 0.2673978805541992
A worker died or was killed while executing task 000000002f1471b2c5fac4cd305015141a5a6a03.
A worker died or was killed while executing task 000000004944a98e88d9bec32a6d5db6b25e5633.

Plasma Store output on one of the workers

(tensorflow_p36) ubuntu@ip-172-31-66-193:~$ cat /tmp/ray/session_2018-11-26_06-58-53_34657/logs/plasma_store_0.* /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:1000: Allowing the Plasma store to use up to 20GB of memory. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:1030: Starting object store with directory /dev/shm and huge page support disabled /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 558 objects to free up 4009582792 bytes. The number of bytes in use (before this eviction) is 13228585856. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 591 objects to free up 4009167768 bytes. The number of bytes in use (before this eviction) is 12822576704. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 591 objects to free up 4009167768 bytes. The number of bytes in use (before this eviction) is 12416567552. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 586 objects to free up 4009165720 bytes. The number of bytes in use (before this eviction) is 11971468336. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 591 objects to free up 4009167768 bytes. The number of bytes in use (before this eviction) is 11461703704. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 477 objects to free up 3184136600 bytes. The number of bytes in use (before this eviction) is 11945395384. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 283 objects to free up 1933679984 bytes. The number of bytes in use (before this eviction) is 12889000400. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 175 objects to free up 1185993568 bytes. The number of bytes in use (before this eviction) is 13530474080. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 114 objects to free up 773472992 bytes. The number of bytes in use (before this eviction) is 13923468800. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 66 objects to free up 464080800 bytes. The number of bytes in use (before this eviction) is 14159265632. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 48 objects to free up 309392192 bytes. The number of bytes in use (before this eviction) is 14316463520. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 29 objects to free up 206257920 bytes. The number of bytes in use (before this eviction) is 14369489136. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 15 objects to free up 103129280 bytes. The number of bytes in use (before this eviction) is 14473661408. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 11 objects to free up 77346800 bytes. The number of bytes in use (before this eviction) is 14512960880. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 9 objects to free up 51565216 bytes. The number of bytes in use (before this eviction) is 14539369560. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 2 objects to free up 25781584 bytes. The number of bytes in use (before this eviction) is 14565151144. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 9 objects to free up 51565216 bytes. The number of bytes in use (before this eviction) is 14539578224. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 2 objects to free up 25781584 bytes. The number of bytes in use (before this eviction) is 14552887448. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 1 objects to free up 12890792 bytes. The number of bytes in use (before this eviction) is 14565778240. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578669032. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578878056. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578878056. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578878056. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578878056. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578878056. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578878056. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578878056. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578878056. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 1 objects to free up 12890792 bytes. The number of bytes in use (before this eviction) is 14565987264. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 1 objects to free up 12890792 bytes. The number of bytes in use (before this eviction) is 14565987264. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14565987264. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14565987264. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14565987264. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14565987264. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14565987264. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14565987264. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14565987264. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14565987264. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14565987264. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14565987264. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14565987264. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14565987264. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14565987264. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 1 objects to free up 12890792 bytes. The number of bytes in use (before this eviction) is 14553096472. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 5 objects to free up 2048 bytes. The number of bytes in use (before this eviction) is 14578878080. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14578878080. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579087104. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579296128. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579505152. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579505152. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579505152. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579505152. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 0 objects to free up 0 bytes. The number of bytes in use (before this eviction) is 14579505152. /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 50 /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 49 /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 48 /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 47 /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 51 /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 46 /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 45 /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 41 /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 18 /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 31 /ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/eviction_policy.cc:85: There is not enough space to create this object, so evicting 674 objects to free up 4008821416 bytes. The number of bytes in use (before this eviction) is 10570683738.

Raylet output logs for one of the worker nodes

(tensorflow_p36) ubuntu@ip-172-31-66-193:~$ tail -n 100 /tmp/ray/session_2018-11-26_06-58-53_34657/logs/raylet_0.* ==> /tmp/ray/session_2018-11-26_06-58-53_34657/logs/raylet_0.err <== Please use alternatives such as official/mnist/dataset.py from tensorflow/models. From sgd.py:49: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version. Instructions for updating: Please use alternatives such as official/mnist/dataset.py from tensorflow/models. WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version. Instructions for updating: Please write your own downloading logic. From /home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version. Instructions for updating: Please write your own downloading logic. WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version. Instructions for updating: Please use tf.data to implement this functionality. From /home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version. Instructions for updating: Please use tf.data to implement this functionality. WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version. Instructions for updating: Please use tf.data to implement this functionality. From /home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version. Instructions for updating: Please use tf.data to implement this functionality. WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version. Instructions for updating: Please use tf.one_hot on tensors. From /home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version. Instructions for updating: Please use tf.one_hot on tensors. WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version. Instructions for updating: Please use alternatives such as official/mnist/dataset.py from tensorflow/models. From /home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version. Instructions for updating: Please use alternatives such as official/mnist/dataset.py from tensorflow/models. WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version. Instructions for updating: Please use tf.data to implement this functionality. From /home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version. Instructions for updating: Please use tf.data to implement this functionality. WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version. Instructions for updating: Please use tf.one_hot on tensors. From /home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version. Instructions for updating: Please use tf.one_hot on tensors. WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version. Instructions for updating: Please use alternatives such as official/mnist/dataset.py from tensorflow/models. From /home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version. Instructions for updating: Please use alternatives such as official/mnist/dataset.py from tensorflow/models. WARNING:tensorflow:From sgd.py:67: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version. Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See @{tf.nn.softmax_cross_entropy_with_logits_v2}.

From sgd.py:67: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See @{tf.nn.softmax_cross_entropy_with_logits_v2}.

WARNING:tensorflow:From sgd.py:67: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See @{tf.nn.softmax_cross_entropy_with_logits_v2}.

From sgd.py:67: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See @{tf.nn.softmax_cross_entropy_with_logits_v2}.

Stats avg: 1.6MiB, median: 3.6KiB, total size: 12.5MiB
len: 8.000000
Stats avg: 6.2MiB, median: 6.2MiB, total size: 12.5MiB
len: 2.000000
Packed grads => 2 tensors
Stats avg: 1.6MiB, median: 3.6KiB, total size: 12.5MiB
len: 8.000000
Stats avg: 6.2MiB, median: 6.2MiB, total size: 12.5MiB
len: 2.000000
Packed grads => 2 tensors
/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.cc:151: Check failed: _s.ok() Bad status: Plasma store is full: object does not fit in the plasma store
/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.cc:151: Check failed: _s.ok() Bad status: Plasma store is full: object does not fit in the plasma store
Fatal Python error: Aborted

Fatal Python error: Aborted

==> /tmp/ray/session_2018-11-26_06-58-53_34657/logs/raylet_0.out <==
/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/pyarrow_files/pyarrow/libarrow.so.12(+0x89ee7)[0x7f4bdc853ee7]
/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/pyarrow_files/pyarrow/libarrow.so.12(+0x89ee7)[0x7f667ba53ee7]
/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/pyarrow_files/pyarrow/libarrow.so.12(_ZN5arrow4util7CerrLogD0Ev+0x8e)[0x7f4bdc8f356e]
/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/pyarrow_files/pyarrow/libarrow.so.12(_ZN5arrow4util7CerrLogD0Ev+0x8e)[0x7f667baf356e]
/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/pyarrow_files/pyarrow/libarrow.so.12(_ZN5arrow4util8ArrowLogD1Ev+0x21)[0x7f4bdc8f3231]
/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.so(_ZN16TensorToPlasmaOpIN5Eigen9GpuDeviceEE12ComputeAsyncEPN10tensorflow15OpKernelContextESt8functionIFvvEE+0x81d)[0x7f4bae1d0d9d]
/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/pyarrow_files/pyarrow/libarrow.so.12(_ZN5arrow4util8ArrowLogD1Ev+0x21)[0x7f667baf3231]
/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.so(_ZN16TensorToPlasmaOpIN5Eigen9GpuDeviceEE12ComputeAsyncEPN10tensorflow15OpKernelContextESt8functionIFvvEE+0x81d)[0x7f664d3d0d9d]
/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so(_ZN10tensorflow13BaseGPUDevice12ComputeAsyncEPNS_13AsyncOpKernelEPNS_15OpKernelContextESt8functionIFvvEE+0x173)[0x7f4c0214bfe3]
/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so(_ZN10tensorflow13BaseGPUDevice12ComputeAsyncEPNS_13AsyncOpKernelEPNS_15OpKernelContextESt8functionIFvvEE+0x173)[0x7f66a132efe3]
/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so(+0x61aae2)[0x7f4c02198ae2]
/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so(+0x61aae2)[0x7f66a137bae2]
/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so(+0x61ad5a)[0x7f66a137bd5a]
/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so(+0x61ad5a)[0x7f4c02198d5a]
/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so(_ZN5Eigen26NonBlockingThreadPoolTemplIN10tensorflow6thread16EigenEnvironmentEE10WorkerLoopEi+0x21a)[0x7f66a13da22a]
/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so(_ZN5Eigen26NonBlockingThreadPoolTemplIN10tensorflow6thread16EigenEnvironmentEE10WorkerLoopEi+0x21a)[0x7f4c021f722a]
/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so(_ZNSt17_Function_handlerIFvvEZN10tensorflow6thread16EigenEnvironment12CreateThreadESt8functionIS0_EEUlvE_E9_M_invokeERKSt9_Any_data+0x32)[0x7f66a13d92f2]
/home/ubuntu/anaconda3/envs/tensorflow_p36/bin/../lib/libstdc++.so.6(+0xafc5c)[0x7f66a377ec5c]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f66b074a6ba]
/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so(_ZNSt17_Function_handlerIFvvEZN10tensorflow6thread16EigenEnvironment12CreateThreadESt8functionIS0_EEUlvE_E9_M_invokeERKSt9_Any_data+0x32)[0x7f4c021f62f2]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f66b048041d]
/home/ubuntu/anaconda3/envs/tensorflow_p36/bin/../lib/libstdc++.so.6(+0xafc5c)[0x7f4c0459bc5c]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f4c115676ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f4c1129d41d]

This will happen in 10 seconds if you set object store memory to 1GB.

[ericl]

Could this be related the the fact that we use anonymous object ids?

Marking as release blocker pending investigation.

[robertnishihara]

@richardliaw I'm seeing

botocore.exceptions.ClientError: An error occurred (AuthFailure) when calling the RunInstances operation: Not authorized for images: [ami-076fb2ffd49699eea]

when starting this config. Is it a private AMI?

[ericl]

@robertnishihara I think you can use any DLAMI here

[richardliaw]

(updated ami in yaml provided) - this also similarly fails on Jenkins, just fyi

[robertnishihara]

Ok, the first error I see when running python mnist_example.py --num-workers=8 --devices-per-worker=4 --redis-address="localhost:6379" --gpu (I'm not using an identical autoscaler setup) is

WARNING: Not updating worker name since `setproctitle` is not installed. Install this with `pip install setproctitle` (or ray[debug]) to enable monitoring of worker processes.
WARNING: Not monitoring node memory since `psutil` is not installed. Install this with `pip install psutil` (or ray[debug]) to enable debugging of memory-related crashes.
Creating SGD workers (8 total, 4 devices per worker)
Waiting for gradient configuration
Remote function __init__ failed with:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/ray/worker.py", line 856, in _process_task
    *arguments)
  File "/opt/conda/lib/python3.6/site-packages/ray/function_manager.py", line 481, in actor_method_executor
    method_returns = method(actor, *args)
  File "/opt/conda/lib/python3.6/site-packages/ray/experimental/sgd/sgd_worker.py", line 116, in __init__
    plasma.build_plasma_tensorflow_op()
  File "/opt/conda/lib/python3.6/site-packages/ray/pyarrow_files/pyarrow/plasma.py", line 73, in build_plasma_tensorflow_op
    tf_plasma_op = tf.load_op_library(TF_PLASMA_OP_PATH)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 60, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /opt/conda/lib/python3.6/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.so: undefined symbol: _ZN5arrow6StatusC1ENS_10StatusCodeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

@pcmoritz any idea about this? Perhaps I'm using the wrong TensorFlow version (I'm using 1.12)?

[pcmoritz]

Anaconda's Tensorflow is using the new C++ ABI it seems: ContinuumIO/anaconda-issues#9826

So we aren't compatible with that at the moment.

[robertnishihara]

#3448 is related, but it does not appear to fix this issue in Jenkins.

[richardliaw]

I tried reproducing this multiple times on different cluster sizes (up to 12 actors, 2 GPUs each) with 10GB obj memory and it seems to be fine. It also doesn't seem to fail on Jenkins anymore.

There was one failure on the first run, but none after the first run so that might be a compilation problem.

I'll try it again with 1GB obj memory but if there is still no issue, I'll close this until it resurfaces.