Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate credis with Ray & route task table entries into credis. #1841

Merged
merged 18 commits into from
May 25, 2018
Merged

Integrate credis with Ray & route task table entries into credis. #1841

Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
6e2c2e8
Integrate credis with Ray: route task table entries into credis.
concretevitamin Apr 6, 2018
aadee29
Fix client_test: qualify chain tests under flag; start correct server
concretevitamin Apr 6, 2018
cd1d28a
Refactor so that gcs::CommandType is propagated per Client -> Table
concretevitamin Apr 11, 2018
0380191
Address all other comments
concretevitamin Apr 11, 2018
f951b6e
Address comments, fix tests
concretevitamin Apr 11, 2018
0ebe563
Fix credis_test.py
concretevitamin Apr 11, 2018
bc5df2c
A bunch of script changes, load credis when on
concretevitamin Apr 13, 2018
f73cc73
Fix tests
concretevitamin May 12, 2018
b308290
Skip sharded test
concretevitamin May 12, 2018
12f8b34
Fix stuff that went bad during rebase
concretevitamin May 13, 2018
88714c1
Address comments
concretevitamin May 15, 2018
6460024
Cleanups to address comments
concretevitamin May 21, 2018
13f0d3a
Fix test
concretevitamin May 21, 2018
0a79cca
Address comments
concretevitamin May 23, 2018
0ce22b5
Fixes
concretevitamin May 23, 2018
a27e235
Fix retried port not being actually used
concretevitamin May 24, 2018
321c9f5
Seems passing now.
concretevitamin May 24, 2018
c5e69f6
Use clang-format-3.8
concretevitamin May 25, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions python/ray/services.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -580,21 +580,23 @@ def _start_redis_instance(node_ip_address="127.0.0.1",
for module in modules:
assert os.path.isfile(module)
counter = 0
print('_start_redis_instance; modules:', modules, '; port', port)
if port is not None:
# If a port is specified, then try only once to connect.
num_retries = 1
else:
port = new_port()
print(' port', port)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you look at the above code block, retries should be happening only when the port is not specified, and if the port is specified, then a different random port should be tried each time. This is inconsistent with the error message I'm seeing

testInvalidTaskTableAdd (__main__.TestGlobalStateStore) ... 14397:M 24 May 03:52:08.898 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
14397:M 24 May 03:52:08.898 # Server started, Redis version 3.9.102
14397:M 24 May 03:52:08.898 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
14397:M 24 May 03:52:08.898 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
Waiting for redis server at 127.0.0.1:51098 to respond...
14401:M 24 May 03:52:09.004 # Creating Server TCP listening socket *:59530: bind: Address already in use
Redis failed to start, retrying now.
14402:M 24 May 03:52:09.108 # Creating Server TCP listening socket *:59530: bind: Address already in use
Redis failed to start, retrying now.
14403:M 24 May 03:52:09.212 # Creating Server TCP listening socket *:59530: bind: Address already in use
Redis failed to start, retrying now.
14404:M 24 May 03:52:09.316 # Creating Server TCP listening socket *:59530: bind: Address already in use
Redis failed to start, retrying now.
14405:M 24 May 03:52:09.420 # Creating Server TCP listening socket *:59530: bind: Address already in use
Redis failed to start, retrying now.
14406:M 24 May 03:52:09.524 # Creating Server TCP listening socket *:59530: bind: Address already in use
Redis failed to start, retrying now.
14407:M 24 May 03:52:09.628 # Creating Server TCP listening socket *:59530: bind: Address already in use
Redis failed to start, retrying now.
14408:M 24 May 03:52:09.732 # Creating Server TCP listening socket *:59530: bind: Address already in use
Redis failed to start, retrying now.
14409:M 24 May 03:52:09.836 # Creating Server TCP listening socket *:59530: bind: Address already in use
Redis failed to start, retrying now.
14410:M 24 May 03:52:09.940 # Creating Server TCP listening socket *:59530: bind: Address already in use
Redis failed to start, retrying now.
14411:M 24 May 03:52:10.044 # Creating Server TCP listening socket *:59530: bind: Address already in use
Redis failed to start, retrying now.
14412:M 24 May 03:52:10.147 # Creating Server TCP listening socket *:59530: bind: Address already in use
Redis failed to start, retrying now.
14413:M 24 May 03:52:10.251 # Creating Server TCP listening socket *:59530: bind: Address already in use
Redis failed to start, retrying now.
14414:M 24 May 03:52:10.355 # Creating Server TCP listening socket *:59530: bind: Address already in use
Redis failed to start, retrying now.
14415:M 24 May 03:52:10.459 # Creating Server TCP listening socket *:59530: bind: Address already in use
Redis failed to start, retrying now.
14416:M 24 May 03:52:10.563 # Creating Server TCP listening socket *:59530: bind: Address already in use
Redis failed to start, retrying now.
14417:M 24 May 03:52:10.667 # Creating Server TCP listening socket *:59530: bind: Address already in use
Redis failed to start, retrying now.
14418:M 24 May 03:52:10.771 # Creating Server TCP listening socket *:59530: bind: Address already in use
Redis failed to start, retrying now.
14419:M 24 May 03:52:10.875 # Creating Server TCP listening socket *:59530: bind: Address already in use
Redis failed to start, retrying now.
14420:M 24 May 03:52:10.979 # Creating Server TCP listening socket *:59530: bind: Address already in use
ERROR

Any idea about this?

load_module_args = []
for module in modules:
load_module_args += ["--loadmodule", module]
command = [executable, "--port",
str(port), "--loglevel", "warning"] + load_module_args

while counter < num_retries:
if counter > 0:
print("Redis failed to start, retrying now.")
command = [executable, "--port",
str(port), "--loglevel", "warning"] + load_module_args
p = subprocess.Popen(command, stdout=stdout_file, stderr=stderr_file)
time.sleep(0.1)
# Check if Redis successfully started (or at least if it the executable
Expand All @@ -604,10 +606,11 @@ def _start_redis_instance(node_ip_address="127.0.0.1",
all_processes[PROCESS_TYPE_REDIS_SERVER].append(p)
break
port = new_port()
print(' port', port)
counter += 1
if counter == num_retries:
raise Exception("Couldn't start Redis. Check stderr file " +
stderr_file.name)
raise Exception("Couldn't start Redis. Check stdout file " +
stdout_file.name)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is still incorrect because stdout_file can be None


# Create a Redis client just for configuring Redis.
redis_client = redis.StrictRedis(host="127.0.0.1", port=port)
Expand Down