[WIP] Bug/log syncer fails with parentheses

[hartikainen]

WIP: Still needs some testing.

What do these changes do?

Changes the rsync command in tune.log_sync to allow special characters to be used in experiment names when logging is turned on.

Related issue number

#2388

[hartikainen]

I don't know if this is ideal or needed. I tested several ways of quoting the local dir without escaping and didn't manage to make it work. Happy to change this if someone has better ideas of how to handle it.

[hartikainen]

This is actually weird since the log sync definitely worked before with whitespaces and some other special characters. Not sure why the parenthesis caused a problem.

[ericl]

Do you have some example input / outputs of this with other quoting strategies?

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/7474/
Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/7473/
Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8566/
Test PASSed.

[hartikainen]

@ericl Sorry for dropping the ball on this earlier. I looked into this again today, and it looks like adding -s flag to the rsync command (as suggested here) together with the changes suggested by you here solves the problem.

Having:

sync_cmd = ((
    """rsync -avz -e "ssh -i {} -o ConnectTimeout=120s """
    """-o StrictHostKeyChecking=no" {} {} """).format(
            quote(ssh_key), source, target))

I get the following error:

Running log sync: rsync -avz -e "ssh -i /home/ubuntu/ray_bootstrap_key.pem -o ConnectTimeout=120s -o StrictHostKeyChecking=no" 'ubuntu@10.138.0.34:'/home/ubuntu/ray_results/my_experiment/my_func_1_alpha=(0.4, None),beta=2_2018-10-06_04-06-590xhbsymz'/' ''/home/ubuntu/ray_results/my_experiment/my_func_1_alpha=(0.4, None),beta=2_2018-10-06_04-06-590xhbsymz'/'
/bin/sh: 1: Syntax error: "(" unexpected

Adding -s to the rsync:

sync_cmd = ((
    """rsync -savz -e "ssh -i {} -o ConnectTimeout=120s """
    """-o StrictHostKeyChecking=no" {} {} """).format(
            quote(ssh_key), source, target))

I get:

Running log sync: rsync -savz -e "ssh -i /home/ubuntu/ray_bootstrap_key.pem -o ConnectTimeout=120s -o StrictHostKeyChecking=no" 'ubuntu@10.138.0.6:/home/ubuntu/ray_results/my_experiment/my_func_1_alpha=(0.4, None),beta=2_2018-10-06_05-44-258v3bql9m/' '/home/ubuntu/ray_results/my_experiment/my_func_1_alpha=(0.4, None),beta=2_2018-10-06_05-44-258v3bql9m/'
receiving incremental file list
./

sent 27 bytes received 59 bytes 57.33 bytes/sec
total size is 0 speedup is 0.00

And

ubuntu@ray-log-sync-parentheses-test-head-09a4127b:~$ ls -la ~/ray_results/my_experiment/my_func_1_alpha\=\(0.4\,\ None\)\,beta\=2_2018-10-06_05-44-258v3bql9m/
total 244
drwxrwxr-x 2 ubuntu ubuntu   4096 Oct  6 05:44 .
drwxrwxr-x 4 ubuntu ubuntu   4096 Oct  6 05:44 ..
-rw-rw-r-- 1 ubuntu ubuntu     33 Oct  6 05:44 params.json
-rw-rw-r-- 1 ubuntu ubuntu  66747 Oct  6 05:48 progress.csv
-rw-rw-r-- 1 ubuntu ubuntu 156693 Oct  6 05:48 result.json

Here's the script I ran:

import ray
from ray import tune


def my_func(config, reporter):  # add the reporter parameter
    import time, numpy as np
    i = 0
    while True:
        reporter(timesteps_total=i, mean_accuracy=i ** config['alpha'][0])
        i += config['beta']
        time.sleep(1)


tune.register_trainable('my_func', my_func)
ray.init(redis_address=ray.services.get_node_ip_address() + ':6379')

tune.run_experiments({
    'my_experiment': {
        'run': 'my_func',
        'trial_resources': {'cpu' : 2},
        'stop': {'mean_accuracy': 100},
        'config': {
            'alpha': tune.grid_search([(0.2, None), (0.4, None)]),
            'beta': tune.grid_search([2]),
        },
        'local_dir': '~/ray_results',
    }
})

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8567/
Test PASSed.

[ericl]

Ah nice, I didn't know about -s!