Feat/Sandbox

[TheExGenesis]

For now, we have a sandboxed server that we can hit up from outside the sandbox with code and it will return a serialized walker object. (see sandbox/server/index.py

1. receive POST requests containing Python encodings for Walkers as strings,
2. generate the Walker in the sandboxed server,
3. return the serialized walker to the caller.

Evaluation can in principle be done outside of the sandbox since the arbitrary code is only used for generating walklers, which are then safe.

Open questions:

• the example walker descriptions in python do generate walkers but then running walker.validate() returns false. I suspect it's a problem with the validate function rather than the descriptions but did not check.

[TheExGenesis]

Just realized I pulled from build-lego which isn't finished yet, will happily pull again when it's ready for merging.

[honglu2875]

A note on the sandbox for my branch #13:
(my code is kind of quick-and-dirty, so please feel free to do what's the best for your branch)
If #13 is merged, for the toy task ImageOptim to run, I will need an api in the sandbox kind of like the following

@app.route("/eval_func", methods=["POST"])
def evaluate_function():
    req_json = request.get_json()
    code_string = req_json["code"]
    func_name = req_json["func_name"]

    return execute_with_result(code_string, func_name, 5.0).__repr__()

where execute_with_result is just to

• run the program under reliability_guard() in a process,
• take in an extra string of function name (func_name) in the parameter,
• execute the function,
• assuming the result is a np.ndarray and return to me the .tolist() result.

We just need to have a slightly different unsafe_execute. Here is my own quick implementation without much considerations of anything else:

def unsafe_execute_with_result(check_program, func_name, result, timeout):
    import shutil, os
    reliability_guard()

    # Run program.
    try:
        exec_globals = {}
        with swallow_io():
            with time_limit(timeout):
                exec(check_program, exec_globals)
                result.append(exec_globals[func_name]().tolist())

    except BaseException as e:
        result.append(str(e))

I don't know what we do with codes with errors, so that except part is really arbitrary.

[herbiebradley]

@TheExGenesis Thanks for the great work! Can you pull from main, make sure your code still works and can interface with @honglu2875's code (see https://github.com/CarperAI/ELM/blob/main/map_elites/environments.py#L213), then I think we're ready to merge!

[TheExGenesis]

@honglu2875 can I ask why the name _with_result and why you're appending to result rather than just returning?

Also it seems to me different environments might benefit from different dedicated functions since e.g. your interface assumes an ndarray and uses the sandbox for a full eval, while the sodaracer only requires the sandbox to generate the racers, not for eval. I'll accommodate the ImageOptim of course.

[honglu2875]

@honglu2875 can I ask why the name _with_result and why you're appending to result rather than just returning?

Also it seems to me different environments might benefit from different dedicated functions since e.g. your interface assumes an ndarray and uses the sandbox for a full eval, while the sodaracer only requires the sandbox to generate the racers, not for eval. I'll accommodate the ImageOptim of course.

Oh yeah I literally copied&pasted the other codes in codex_execute.py. The name _with_result is nothing important (I was thinking I'm evaluating the function and returning its result). Appending instead of returning is also unnecessary (simply because I copied from the other unsafe_execute function). My codes here are really ugly (plus a serious concurrency issue if we receive a batch of codes and use multiple processes) and only want to give an idea about what it should return. Please forget about them and only keep the idea that it should return the numpy array or list.

Yes, I agree that a different function and interface make total sense. Feel free to also change the name of the endpoint if necessary. The modification on my side is trivial.

I remember @herbiebradley had some codes somewhere about classifying different errors into different numbers. Maybe we can incorporate this into the sandbox and let it return json message like {'status': ..., 'data': ...} for both interface?
Oh no never mind!! You already implemented it for sodaracer! I got confused between the two unsafe_execute in util.py and codex_execute.py (I was modifying the latter instead of the former).
My ImageOptim hasn't taken error codes into account but it would be nice if we also receive it from the sandbox, in case there is the need to improve this image toy task in the future.

[TheExGenesis]

Thanks for responding! I'll make the error codes nicer by returning json. Don't have a great idea of what to do concurrency-wise for the execs, open to suggestions.

EDIT: I added batch processing, it's not concurrent tho.

[TheExGenesis]

@herbiebradley confirmed that the code still works and interface is compatible with H's work.

[herbiebradley]

Looks good, I added a few comments. Thanks for the work!

[TheExGenesis]

Looks good, I added a few comments. Thanks for the work!

all good, handled the comments :)

[herbiebradley]

LGTM, I'll merge now!