-
Notifications
You must be signed in to change notification settings - Fork 425
Open
Description
The library directly calls exec() directly on model-generated code. The warning notes this should be commented out by default, but it isn't.
human-eval/human_eval/execution.py
Lines 40 to 50 in 6d43fb9
| # WARNING | |
| # This program exists to execute untrusted model-generated code. Although | |
| # it is highly unlikely that model-generated code will do something overtly | |
| # malicious in response to this test suite, model-generated code may act | |
| # destructively due to a lack of model capability or alignment. | |
| # Users are strongly encouraged to sandbox this evaluation suite so that it | |
| # does not perform destructive actions on their host or network. For more | |
| # information on how OpenAI sandboxes its code, see the accompanying paper. | |
| # Once you have read this disclaimer and taken appropriate precautions, | |
| # uncomment the following line and proceed at your own risk: | |
| exec(check_program, exec_globals) |
The best solution would be to provide at least one sandbox integration as a reasonable default. I'm open to contributing this.
Metadata
Metadata
Assignees
Labels
No labels