GODRIVER-2480 create branching for windows KMS venv #1010
Conversation
| @@ -974,18 +974,26 @@ functions: | |||
| ${PREPARE_SHELL} | |||
|
|
|||
| cd ${DRIVERS_TOOLS}/.evergreen/csfle | |||
| source activate_venv.sh || : | |||
There was a problem hiding this comment.
This pipe was being used to prevent errors on the 3.0 1404 ubunutu version, which doesn't appear to have venv installed. To get around this, we check to see if it's installed before running activate_venv, since source is not cygwin-friendly.
| source activate_venv.sh || : | ||
|
|
||
| # only run this if virtualenv is installed | ||
| if ${PYTHON3_BINARY} -m venv ./venv; then |
There was a problem hiding this comment.
Were we getting errors from virtualenv not being installed? Or is this just precautionary?
There was a problem hiding this comment.
Yes, we get this error on 3.0 Ubuntu 14.04:
[2022/06/30 19:38:34.921] /usr/bin/python3: No module named virtualenv
There was a problem hiding this comment.
Should we not even try to start CSE servers when virtualenv is not available, then? Otherwise, I imagine 3.0 Ubuntu will start system failing.
There was a problem hiding this comment.
This branch prevents a script error from exiting when a dependency of activate_venv.sh is not available on the distribution. The resulting background job won't have the necessary tools to start the server if activate_venv.sh doesn't run. Do you have an alternative approach in mind?
There was a problem hiding this comment.
Ah I see, I keep forgetting that because we have pre_error_fails_tasks: true in the config, using something like exit 1 in a pre task will cause the entire test to fail. I think the current setup is fine, then.
To check my understanding: if virtualenv is not an available module attached to the PYTHON3_BINARY, we'll echo "error: no Module named venv", and the background task that starts the CSE servers in the next command will fail silently?
There was a problem hiding this comment.
That is my understanding, here is a test that does that: https://evergreen.mongodb.com/task/mongo_go_driver_tests_legacy_noauth_nossl__version~3.0_os_ssl_32~ubuntu1604_64_go_1_17_test_replicaset_noauth_nossl_patch_404a99a20576ef7d86f62910917c53baf0dd9319_62be16763e8e866769a62b83_22_06_30_21_33_07##comparehashes=404a99a20576ef7d86f62910917c53baf0dd9319&threads=maxonly
We could wrap the logic that starts the server in the next command, though? IIUC the background job will error silently, but that may not be the most graceful way of handling this?
There was a problem hiding this comment.
Please add at least one Windows task that currently fails on the waterfall to the Evergreen build for this PR to confirm it resolves the failures.
| kmstlsvenv/Scripts/python.exe -u kms_kmip_server.py --port 5698 | ||
| else | ||
| ./kmstlsvenv/bin/python3 -u kms_kmip_server.py --port 5698 | ||
| fi |
There was a problem hiding this comment.
Is it necessary to use the Python 3 executable from the venv? Most scripts use the Evergreen expansion ${PYTHON3_BINARY} which is set to a different value on Windows vs Linux vs macOS builds. Could we replace these two different commands with
${PYTHON3_BINARY} -u kms_kmip_server.py --port 5698?
There was a problem hiding this comment.
It is necessary, the python executable created through activate_venv.sh has dependencies that the pre-installed python3 binary does not have. This is what happens when using the pre-install exe:
Administrator@EC2AMAZ-KTJHJ9U ~/drivers-evergreen-tools/.evergreen/csfle
$ C:/python/Python38/python.exe -u kms_kmip_server.py --port 5698
Traceback (most recent call last):
File "kms_kmip_server.py", line 6, in <module>
from kmip.services.server import KmipServer
ModuleNotFoundError: No module named 'kmip'
There was a problem hiding this comment.
@matthewdale Most of the Windows x64 tasks pass but we do get some failures due to this prose test: https://evergreen.mongodb.com/task/mongo_go_driver_tests_42_plus_zlib_zstd_support__version~latest_os_ssl_40~windows_64_go_1_17_test_standalone_auth_ssl_zlib_compression_patch_404a99a20576ef7d86f62910917c53baf0dd9319_62be1a701e2d177d7c214011_22_06_30_21_49_37##comparehashes=404a99a20576ef7d86f62910917c53baf0dd9319&threads=maxonly
I think they are unrelated to this ticket, though. @benjirewis could you opine?
There was a problem hiding this comment.
They should be totally unrelated to this ticket, feel free to ignore. Opened GODRIVER-2483.
| source activate_venv.sh || : | ||
|
|
||
| # only run this if virtualenv is installed | ||
| if ${PYTHON3_BINARY} -m venv ./venv; then |
There was a problem hiding this comment.
Ah I see, I keep forgetting that because we have pre_error_fails_tasks: true in the config, using something like exit 1 in a pre task will cause the entire test to fail. I think the current setup is fine, then.
To check my understanding: if virtualenv is not an available module attached to the PYTHON3_BINARY, we'll echo "error: no Module named venv", and the background task that starts the CSE servers in the next command will fail silently?
.evergreen/config.yml
Outdated
| # TODO (GODRIVER-2239): Stabilize this pip install with a non-forked version of PyKMIP in | ||
| pip install git+https://github.com/kevinAlbs/PyKMIP.git@expand_tls12_ciphers | ||
| else | ||
| echo "error: No module named venv" |
There was a problem hiding this comment.
Optional: Is this actually an error case, or is it expected? If it's expected, consider changing the wording of this line to prevent confusion.
E.g.
Python module venv not found, skipping virtual environment setup...
GODRIVER-2480