Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

regtests: mitigate file permissions issue with podman compose #205

Merged
merged 1 commit into from
Aug 26, 2024

Conversation

snazy
Copy link
Member

@snazy snazy commented Aug 24, 2024

This change works around a file-permissions issue when running regtests via podman compose up --build --exit-code-from regtest (not Docker). CI is happy without this change.

Generally speaking, a chmod -R go+rwx is not good, however, it is only for tests running in a container.

[regtest] | t_cli/src/test_cli.py ....                                               [ 94%]
[regtest] | t_pyspark/src/test_spark_sql_s3_with_privileges.py ssssssssssss          [100%]
[regtest] |
[regtest] | =============================== warnings summary ===============================
[regtest] | ../polaris-venv/lib/python3.8/site-packages/_pytest/cacheprovider.py:479
[regtest] |   /home/spark/polaris-venv/lib/python3.8/site-packages/_pytest/cacheprovider.py:479: PytestCacheWarning: could not create cache path /home/spark/regtests/.pytest_cache/v/cache/nodeids: [Errno 13] Permission denied: '/home/spark/regtests/pytest-cache-files-nm7of75m'
[regtest] |     config.cache.set("cache/nodeids", sorted(self.cached_nodeids))
[regtest] |
[regtest] | ../polaris-venv/lib/python3.8/site-packages/_pytest/stepwise.py:57
[regtest] |   /home/spark/polaris-venv/lib/python3.8/site-packages/_pytest/stepwise.py:57: PytestCacheWarning: could not create cache path /home/spark/regtests/.pytest_cache/v/cache/stepwise: [Errno 13] Permission denied: '/home/spark/regtests/pytest-cache-files-gxv4jemz'
[regtest] |     session.config.cache.set(STEPWISE_CACHE_DIR, [])
[regtest] |
[regtest] | -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
[regtest] | =========== 218 passed, 12 skipped, 2 warnings in 139.01s (0:02:19) ============
[regtest] | Wed 21 Aug 2024 11:21:46 AM UTC: Test SUCCEEDED: t_cli:test_cli.py
[regtest] | Wed 21 Aug 2024 11:21:46 AM UTC: Starting test t_hello_world:hello_world.sh
mkdir: cannot create directory ‘/tmp/polaris-regtests/t_hello_world’: Permission denied
/home/spark/regtests/run.sh: line 121: /tmp/polaris-regtests/t_hello_world/hello_world.sh.stderr: No such file or directory
/home/spark/regtests/run.sh: line 121: /tmp/polaris-regtests/t_hello_world/hello_world.sh.stdout: No such file or directory
[regtest] | Wed 21 Aug 2024 11:21:46 AM UTC: Test run concluded for t_hello_world:hello_world.sh
/home/spark/regtests/run.sh: line 132: /tmp/polaris-regtests/t_hello_world/hello_world.sh.fixdiffs.sh: No such file or directory
[regtest] | Wed 21 Aug 2024 11:21:46 AM UTC: Test FAILED: t_hello_world:hello_world.sh
/home/spark/regtests/run.sh: line 133: /tmp/polaris-regtests/t_hello_world/hello_world.sh.fixdiffs.sh: No such file or directory
chmod: cannot access '/tmp/polaris-regtests/t_hello_world/hello_world.sh.fixdiffs.sh': No such file or directory
[regtest] | Wed 21 Aug 2024 11:21:46 AM UTC: To compare and fix diffs (if 'meld' installed): /tmp/polaris-regtests/t_hello_world/hello_world.sh.fixdiffs.sh
[regtest] | Wed 21 Aug 2024 11:21:46 AM UTC: Or manually diff: diff /tmp/polaris-regtests/t_hello_world/hello_world.sh.stdout /home/spark/regtests/t_hello_world/ref/hello_world.sh.ref
[regtest] | Wed 21 Aug 2024 11:21:46 AM UTC: See stderr from test run for additional diagnostics: /tmp/polaris-regtests/t_hello_world/hello_world.sh.stderr
diff: /tmp/polaris-regtests/t_hello_world/hello_world.sh.stdout: No such file or directory
[regtest] | Wed 21 Aug 2024 11:21:46 AM UTC: Starting pytest t_pyspark:test_spark_sql_s3_with_privileges.py
[regtest] | ============================= test session starts ==============================
[regtest] | platform linux -- Python 3.8.10, pytest-8.2.2, pluggy-1.5.0
[regtest] | rootdir: /home/spark/regtests

This change works around a file-permissions issue when running regtests via `podman compose up --build --exit-code-from regtest` (not Docker). CI is happy without this change.

Generally speaking, a `chmod -R go+rwx` is not good, however, it is only for tests running in a container.

```
[regtest] | t_cli/src/test_cli.py ....                                               [ 94%]
[regtest] | t_pyspark/src/test_spark_sql_s3_with_privileges.py ssssssssssss          [100%]
[regtest] |
[regtest] | =============================== warnings summary ===============================
[regtest] | ../polaris-venv/lib/python3.8/site-packages/_pytest/cacheprovider.py:479
[regtest] |   /home/spark/polaris-venv/lib/python3.8/site-packages/_pytest/cacheprovider.py:479: PytestCacheWarning: could not create cache path /home/spark/regtests/.pytest_cache/v/cache/nodeids: [Errno 13] Permission denied: '/home/spark/regtests/pytest-cache-files-nm7of75m'
[regtest] |     config.cache.set("cache/nodeids", sorted(self.cached_nodeids))
[regtest] |
[regtest] | ../polaris-venv/lib/python3.8/site-packages/_pytest/stepwise.py:57
[regtest] |   /home/spark/polaris-venv/lib/python3.8/site-packages/_pytest/stepwise.py:57: PytestCacheWarning: could not create cache path /home/spark/regtests/.pytest_cache/v/cache/stepwise: [Errno 13] Permission denied: '/home/spark/regtests/pytest-cache-files-gxv4jemz'
[regtest] |     session.config.cache.set(STEPWISE_CACHE_DIR, [])
[regtest] |
[regtest] | -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
[regtest] | =========== 218 passed, 12 skipped, 2 warnings in 139.01s (0:02:19) ============
[regtest] | Wed 21 Aug 2024 11:21:46 AM UTC: Test SUCCEEDED: t_cli:test_cli.py
[regtest] | Wed 21 Aug 2024 11:21:46 AM UTC: Starting test t_hello_world:hello_world.sh
mkdir: cannot create directory ‘/tmp/polaris-regtests/t_hello_world’: Permission denied
/home/spark/regtests/run.sh: line 121: /tmp/polaris-regtests/t_hello_world/hello_world.sh.stderr: No such file or directory
/home/spark/regtests/run.sh: line 121: /tmp/polaris-regtests/t_hello_world/hello_world.sh.stdout: No such file or directory
[regtest] | Wed 21 Aug 2024 11:21:46 AM UTC: Test run concluded for t_hello_world:hello_world.sh
/home/spark/regtests/run.sh: line 132: /tmp/polaris-regtests/t_hello_world/hello_world.sh.fixdiffs.sh: No such file or directory
[regtest] | Wed 21 Aug 2024 11:21:46 AM UTC: Test FAILED: t_hello_world:hello_world.sh
/home/spark/regtests/run.sh: line 133: /tmp/polaris-regtests/t_hello_world/hello_world.sh.fixdiffs.sh: No such file or directory
chmod: cannot access '/tmp/polaris-regtests/t_hello_world/hello_world.sh.fixdiffs.sh': No such file or directory
[regtest] | Wed 21 Aug 2024 11:21:46 AM UTC: To compare and fix diffs (if 'meld' installed): /tmp/polaris-regtests/t_hello_world/hello_world.sh.fixdiffs.sh
[regtest] | Wed 21 Aug 2024 11:21:46 AM UTC: Or manually diff: diff /tmp/polaris-regtests/t_hello_world/hello_world.sh.stdout /home/spark/regtests/t_hello_world/ref/hello_world.sh.ref
[regtest] | Wed 21 Aug 2024 11:21:46 AM UTC: See stderr from test run for additional diagnostics: /tmp/polaris-regtests/t_hello_world/hello_world.sh.stderr
diff: /tmp/polaris-regtests/t_hello_world/hello_world.sh.stdout: No such file or directory
[regtest] | Wed 21 Aug 2024 11:21:46 AM UTC: Starting pytest t_pyspark:test_spark_sql_s3_with_privileges.py
[regtest] | ============================= test session starts ==============================
[regtest] | platform linux -- Python 3.8.10, pytest-8.2.2, pluggy-1.5.0
[regtest] | rootdir: /home/spark/regtests
```
@snazy snazy force-pushed the podman-file-permissions branch from 1fc07a1 to d905918 Compare August 24, 2024 08:05
@snazy snazy marked this pull request as ready for review August 24, 2024 08:06
@jbonofre jbonofre merged commit 7b4aa0b into apache:main Aug 26, 2024
3 checks passed
@snazy snazy deleted the podman-file-permissions branch August 26, 2024 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants