Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert "Set DJANGO_DANDI_WEB_APP_URL for dandi-cli integration tests" #137

Merged
merged 1 commit into from
Jun 3, 2022

Conversation

yarikoptic
Copy link
Member

This reverts commit 32fc8f6.

Long analysis:

test_metadata2asset of dandi-cli fails integration testing started to fail regularly.

First I thought that it started to happen since #126 according to

(base) dandi@drogon:/mnt/backup/dandi/tinuous-logs/dandischema/2022/03$ git grep 'FAILED.*test_metadata2asset.*metadata2asset.json-metadata0' | head
24/github/pr/126/e1853d9/test-dandi-cli.yml/471-failed/3_test-dandi-cli (ubuntu-18.04, 3.8, master, normal).txt:2022-03-24T23:23:13.4506531Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
24/github/pr/126/e1853d9/test-dandi-cli.yml/471-failed/5_test-dandi-cli (macos-latest, 3.8, master, normal).txt:2022-03-24T23:24:11.6748940Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
24/github/pr/126/e1853d9/test-dandi-cli.yml/471-failed/7_test-dandi-cli (ubuntu-18.04, 3.8, dandi-devel, master).txt:2022-03-24T23:24:07.6147814Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
24/github/pr/126/e1853d9/test-dandi-cli.yml/471-failed/test-dandi-cli (macos-latest, 3.8, master, normal)/12_Run dandi-cli tests.txt:2022-03-24T23:24:11.6748630Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
24/github/pr/126/e1853d9/test-dandi-cli.yml/471-failed/test-dandi-cli (ubuntu-18.04, 3.8, dandi-devel, master)/12_Run dandi-cli tests.txt:2022-03-24T23:24:07.6147810Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
24/github/pr/126/e1853d9/test-dandi-cli.yml/471-failed/test-dandi-cli (ubuntu-18.04, 3.8, master, normal)/12_Run dandi-cli tests.txt:2022-03-24T23:23:13.4506528Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
25/github/cron/20220325T060244/3e0edbf/test-dandi-cli.yml/472-failed/3_test-dandi-cli (ubuntu-18.04, 3.8, master, normal).txt:2022-03-25T06:07:01.8173971Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
25/github/cron/20220325T060244/3e0edbf/test-dandi-cli.yml/472-failed/5_test-dandi-cli (macos-latest, 3.8, master, normal).txt:2022-03-25T06:06:32.5237510Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
25/github/cron/20220325T060244/3e0edbf/test-dandi-cli.yml/472-failed/7_test-dandi-cli (ubuntu-18.04, 3.8, dandi-devel, master).txt:2022-03-25T06:07:46.3313297Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
25/github/cron/20220325T060244/3e0edbf/test-dandi-cli.yml/472-failed/test-dandi-cli (macos-latest, 3.8, master, normal)/12_Run dandi-cli tests.txt:2022-03-25T06:06:32.5237500Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]

where it did fail and PR provides little background besides "As discussed in Slack.", then it started to fail until #132 fix where we did set DJANGO_DANDI_WEB_APP_URL but it was still an older dandi-cli, then we boosted dandischema in dandi-cli to 0.7.0 in dandi/dandi-cli#987 , released in 0.39.5 on May 5th, and we immediately started to fail in cron jobs:

(base) dandi@drogon:/mnt/backup/dandi/tinuous-logs/dandischema/2022/05$ git grep 'FAILED.*test_metadata2asset.*metadata2asset.json-metadata0' | grep cron | head
06/github/cron/20220506T060309/34482a2/test-dandi-cli.yml/527-failed/1_test-dandi-cli (windows-2019, 3.8, master, normal).txt:2022-05-06T06:09:25.9624055Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
06/github/cron/20220506T060309/34482a2/test-dandi-cli.yml/527-failed/2_test-dandi-cli (windows-2019, 3.8, release, normal).txt:2022-05-06T06:07:13.7676374Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
06/github/cron/20220506T060309/34482a2/test-dandi-cli.yml/527-failed/3_test-dandi-cli (ubuntu-18.04, 3.8, master, normal).txt:2022-05-06T06:13:43.1372133Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
06/github/cron/20220506T060309/34482a2/test-dandi-cli.yml/527-failed/4_test-dandi-cli (ubuntu-18.04, 3.8, release, normal).txt:2022-05-06T06:11:30.5720453Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
06/github/cron/20220506T060309/34482a2/test-dandi-cli.yml/527-failed/5_test-dandi-cli (macos-latest, 3.8, master, normal).txt:2022-05-06T06:06:44.9864360Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
06/github/cron/20220506T060309/34482a2/test-dandi-cli.yml/527-failed/6_test-dandi-cli (macos-latest, 3.8, release, normal).txt:2022-05-06T06:06:39.2250540Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
06/github/cron/20220506T060309/34482a2/test-dandi-cli.yml/527-failed/7_test-dandi-cli (ubuntu-18.04, 3.8, dandi-devel, master).txt:2022-05-06T06:11:31.5085106Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
06/github/cron/20220506T060309/34482a2/test-dandi-cli.yml/527-failed/8_test-dandi-cli (ubuntu-18.04, 3.8, dandi-devel, release).txt:2022-05-06T06:12:21.3105978Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
...

the fail report

>       assert data.json_dict() == bare_dict
E       AssertionError: assert {'access': [{...n/x-nwb', ...} == {'access': [{...n/x-nwb', ...}
E         Omitting 12 identical items, use -vv to show
E         Left contains 1 more item:
E         {'repository': 'https://dandiarchive.org/'}
E         Full diff:
E           {
E            'access': [{'schemaKey': 'AccessRequirements',
E                        'status': 'dandi:OpenAccess'}],
E            'contentSize': 69105,
E            'digest': {'dandi:dandi-etag': 'e455839e5ab2fa659861f58a423fd17f-1'},
E            'encodingFormat': 'application/x-nwb',
E            'id': 'dandiasset:bfc23fb6192b41c083a7257e09a3702b',
E            'keywords': ['keyword1',
E                         'keyword 2'],
E            'path': '/test/path',
E         +  'repository': 'https://dandiarchive.org/',
E            'schemaKey': 'Asset',
E            'schemaVersion': '0.6.3',
E            'wasAttributedTo': [{'identifier': 'sub-01',
E                                 'schemaKey': 'Participant'}],
E            'wasDerivedFrom': [{'identifier': 'tissue42',
E                                'sampleType': {'name': 'tissuesample',
E                                               'schemaKey': 'SampleType'},
E                                'schemaKey': 'BioSample'}],
E            'wasGeneratedBy': [{'description': 'session_description1',
E                                'identifier': 'session_id1',
E                                'name': 'session_id1',
E                                'schemaKey': 'Session',
E                                'startDate': '2017-04-15T12:00:00+00:00'}],
E           }

suggests that it is about the same repository not being set , and that is ok since dandischema no longer sets it unless DJANGO_DANDI_WEB_APP_URL ... dandi-cli relies on the fact that it is not set, and here we set it via env var, so we should stop doing that.

This reverts commit 32fc8f6.

Long analysis:

test_metadata2asset of dandi-cli fails integration testing started to fail regularly.

First I thought that it started to happen since #126 according to

```
(base) dandi@drogon:/mnt/backup/dandi/tinuous-logs/dandischema/2022/03$ git grep 'FAILED.*test_metadata2asset.*metadata2asset.json-metadata0' | head
24/github/pr/126/e1853d9/test-dandi-cli.yml/471-failed/3_test-dandi-cli (ubuntu-18.04, 3.8, master, normal).txt:2022-03-24T23:23:13.4506531Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
24/github/pr/126/e1853d9/test-dandi-cli.yml/471-failed/5_test-dandi-cli (macos-latest, 3.8, master, normal).txt:2022-03-24T23:24:11.6748940Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
24/github/pr/126/e1853d9/test-dandi-cli.yml/471-failed/7_test-dandi-cli (ubuntu-18.04, 3.8, dandi-devel, master).txt:2022-03-24T23:24:07.6147814Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
24/github/pr/126/e1853d9/test-dandi-cli.yml/471-failed/test-dandi-cli (macos-latest, 3.8, master, normal)/12_Run dandi-cli tests.txt:2022-03-24T23:24:11.6748630Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
24/github/pr/126/e1853d9/test-dandi-cli.yml/471-failed/test-dandi-cli (ubuntu-18.04, 3.8, dandi-devel, master)/12_Run dandi-cli tests.txt:2022-03-24T23:24:07.6147810Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
24/github/pr/126/e1853d9/test-dandi-cli.yml/471-failed/test-dandi-cli (ubuntu-18.04, 3.8, master, normal)/12_Run dandi-cli tests.txt:2022-03-24T23:23:13.4506528Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
25/github/cron/20220325T060244/3e0edbf/test-dandi-cli.yml/472-failed/3_test-dandi-cli (ubuntu-18.04, 3.8, master, normal).txt:2022-03-25T06:07:01.8173971Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
25/github/cron/20220325T060244/3e0edbf/test-dandi-cli.yml/472-failed/5_test-dandi-cli (macos-latest, 3.8, master, normal).txt:2022-03-25T06:06:32.5237510Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
25/github/cron/20220325T060244/3e0edbf/test-dandi-cli.yml/472-failed/7_test-dandi-cli (ubuntu-18.04, 3.8, dandi-devel, master).txt:2022-03-25T06:07:46.3313297Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
25/github/cron/20220325T060244/3e0edbf/test-dandi-cli.yml/472-failed/test-dandi-cli (macos-latest, 3.8, master, normal)/12_Run dandi-cli tests.txt:2022-03-25T06:06:32.5237500Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
```

where it did fail and PR provides little background besides "As discussed in Slack.", then it started to fail until #132 fix where we did set `DJANGO_DANDI_WEB_APP_URL` but it was still an older dandi-cli, then we boosted dandischema in dandi-cli to 0.7.0 in dandi/dandi-cli#987 , released in [0.39.5](https://github.com/dandi/dandi-cli/releases/tag/0.39.5) on May 5th, and we immediately started to fail in cron jobs:

```
(base) dandi@drogon:/mnt/backup/dandi/tinuous-logs/dandischema/2022/05$ git grep 'FAILED.*test_metadata2asset.*metadata2asset.json-metadata0' | grep cron | head
06/github/cron/20220506T060309/34482a2/test-dandi-cli.yml/527-failed/1_test-dandi-cli (windows-2019, 3.8, master, normal).txt:2022-05-06T06:09:25.9624055Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
06/github/cron/20220506T060309/34482a2/test-dandi-cli.yml/527-failed/2_test-dandi-cli (windows-2019, 3.8, release, normal).txt:2022-05-06T06:07:13.7676374Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
06/github/cron/20220506T060309/34482a2/test-dandi-cli.yml/527-failed/3_test-dandi-cli (ubuntu-18.04, 3.8, master, normal).txt:2022-05-06T06:13:43.1372133Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
06/github/cron/20220506T060309/34482a2/test-dandi-cli.yml/527-failed/4_test-dandi-cli (ubuntu-18.04, 3.8, release, normal).txt:2022-05-06T06:11:30.5720453Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
06/github/cron/20220506T060309/34482a2/test-dandi-cli.yml/527-failed/5_test-dandi-cli (macos-latest, 3.8, master, normal).txt:2022-05-06T06:06:44.9864360Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
06/github/cron/20220506T060309/34482a2/test-dandi-cli.yml/527-failed/6_test-dandi-cli (macos-latest, 3.8, release, normal).txt:2022-05-06T06:06:39.2250540Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
06/github/cron/20220506T060309/34482a2/test-dandi-cli.yml/527-failed/7_test-dandi-cli (ubuntu-18.04, 3.8, dandi-devel, master).txt:2022-05-06T06:11:31.5085106Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
06/github/cron/20220506T060309/34482a2/test-dandi-cli.yml/527-failed/8_test-dandi-cli (ubuntu-18.04, 3.8, dandi-devel, release).txt:2022-05-06T06:12:21.3105978Z FAILED tests/test_metadata.py::test_metadata2asset[metadata2asset.json-metadata0]
...
```

the fail report

```
>       assert data.json_dict() == bare_dict
E       AssertionError: assert {'access': [{...n/x-nwb', ...} == {'access': [{...n/x-nwb', ...}
E         Omitting 12 identical items, use -vv to show
E         Left contains 1 more item:
E         {'repository': 'https://dandiarchive.org/'}
E         Full diff:
E           {
E            'access': [{'schemaKey': 'AccessRequirements',
E                        'status': 'dandi:OpenAccess'}],
E            'contentSize': 69105,
E            'digest': {'dandi:dandi-etag': 'e455839e5ab2fa659861f58a423fd17f-1'},
E            'encodingFormat': 'application/x-nwb',
E            'id': 'dandiasset:bfc23fb6192b41c083a7257e09a3702b',
E            'keywords': ['keyword1',
E                         'keyword 2'],
E            'path': '/test/path',
E         +  'repository': 'https://dandiarchive.org/',
E            'schemaKey': 'Asset',
E            'schemaVersion': '0.6.3',
E            'wasAttributedTo': [{'identifier': 'sub-01',
E                                 'schemaKey': 'Participant'}],
E            'wasDerivedFrom': [{'identifier': 'tissue42',
E                                'sampleType': {'name': 'tissuesample',
E                                               'schemaKey': 'SampleType'},
E                                'schemaKey': 'BioSample'}],
E            'wasGeneratedBy': [{'description': 'session_description1',
E                                'identifier': 'session_id1',
E                                'name': 'session_id1',
E                                'schemaKey': 'Session',
E                                'startDate': '2017-04-15T12:00:00+00:00'}],
E           }
```

suggests that it is about the same `repository` not being set  , and that is ok since dandischema no longer sets it unless DJANGO_DANDI_WEB_APP_URL ... dandi-cli relies on the fact that it is not set, and here we set it via env var, so we should stop doing that.
@yarikoptic yarikoptic added the tests Add or improve existing tests label Jun 3, 2022
@codecov
Copy link

codecov bot commented Jun 3, 2022

Codecov Report

Merging #137 (fcda7da) into master (34482a2) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #137   +/-   ##
=======================================
  Coverage   96.67%   96.67%           
=======================================
  Files          18       18           
  Lines        1654     1654           
=======================================
  Hits         1599     1599           
  Misses         55       55           
Flag Coverage Δ
unittests 96.67% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.


Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 34482a2...fcda7da. Read the comment docs.

@yarikoptic
Copy link
Member Author

good -- old green as it should

@yarikoptic yarikoptic merged commit 148a18c into master Jun 3, 2022
@yarikoptic yarikoptic deleted the bf-integration branch June 3, 2022 21:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tests Add or improve existing tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant