Magpie integrate new feats #224

ChaamC · 2021-11-19T18:37:00Z

Overview

Update to latest Magpie version 3.19.0
Tested on a local birdhouse setup.

Changes

Non-breaking changes

Adds new environment variables to handle new Magpie features, which rely on email usage. These features include user registration and approvals + user assignment to groups with terms and conditions.

Breaking changes

N/A

Following PR will have to be merged prior to merging this current PR : pavics-e2e #90

Also, PR birdhouse #197's content has been transfered to this PR.

crim-jenkins-bot · 2021-11-19T18:58:40Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/777/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : magpie-integrate-new-feats
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-46.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/662/

NOTEBOOK TEST RESULTS

    
[2021-11-19T18:45:45.477Z] ============================= test session starts ==============================
[2021-11-19T18:45:45.478Z] platform linux -- Python 3.7.10, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
[2021-11-19T18:45:45.478Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2021-11-19T18:45:45.478Z] plugins: anyio-3.3.0, dash-1.21.0, nbval-0.9.6, tornasync-0.6.0.post2
[2021-11-19T18:45:45.478Z] collected 221 items
[2021-11-19T18:45:45.478Z] 
[2021-11-19T18:45:55.797Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2021-11-19T18:46:46.027Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  8%]
[2021-11-19T18:46:55.830Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2021-11-19T18:47:08.485Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 14%]
[2021-11-19T18:47:11.056Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 19%]
[2021-11-19T18:47:25.980Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb .                 [ 19%]
[2021-11-19T18:47:26.924Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 19%]
[2021-11-19T18:47:32.643Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 23%]
[2021-11-19T18:47:39.879Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 25%]
[2021-11-19T18:50:26.221Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 32%]
[2021-11-19T18:51:28.374Z] ..............                                                           [ 38%]
[2021-11-19T18:51:35.125Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 40%]
[2021-11-19T18:51:37.063Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 43%]
[2021-11-19T18:52:00.045Z] .................                                                        [ 51%]
[2021-11-19T18:52:06.755Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 53%]
[2021-11-19T18:52:08.143Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 54%]
[2021-11-19T18:52:09.205Z] .FFFFFFFF.                                                               [ 58%]
[2021-11-19T18:52:20.393Z] finch-master/docs/source/notebooks/dap_subset.ipynb ..........           [ 63%]
[2021-11-19T18:52:29.316Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 66%]
[2021-11-19T18:53:11.157Z] finch-master/docs/source/notebooks/subset.ipynb ....................     [ 75%]
[2021-11-19T18:53:13.075Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 75%]
[2021-11-19T18:53:16.675Z] ......                                                                   [ 78%]
[2021-11-19T18:53:38.621Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 78%]
[2021-11-19T18:53:51.479Z] .............                                                            [ 84%]
[2021-11-19T18:54:01.489Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 85%]
[2021-11-19T18:54:42.325Z] ....s.                                                                   [ 87%]
[2021-11-19T18:54:57.239Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 88%]
[2021-11-19T18:55:02.457Z] ...                                                                      [ 89%]
[2021-11-19T18:55:29.034Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2021-11-19T18:55:55.338Z] ......                                                                   [ 92%]
[2021-11-19T18:56:00.069Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2021-11-19T18:57:38.170Z] notebooks/stress-tests.ipynb ....                                        [100%]
[2021-11-19T18:57:38.171Z] 
[2021-11-19T18:57:38.171Z] =================================== FAILURES ===================================

tlvu · 2021-11-22T14:14:09Z

@ChaamC can you port the switch to gunicorn from #197 here and then dump that PR. The DEBUG logging, we do not want to keep it in prod, see comment #197 (comment)

birdhouse/env.local.example

birdhouse/config/magpie/magpie.ini.template

CHANGES.md

birdhouse/env.local.example

crim-jenkins-bot · 2021-11-22T21:11:59Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/778/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : magpie-integrate-new-feats
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS

    
</code>

ChaamC · 2021-11-22T21:15:14Z

PR updated with latest feedback and with the switch to gunicorn from #197 (as requested by comment.)

PR #197 will be dumped after this PR is merged, if it works for you. @tlvu @fmigneault

tlvu

LGTM, minor change requested, then good to merge.

Remember:

to merge with latest if applicable so you bump to the good version
to perform the bump2version stuff before merging
to copy/paste the content of the PR description (and update the description if needed) into the merge commit so it shows up nicely in https://github.com/bird-house/birdhouse-deploy/tags
to push the matching tag after you merge

If this new Magpie is 100% backward-compat, then bump2version patch, else bump2version minor.

Thanks.

birdhouse/config/magpie/magpie.ini.template

tlvu · 2021-11-23T18:25:01Z

Sorry, just a note that to check whether this PR is 100% backward-compat, check the diff versus the previous Magpie in this repo, meaning an upgrade from 3.14.0 to 3.18.0 to avoid the previous oopsies here #182 (comment).

Edit:

if not 100% backward-compat, please document in CHANGES.md manual migration steps.

…_port and smtp_ssl

ChaamC · 2021-11-25T20:56:08Z

The Magpie update should be backward compatible, since it includes optional features. Some column was added to the db, but the modifications shouldn't create any conflicts.

PR was updated with latest feedback.

Also, Magpie 3.18.1 will be included in this PR soon (instead of 3.18.0), waiting for a go from @fmigneault to continue.
Then I will be able to bump and merge!

crim-jenkins-bot · 2021-11-25T20:59:16Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/782/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : magpie-integrate-new-feats
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-91.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/666/

NOTEBOOK TEST RESULTS

    
[2021-11-25T20:47:44.722Z] ============================= test session starts ==============================
[2021-11-25T20:47:44.722Z] platform linux -- Python 3.7.10, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
[2021-11-25T20:47:44.722Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2021-11-25T20:47:44.722Z] plugins: anyio-3.3.0, dash-1.21.0, nbval-0.9.6, tornasync-0.6.0.post2
[2021-11-25T20:47:44.722Z] collected 222 items
[2021-11-25T20:47:44.722Z] 
[2021-11-25T20:47:49.231Z] notebooks-auth/test_thredds.ipynb ......FFFF.                            [  4%]
[2021-11-25T20:48:30.772Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  8%]
[2021-11-25T20:48:40.325Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2021-11-25T20:48:49.061Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 14%]
[2021-11-25T20:48:50.730Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb .........F     [ 18%]
[2021-11-25T20:49:05.683Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb .                 [ 19%]
[2021-11-25T20:49:05.683Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 19%]
[2021-11-25T20:50:14.021Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .FFFFFF            [ 22%]
[2021-11-25T20:50:14.976Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb ....F       [ 25%]
[2021-11-25T20:50:45.986Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............F.. [ 31%]
[2021-11-25T20:51:18.543Z] F.FFFF........                                                           [ 38%]
[2021-11-25T20:51:48.445Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb F...             [ 40%]
[2021-11-25T20:51:49.276Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 43%]
[2021-11-25T20:52:13.007Z] .................                                                        [ 51%]
[2021-11-25T20:52:17.219Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb FFFFF           [ 53%]
[2021-11-25T20:52:19.171Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 54%]
[2021-11-25T20:52:19.957Z] .FFFFFFFF.                                                               [ 58%]
[2021-11-25T20:52:28.775Z] finch-master/docs/source/notebooks/dap_subset.ipynb .FF...FFFF           [ 63%]
[2021-11-25T20:52:32.083Z] finch-master/docs/source/notebooks/finch-usage.ipynb FFFFFF              [ 65%]
[2021-11-25T20:52:55.859Z] finch-master/docs/source/notebooks/subset.ipynb .F.FF....FFFFF....FF     [ 74%]
[2021-11-25T20:52:57.233Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 75%]
[2021-11-25T20:53:01.440Z] ......                                                                   [ 77%]
[2021-11-25T20:53:57.852Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 78%]
[2021-11-25T20:54:17.755Z] .............                                                            [ 84%]
[2021-11-25T20:54:27.749Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 84%]
[2021-11-25T20:55:22.273Z] ....s.                                                                   [ 87%]
[2021-11-25T20:55:32.270Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 87%]
[2021-11-25T20:55:40.108Z] ...                                                                      [ 89%]
[2021-11-25T20:55:58.224Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 89%]
[2021-11-25T20:56:27.250Z] ......                                                                   [ 92%]
[2021-11-25T20:56:34.474Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2021-11-25T20:58:29.181Z] notebooks/stress-tests.ipynb ...FF                                       [100%]
[2021-11-25T20:58:29.181Z] 
[2021-11-25T20:58:29.181Z] =================================== FAILURES ===================================

crim-jenkins-bot · 2021-12-02T21:27:25Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/792/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : magpie-integrate-new-feats
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-88.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/675/

NOTEBOOK TEST RESULTS

    
[2021-12-02T21:10:25.599Z] ============================= test session starts ==============================
[2021-12-02T21:10:25.599Z] platform linux -- Python 3.7.10, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
[2021-12-02T21:10:25.599Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2021-12-02T21:10:25.599Z] plugins: anyio-3.3.0, dash-1.21.0, nbval-0.9.6, tornasync-0.6.0.post2
[2021-12-02T21:10:25.599Z] collected 222 items
[2021-12-02T21:10:25.599Z] 
[2021-12-02T21:10:36.444Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2021-12-02T21:10:55.892Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  8%]
[2021-12-02T21:11:05.128Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2021-12-02T21:11:13.326Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 14%]
[2021-12-02T21:11:16.042Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 18%]
[2021-12-02T21:13:07.580Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb F                 [ 19%]
[2021-12-02T21:13:07.580Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 19%]
[2021-12-02T21:13:12.336Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 22%]
[2021-12-02T21:13:20.367Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 25%]
[2021-12-02T21:16:04.849Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 31%]
[2021-12-02T21:17:03.867Z] ..............                                                           [ 38%]
[2021-12-02T21:17:09.968Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 40%]
[2021-12-02T21:17:11.726Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 43%]
[2021-12-02T21:17:34.555Z] .................                                                        [ 51%]
[2021-12-02T21:17:40.114Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 53%]
[2021-12-02T21:17:41.519Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 54%]
[2021-12-02T21:18:12.773Z] ..F...FFF.                                                               [ 58%]
[2021-12-02T21:18:24.264Z] finch-master/docs/source/notebooks/dap_subset.ipynb ..........           [ 63%]
[2021-12-02T21:18:32.912Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 65%]
[2021-12-02T21:19:16.638Z] finch-master/docs/source/notebooks/subset.ipynb ....................     [ 74%]
[2021-12-02T21:19:17.210Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 75%]
[2021-12-02T21:19:22.236Z] ......                                                                   [ 77%]
[2021-12-02T21:20:18.506Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 78%]
[2021-12-02T21:20:40.083Z] .............                                                            [ 84%]
[2021-12-02T21:20:50.064Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 84%]
[2021-12-02T21:22:01.062Z] ....s.                                                                   [ 87%]
[2021-12-02T21:22:11.065Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 87%]
[2021-12-02T21:22:17.874Z] ...                                                                      [ 89%]
[2021-12-02T21:22:36.017Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 89%]
[2021-12-02T21:23:01.597Z] ......                                                                   [ 92%]
[2021-12-02T21:23:07.702Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2021-12-02T21:26:32.775Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2021-12-02T21:26:32.775Z] 
[2021-12-02T21:26:32.775Z] =================================== FAILURES ===================================

fmigneault · 2021-12-02T21:47:51Z

closes #197

fmigneault · 2021-12-03T02:19:23Z

run tests

crim-jenkins-bot · 2021-12-03T02:44:08Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/793/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : magpie-integrate-new-feats
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-88.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/676/

NOTEBOOK TEST RESULTS

    
[2021-12-03T02:27:49.456Z] ============================= test session starts ==============================
[2021-12-03T02:27:49.456Z] platform linux -- Python 3.7.10, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
[2021-12-03T02:27:49.456Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2021-12-03T02:27:49.456Z] plugins: anyio-3.3.0, dash-1.21.0, nbval-0.9.6, tornasync-0.6.0.post2
[2021-12-03T02:27:49.456Z] collected 222 items
[2021-12-03T02:27:49.456Z] 
[2021-12-03T02:27:58.279Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2021-12-03T02:28:41.891Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  8%]
[2021-12-03T02:28:51.462Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2021-12-03T02:29:03.921Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 14%]
[2021-12-03T02:29:05.963Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 18%]
[2021-12-03T02:30:57.487Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb F                 [ 19%]
[2021-12-03T02:30:57.487Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 19%]
[2021-12-03T02:31:02.564Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 22%]
[2021-12-03T02:31:09.443Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 25%]
[2021-12-03T02:33:36.952Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 31%]
[2021-12-03T02:34:30.600Z] ..............                                                           [ 38%]
[2021-12-03T02:34:36.426Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 40%]
[2021-12-03T02:34:38.355Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 43%]
[2021-12-03T02:35:02.458Z] .................                                                        [ 51%]
[2021-12-03T02:35:08.638Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 53%]
[2021-12-03T02:35:10.018Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 54%]
[2021-12-03T02:35:20.595Z] .......FF.                                                               [ 58%]
[2021-12-03T02:35:31.265Z] finch-master/docs/source/notebooks/dap_subset.ipynb ..........           [ 63%]
[2021-12-03T02:35:40.119Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 65%]
[2021-12-03T02:36:24.085Z] finch-master/docs/source/notebooks/subset.ipynb ....................     [ 74%]
[2021-12-03T02:36:24.085Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 75%]
[2021-12-03T02:36:27.905Z] ......                                                                   [ 77%]
[2021-12-03T02:37:24.201Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 78%]
[2021-12-03T02:37:47.203Z] .............                                                            [ 84%]
[2021-12-03T02:37:57.187Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 84%]
[2021-12-03T02:38:51.595Z] ....s.                                                                   [ 87%]
[2021-12-03T02:39:01.576Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 87%]
[2021-12-03T02:39:08.318Z] ...                                                                      [ 89%]
[2021-12-03T02:39:26.430Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 89%]
[2021-12-03T02:39:58.031Z] ......                                                                   [ 92%]
[2021-12-03T02:40:05.961Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2021-12-03T02:43:10.838Z] notebooks/stress-tests.ipynb ...F.                                       [100%]
[2021-12-03T02:43:10.838Z] 
[2021-12-03T02:43:10.838Z] =================================== FAILURES ===================================

fmigneault · 2021-12-03T03:36:52Z

@ChaamC
@ahandan
Will need to investigate.
Last stress-test had a failure but the previous one didn't.
Seems to be sporadic again, or insufficient amount of requests.

fmigneault · 2021-12-03T15:58:25Z

run tests

crim-jenkins-bot · 2021-12-03T16:22:40Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/794/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : magpie-integrate-new-feats
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-88.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/677/

NOTEBOOK TEST RESULTS

    
[2021-12-03T16:06:29.982Z] ============================= test session starts ==============================
[2021-12-03T16:06:29.983Z] platform linux -- Python 3.7.10, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
[2021-12-03T16:06:29.983Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2021-12-03T16:06:29.983Z] plugins: anyio-3.3.0, dash-1.21.0, nbval-0.9.6, tornasync-0.6.0.post2
[2021-12-03T16:06:29.983Z] collected 222 items
[2021-12-03T16:06:29.983Z] 
[2021-12-03T16:06:40.605Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2021-12-03T16:07:07.352Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  8%]
[2021-12-03T16:07:18.509Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2021-12-03T16:07:27.190Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 14%]
[2021-12-03T16:07:29.668Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 18%]
[2021-12-03T16:07:44.607Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb .                 [ 19%]
[2021-12-03T16:07:46.010Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 19%]
[2021-12-03T16:07:51.438Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 22%]
[2021-12-03T16:07:57.913Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 25%]
[2021-12-03T16:11:30.233Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 31%]
[2021-12-03T16:12:22.218Z] ..............                                                           [ 38%]
[2021-12-03T16:12:28.061Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 40%]
[2021-12-03T16:12:30.404Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 43%]
[2021-12-03T16:12:59.417Z] .................                                                        [ 51%]
[2021-12-03T16:13:05.923Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 53%]
[2021-12-03T16:13:07.854Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 54%]
[2021-12-03T16:13:10.029Z] .FFFFFFFF.                                                               [ 58%]
[2021-12-03T16:13:21.595Z] finch-master/docs/source/notebooks/dap_subset.ipynb ..........           [ 63%]
[2021-12-03T16:13:30.418Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 65%]
[2021-12-03T16:14:13.607Z] finch-master/docs/source/notebooks/subset.ipynb ....................     [ 74%]
[2021-12-03T16:14:15.543Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 75%]
[2021-12-03T16:14:20.543Z] ......                                                                   [ 77%]
[2021-12-03T16:14:47.151Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 78%]
[2021-12-03T16:15:11.748Z] .............                                                            [ 84%]
[2021-12-03T16:15:21.766Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 84%]
[2021-12-03T16:17:18.795Z] ....s.                                                                   [ 87%]
[2021-12-03T16:17:31.042Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 87%]
[2021-12-03T16:17:38.869Z] ...                                                                      [ 89%]
[2021-12-03T16:18:05.476Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 89%]
[2021-12-03T16:18:34.187Z] ......                                                                   [ 92%]
[2021-12-03T16:18:38.576Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2021-12-03T16:21:43.596Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2021-12-03T16:21:43.596Z] 
[2021-12-03T16:21:43.596Z] =================================== FAILURES ===================================

fmigneault · 2021-12-03T16:45:01Z

@ChaamC
I have investigated and the error doesn't seem to be the usual NoTransaction or DetachedInstanceError.
The logs have too few details to know the deeper cause that raised.
You can move on with your PR. I will add more logs and re-test/create a new PR when if I can replicate/fix.

For later reference, this was the error:
http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/676/

2021-12-03 02:41:59,675 INFO  [TWITCHER:57][ThreadPoolExecutor-0_3] Using adapter: '<class 'magpie.adapter.MagpieAdapter'>'
twitcher           | 2021-12-03 02:41:59,678 INFO  [TWITCHER|magpie.adapter.magpieowssecurity:216][ThreadPoolExecutor-0_1] User None is requesting 'getcapabilities' permission on [/ows/proxy/hummingbird/wps]
twitcher           | 2021-12-03 02:41:59,678 INFO  [TWITCHER:57][ThreadPoolExecutor-0_1] Using adapter: '<class 'magpie.adapter.MagpieAdapter'>'
twitcher           | 2021-12-03 02:41:59,679 WARNI [TWITCHER|magpie.adapter.magpieowssecurity:118][ThreadPoolExecutor-0_1] Using cached service
twitcher           | 2021-12-03 02:41:59,680 INFO  [TWITCHER|magpie.adapter.magpieowssecurity:216][ThreadPoolExecutor-0_1] User None is requesting 'getcapabilities' permission on [/ows/proxy/hummingbird/wps]
twitcher           | 2021-12-03 02:41:59,692 INFO  [TWITCHER|magpie.adapter.magpieowssecurity:231][ThreadPoolExecutor-0_1] User None resolved with allowed 'getcapabilities' access to [/ows/proxy/hummingbird/wps]
twitcher           | 2021-12-03 02:41:59,686 ERROR [TWITCHER|magpie.adapter.magpieowssecurity:152][ThreadPoolExecutor-0_3] Unhandled exception. Derived OWSException is expected for unauthorized access.
twitcher           | Traceback (most recent call last):
twitcher           |   File "/opt/local/src/magpie/magpie/adapter/magpieowssecurity.py", line 147, in verify_request
twitcher           |     self.check_request(request)
twitcher           |   File "/opt/local/src/magpie/magpie/adapter/magpieowssecurity.py", line 183, in check_request
twitcher           |     service_impl = self.get_service(request)
twitcher           |   File "/opt/local/src/magpie/magpie/adapter/magpieowssecurity.py", line 109, in get_service
twitcher           |     service_impl, service_data = self._get_service_cached(service_name)
twitcher           |   File "/usr/local/lib/python3.7/site-packages/beaker/cache.py", line 601, in cached
twitcher           |     return cache[0].get_value(cache_key, createfunc=go)
twitcher           |   File "/usr/local/lib/python3.7/site-packages/beaker/cache.py", line 322, in get
twitcher           |     return self._get_value(key, **kw).get_value()
twitcher           |   File "/usr/local/lib/python3.7/site-packages/beaker/container.py", line 380, in get_value
twitcher           |     v = self.createfunc()
twitcher           |   File "/usr/local/lib/python3.7/site-packages/beaker/cache.py", line 597, in go
twitcher           |     return func(*args, **kwargs)
twitcher           |   File "/opt/local/src/magpie/magpie/adapter/magpieowssecurity.py", line 85, in _get_service_cached
twitcher           |     service_impl = service_factory(service, self._cached_request)
twitcher           |   File "/opt/local/src/magpie/magpie/services.py", line 984, in service_factory
twitcher           |     msg_on_fail="Failed to find requested service type.")
twitcher           |   File "/opt/local/src/magpie/magpie/api/exception.py", line 347, in evaluate_call
twitcher           |     content={"call": exc_call}, content_type=content_type)
twitcher           |   File "/opt/local/src/magpie/magpie/api/exception.py", line 426, in raise_http
twitcher           |     raise resp
twitcher           | pyramid.httpexceptions.HTTPInternalServerError: The server has either erred or is incapable of performing the requested operation.
twitcher           | 2021-12-03 02:41:59,698 WARNI [TWITCHER:172][ThreadPoolExecutor-0_3] Security check failed but was not handled as expected by 'is_verified' method.
twitcher           | Traceback (most recent call last):
twitcher           |   File "/opt/birdhouse/src/twitcher/twitcher/owsproxy.py", line 169, in owsproxy_view
twitcher           |     raise OWSAccessForbidden("Access to service is forbidden.")
twitcher           | twitcher.owsexceptions.OWSAccessForbidden: Access to service is forbidden.

Error raised when calling:

service_factory(service, self._cached_request)
# ...
SERVICE_TYPE_DICT[service_type](service, request)

Service object creation fails for some obscure reason.

…d-house/birdhouse-deploy#224 (comment))

crim-jenkins-bot · 2021-12-03T17:30:42Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/795/
Result : success

BIRDHOUSE_DEPLOY_BRANCH : magpie-integrate-new-feats
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-88.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/678/

NOTEBOOK TEST RESULTS

    
[2021-12-03T17:16:33.090Z] ============================= test session starts ==============================
[2021-12-03T17:16:33.090Z] platform linux -- Python 3.7.10, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
[2021-12-03T17:16:33.090Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2021-12-03T17:16:33.090Z] plugins: anyio-3.3.0, dash-1.21.0, nbval-0.9.6, tornasync-0.6.0.post2
[2021-12-03T17:16:33.090Z] collected 222 items
[2021-12-03T17:16:33.090Z] 
[2021-12-03T17:16:41.516Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2021-12-03T17:17:01.998Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  8%]
[2021-12-03T17:17:12.095Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2021-12-03T17:17:21.786Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 14%]
[2021-12-03T17:17:24.322Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 18%]
[2021-12-03T17:17:36.552Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb .                 [ 19%]
[2021-12-03T17:17:37.496Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 19%]
[2021-12-03T17:17:43.429Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 22%]
[2021-12-03T17:17:50.685Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 25%]
[2021-12-03T17:20:34.182Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 31%]
[2021-12-03T17:21:26.675Z] ..............                                                           [ 38%]
[2021-12-03T17:21:33.424Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 40%]
[2021-12-03T17:21:35.657Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 43%]
[2021-12-03T17:21:57.816Z] .................                                                        [ 51%]
[2021-12-03T17:22:03.994Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 53%]
[2021-12-03T17:22:05.387Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 54%]
[2021-12-03T17:22:15.890Z] ..........                                                               [ 58%]
[2021-12-03T17:22:27.459Z] finch-master/docs/source/notebooks/dap_subset.ipynb ..........           [ 63%]
[2021-12-03T17:22:36.115Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 65%]
[2021-12-03T17:23:16.471Z] finch-master/docs/source/notebooks/subset.ipynb ....................     [ 74%]
[2021-12-03T17:23:17.410Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 75%]
[2021-12-03T17:23:22.148Z] ......                                                                   [ 77%]
[2021-12-03T17:24:18.437Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 78%]
[2021-12-03T17:24:37.396Z] .............                                                            [ 84%]
[2021-12-03T17:24:49.635Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 84%]
[2021-12-03T17:25:44.525Z] ....s.                                                                   [ 87%]
[2021-12-03T17:25:52.667Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 87%]
[2021-12-03T17:25:59.891Z] ...                                                                      [ 89%]
[2021-12-03T17:26:14.802Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 89%]
[2021-12-03T17:26:41.518Z] ......                                                                   [ 92%]
[2021-12-03T17:26:49.651Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2021-12-03T17:29:51.047Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2021-12-03T17:29:51.047Z] 
[2021-12-03T17:29:51.048Z] ================== 221 passed, 1 skipped in 798.93s (0:13:18) ==================

tlvu · 2021-12-07T14:48:13Z

@fmigneault @ChaamC FYI Ouranos will deploy this new version of Magpie but without activating the cache feature because with the cache enabled on our staging, stress-test.ipynb fails again:

14:32:15  _____________________ notebooks/stress-tests.ipynb::Cell 3 _____________________
14:32:15  Notebook cell execution failed
14:32:15  Cell 3: Cell execution caused an exception
14:32:15  
14:32:15  Input:
14:32:15  # NBVAL_IGNORE_OUTPUT
14:32:15  
14:32:15  # Executing WPS test on multiple threads
14:32:15  wps_urls = [
14:32:15      f"{TWITCHER_URL}/{bird}/wps?service=wps&request=getcapabilities"
14:32:15      for bird in TEST_WPS_BIRDS
14:32:15  ]
14:32:15  
14:32:15  for execution, url in enumerate(wps_urls):
14:32:15      run_threads(target_test = stress_test,
14:32:15                  url = url,
14:32:15                  execution_id = execution,
14:32:15                  n_threads = TEST_N_THREADS_WPS,
14:32:15                  runs_per_threads = TEST_RUNS_WPS)
14:32:15  
14:32:15  Traceback:
14:32:15  
14:32:15  ---------------------------------------------------------------------------
14:32:15  AssertionError                            Traceback (most recent call last)
14:32:15  /tmp/ipykernel_1302/3472822828.py in <module>
14:32:15       12                 execution_id = execution,
14:32:15       13                 n_threads = TEST_N_THREADS_WPS,
14:32:15  ---> 14                 runs_per_threads = TEST_RUNS_WPS)
14:32:15  
14:32:15  /tmp/ipykernel_1302/1081959450.py in run_threads(target_test, url, execution_id, n_threads, runs_per_threads)
14:32:15       37     assert (
14:32:15       38         stress_test_progression.failed_threads_count == 0
14:32:15  ---> 39     ), stress_test_progression.fail_report()
14:32:15       40 
14:32:15       41     print(f"All threads passed {target_test.__name__} function.")
14:32:15  
14:32:15  AssertionError: This test failed due to bad code received from the requests.
14:32:15   Number of failed thread(s) : 1/ 3
14:32:15   Number of failed requests : 1/300
14:32:15  
14:32:15  
14:32:15   Failed Test
14:32:15   Stress Test:
14:32:15      Test:
14:32:15          code: 200
14:32:15          runs: 100
14:32:15          max-avg-time:  1s
14:32:15          max-err-code:  0
14:32:15          sum-err-code:  1
14:32:15          timeout-abort: 0s
14:32:15          timeout-retry: 0
14:32:15          timeout-count: 1
14:32:15      Request:
14:32:15          method: GET
14:32:15          url:    https://medus.ouranos.ca/twitcher/ows/proxy/hummingbird/wps?service=wps&request=getcapabilities
14:32:15          args:   {}
14:32:15      Times:
14:32:15          min: 5.000s
14:32:15          avg: 0.050s
14:32:15          max: 5.000s
14:32:15      Results:
14:32:15          Run                 Codes                 Delta                 Times            Timestamps
14:32:15            1               (!) 408                0.066s                5.000s   2021-12-07 14:29:55
14:32:15      Summary:
14:32:15          Detected 1 erroneous HTTP codes not equal to expected 200.
14:32:15          Test failed (status=-1).
14:32:15  
14:32:15   === Exceptions ===
14:32:15  
14:32:15   1. 408 Request TimeOut (Client Side)
14:32:15   HTTPSConnectionPool(host='medus.ouranos.ca', port=443): Read timed out. (read timeout=5)

Edit:

there was not matching stack-trace in docker logs twitcher

fmigneault · 2021-12-07T15:35:12Z

@tlvu

I believe that what you are seeing might be real client timeout, completely unrelated to caching.
Following are the 2 errors that are direct cause of caching.
Notice that the status codes are different (not related to timeout) and the Time is not at the max value of 5s.

Run	Codes	Delta	Times	Timestamps
15:58:29	(!) 500	0.073s	0.047s	2021-11-25 20:58:10
21:43:10	(!) 401	0.000s	0.102s	2021-12-03 02:41:59

I'm just throwing a wild guess here, but it could be possible that because of caching, requests/sec become much higher since each one takes less time to complete, and the server sometimes might not be able to take this heavier load. This could be a cause of sporadic timeout. The fact that you cannot find logs in twitcher also works with this hypothesis. That means the request never reaches twitcher/magpie caching and the problem is at the proxy/nginx level.

tlvu · 2021-12-07T16:30:48Z

Houston, we have a bigger problem here. stress-test.ipynb failure on our production (no caching enabled) with code 401.

10:46:02  _____________________ notebooks/stress-tests.ipynb::Cell 3 _____________________
10:46:02  Notebook cell execution failed
10:46:02  Cell 3: Cell execution caused an exception
10:46:02  
10:46:02  Input:
10:46:02  # NBVAL_IGNORE_OUTPUT
10:46:02  
10:46:02  # Executing WPS test on multiple threads
10:46:02  wps_urls = [
10:46:02      f"{TWITCHER_URL}/{bird}/wps?service=wps&request=getcapabilities"
10:46:02      for bird in TEST_WPS_BIRDS
10:46:02  ]
10:46:02  
10:46:02  for execution, url in enumerate(wps_urls):
10:46:02      run_threads(target_test = stress_test,
10:46:02                  url = url,
10:46:02                  execution_id = execution,
10:46:02                  n_threads = TEST_N_THREADS_WPS,
10:46:02                  runs_per_threads = TEST_RUNS_WPS)
10:46:02  
10:46:02  Traceback:
10:46:02  
10:46:02  ---------------------------------------------------------------------------
10:46:02  AssertionError                            Traceback (most recent call last)
10:46:02  /tmp/ipykernel_646/3472822828.py in <module>
10:46:02       12                 execution_id = execution,
10:46:02       13                 n_threads = TEST_N_THREADS_WPS,
10:46:02  ---> 14                 runs_per_threads = TEST_RUNS_WPS)
10:46:02  
10:46:02  /tmp/ipykernel_646/1081959450.py in run_threads(target_test, url, execution_id, n_threads, runs_per_threads)
10:46:02       37     assert (
10:46:02       38         stress_test_progression.failed_threads_count == 0
10:46:02  ---> 39     ), stress_test_progression.fail_report()
10:46:02       40 
10:46:02       41     print(f"All threads passed {target_test.__name__} function.")
10:46:02  
10:46:02  AssertionError: This test failed due to bad code received from the requests.
10:46:02   Number of failed thread(s) : 1/ 3
10:46:02   Number of failed requests : 1/300
10:46:02  
10:46:02  
10:46:02   Failed Test
10:46:02   Stress Test:
10:46:02      Test:
10:46:02          code: 200
10:46:02          runs: 100
10:46:02          max-avg-time:  1s
10:46:02          max-err-code:  0
10:46:02          sum-err-code:  1
10:46:02          timeout-abort: 0s
10:46:02          timeout-retry: 0
10:46:02          timeout-count: 0
10:46:02      Request:
10:46:02          method: GET
10:46:02          url:    https://pavics.ouranos.ca/twitcher/ows/proxy/hummingbird/wps?service=wps&request=getcapabilities
10:46:02          args:   {}
10:46:02      Times:
10:46:02          min: 0.122s
10:46:02          avg: 0.001s
10:46:02          max: 0.122s
10:46:02      Results:
10:46:02          Run                 Codes                 Delta                 Times            Timestamps
10:46:02            1               (!) 401                0.035s                0.122s   2021-12-07 15:44:40
10:46:02      Summary:
10:46:02          Detected 1 erroneous HTTP codes not equal to expected 200.
10:46:02          Test failed (status=-1).
10:46:02  
10:46:02   === Exceptions ===
10:46:02  
10:46:02   1. 401 Unauthorized
10:46:02   <?xml version="1.0" encoding="utf-8"?>
10:46:02  <ExceptionReport version="1.0.0"
10:46:02      xmlns="http://www.opengis.net/ows/1.1"
10:46:02      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
10:46:02      xsi:schemaLocation="http://www.opengis.net/ows/1.1 http://schemas.opengis.net/ows/1.1.0/owsExceptionReport.xsd">
10:46:02      <Exception exceptionCode="NoApplicableCode" locator="AccessForbidden">
10:46:02          <ExceptionText>Access to service is forbidden.</ExceptionText>
10:46:02      </Exception>
10:46:02  </ExceptionReport>

tlvu · 2021-12-07T16:32:22Z

Please advice rollback procedure or if hotfix is possible. It's intermittent again, Happens on my 3rd Jenkins run only.

tlvu · 2021-12-07T16:37:57Z

docker logs twitcher:

I got many of those, no other stack-trace:

2021-12-07 15:15:42,471 WARNI [TWITCHER:172][ThreadPoolExecutor-0_1] Security check failed but was not handled as expected by 'is_verified' method.
Traceback (most recent call last):
  File "/opt/birdhouse/src/twitcher/twitcher/owsproxy.py", line 169, in owsproxy_view
    raise OWSAccessForbidden("Access to service is forbidden.")

…vice completely finishes computing the object (relates to #490 and bird-house/birdhouse-deploy#224 (comment))

fmigneault · 2021-12-07T18:13:44Z

@tlvu
I'm aware of one case that should be fixed by Ouranosinc/Magpie#491 and testing in #226.
Not sure if the one you're facing is the same.

Note that unless you enable debug logs on some instance, most of the time this stack-trace means nothing because it is a catch-all for any exception (including valid unauthorized/forbidden access).

tlvu · 2021-12-07T18:29:43Z

It looks like as soon as the error started to happen, it happens more regularly. See screenshot of consecutive failure after the first failure in prod.

tlvu · 2021-12-07T18:32:48Z

I'm aware of one case that should be fixed by Ouranosinc/Magpie#491 and testing in #226.
Not sure if the one you're facing is the same.

Do you suggest waiting for that fix or rollback now? What would be the rollback procedure?

Note that unless you enable debug logs on some instance

It's our prod instance, I'd rather not enable the verbose debug logging.

tlvu · 2021-12-07T18:34:53Z

I think I'll move all the findings to a new Magpie issue. Commenting on a closed PR do not have much visibility.

tlvu · 2021-12-07T18:38:43Z

Ouranosinc/Magpie#492

## Overview Pin new `magpie==3.19.1` image to fix #224 (comment). ## Related Issue / Discussion - Relates to #224 - Relates to Ouranosinc/Magpie#491 - Relates to Ouranosinc/Magpie#492

cwcummings added 4 commits November 19, 2021 09:26

bump magpie version + add new .ini config variables

25fbe10

add environment variables for magpie config

13a02d7

update changelog

f460bf5

fix magpie variable name

5a0751b

ChaamC requested review from fmigneault and tlvu November 19, 2021 18:37

fmigneault requested changes Nov 22, 2021

View reviewed changes

birdhouse/env.local.example Outdated Show resolved Hide resolved

birdhouse/config/magpie/magpie.ini.template Outdated Show resolved Hide resolved

CHANGES.md Outdated Show resolved Hide resolved

fmigneault reviewed Nov 22, 2021

View reviewed changes

birdhouse/env.local.example Show resolved Hide resolved

fmigneault reviewed Nov 22, 2021

View reviewed changes

birdhouse/env.local.example Show resolved Hide resolved

cwcummings added 2 commits November 22, 2021 16:06

update changelog and add ref links to env.local.example

e024066

update twitcher runner to employ gunicorn instead of waitress

23d1dd7

ChaamC requested a review from fmigneault November 22, 2021 21:15

fmigneault approved these changes Nov 22, 2021

View reviewed changes

tlvu approved these changes Nov 23, 2021

View reviewed changes

birdhouse/config/magpie/magpie.ini.template Show resolved Hide resolved

birdhouse/config/magpie/magpie.ini.template Outdated Show resolved Hide resolved

fmigneault mentioned this pull request Nov 24, 2021

Magpie/Twitcher 3.15.1 + debug logging & gunicorn #197

Closed

magpie config fix: comment for token_expire and now configurable smtp…

f9aa083

…_port and smtp_ssl

bump magpie to version 3.19.0

537a3ae

fmigneault mentioned this pull request Dec 2, 2021

add new wps processes found in finch & update weaver describe responses Ouranosinc/pavics-sdi#239

Merged

fmigneault added a commit to Ouranosinc/Magpie that referenced this pull request Dec 3, 2021

add log entry to investigate failing service creation (relates to bir…

380ed34

…d-house/birdhouse-deploy#224 (comment))

Bump version: 1.17.5 → 1.17.6

6659456

github-actions bot added ci/operations Continuous Integration components documentation Improvements or additions to documentation labels Dec 3, 2021

ChaamC merged commit dfebae7 into master Dec 3, 2021

ChaamC deleted the magpie-integrate-new-feats branch December 3, 2021 17:12

fmigneault added a commit to Ouranosinc/Magpie that referenced this pull request Dec 3, 2021

add log entry to investigate failing service creation (relates to bir…

0d3b4ac

…d-house/birdhouse-deploy#224 (comment))

fmigneault added a commit to Ouranosinc/Magpie that referenced this pull request Dec 3, 2021

add log entry to investigate failing service creation (relates to bir…

4cdf19f

…d-house/birdhouse-deploy#224 (comment))

fmigneault mentioned this pull request Dec 3, 2021

add log entry to investigate failing service creation Ouranosinc/Magpie#490

Merged

fmigneault mentioned this pull request Dec 3, 2021

add failing values in raised error message for debugging Ouranosinc/pavics-sdi#240

Merged

fmigneault added a commit to Ouranosinc/Magpie that referenced this pull request Dec 7, 2021

use uuid/request mapping to persist distinct handles until cached ser…

7535627

…vice completely finishes computing the object (relates to #490 and bird-house/birdhouse-deploy#224 (comment))

This was referenced Dec 7, 2021

fix failing cached service creation Ouranosinc/Magpie#491

Merged

magpie: validate service cache fix #226

Merged

tlvu mentioned this pull request Dec 7, 2021

[BUG] Intermittent failure since 3.19.0 on PAVICS, even without caching enabled Ouranosinc/Magpie#492

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Magpie integrate new feats #224

Magpie integrate new feats #224

ChaamC commented Nov 19, 2021 •

edited

Loading

crim-jenkins-bot commented Nov 19, 2021

tlvu commented Nov 22, 2021

crim-jenkins-bot commented Nov 22, 2021

ChaamC commented Nov 22, 2021

tlvu left a comment

tlvu commented Nov 23, 2021 •

edited

Loading

ChaamC commented Nov 25, 2021

crim-jenkins-bot commented Nov 25, 2021

crim-jenkins-bot commented Dec 2, 2021

fmigneault commented Dec 2, 2021

fmigneault commented Dec 3, 2021

crim-jenkins-bot commented Dec 3, 2021

fmigneault commented Dec 3, 2021

fmigneault commented Dec 3, 2021

crim-jenkins-bot commented Dec 3, 2021

fmigneault commented Dec 3, 2021

crim-jenkins-bot commented Dec 3, 2021

tlvu commented Dec 7, 2021 •

edited

Loading

fmigneault commented Dec 7, 2021 •

edited

Loading

tlvu commented Dec 7, 2021

tlvu commented Dec 7, 2021

tlvu commented Dec 7, 2021

fmigneault commented Dec 7, 2021

tlvu commented Dec 7, 2021

tlvu commented Dec 7, 2021

tlvu commented Dec 7, 2021

tlvu commented Dec 7, 2021

Magpie integrate new feats #224

Magpie integrate new feats #224

Conversation

ChaamC commented Nov 19, 2021 • edited Loading

Overview

Changes

crim-jenkins-bot commented Nov 19, 2021

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

tlvu commented Nov 22, 2021

crim-jenkins-bot commented Nov 22, 2021

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

ChaamC commented Nov 22, 2021

tlvu left a comment

Choose a reason for hiding this comment

tlvu commented Nov 23, 2021 • edited Loading

ChaamC commented Nov 25, 2021

crim-jenkins-bot commented Nov 25, 2021

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Dec 2, 2021

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

fmigneault commented Dec 2, 2021

fmigneault commented Dec 3, 2021

crim-jenkins-bot commented Dec 3, 2021

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

fmigneault commented Dec 3, 2021

fmigneault commented Dec 3, 2021

crim-jenkins-bot commented Dec 3, 2021

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

fmigneault commented Dec 3, 2021

crim-jenkins-bot commented Dec 3, 2021

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

tlvu commented Dec 7, 2021 • edited Loading

fmigneault commented Dec 7, 2021 • edited Loading

tlvu commented Dec 7, 2021

tlvu commented Dec 7, 2021

tlvu commented Dec 7, 2021

fmigneault commented Dec 7, 2021

tlvu commented Dec 7, 2021

tlvu commented Dec 7, 2021

tlvu commented Dec 7, 2021

tlvu commented Dec 7, 2021

ChaamC commented Nov 19, 2021 •

edited

Loading

tlvu commented Nov 23, 2021 •

edited

Loading

tlvu commented Dec 7, 2021 •

edited

Loading

fmigneault commented Dec 7, 2021 •

edited

Loading