has been failing testing in datalad core for abit

[yarikoptic]

from a sample run https://github.com/datalad/datalad/actions/runs/4270688403/jobs/7434642131

=========================== short test summary info ============================
FAILED annexremotes/tests/test_uncurl.py::test_uncurl_checkurl - ValueError: unsupported URL 'garbledhttps://httpbin.org/status/404'
FAILED annexremotes/tests/test_uncurl.py::test_uncurl_addurl_unredirected - datalad.runner.exception.CommandError: CommandError: 'git -c diff.ignoreSubmodules=none annex addurl --file=dummy garbledhttps://httpbin.org/bytes/24 -c annex.dotfiles=true' failed with exitcode 1 under /tmp/pytest-of-runner/pytest-0/test_uncurl_addurl_unredirecte0/ds [out: 'addurl garbledhttps://httpbin.org/bytes/24 
failed'] [err: 'Configuration of annex.security.allowed-url-schemes does not allow accessing garbledhttps://httpbin.org/bytes/24
addurl: 1 failed']
====== 2 failed, 204 passed, 10 skipped, 12 warnings in 492.97s (0:08:12) ======

Seems started to fail on Feb 23rd

(git)smaug:/mnt/datasets/datalad/ci/logs/2023/02[master]git
$> GIT_TERMINAL_PROMPT=0 datalad foreach-dataset --o-s relpath -r -J10 'git grep "annexremotes/tests/test_uncurl.py::test_uncurl_checkurl"' | grep PASS
07/pr/7273/5852580/github-Extensions-6132-success/6_test (datalad-next).txt:2023-02-07T03:45:45.3941139Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
07/pr/7273/5852580/github-Extensions-6132-success/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-07T03:45:45.3940721Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
07/pr/7273/5b40500/github-Extensions-6134-success/6_test (datalad-next).txt:2023-02-07T04:07:46.3473560Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
07/pr/7273/5b40500/github-Extensions-6134-success/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-07T04:07:46.3473203Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
07/pr/7280/9d0607f/github-Extensions-6131-success/6_test (datalad-next).txt:2023-02-07T03:37:00.8808077Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
07/pr/7280/9d0607f/github-Extensions-6131-success/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-07T03:37:00.8807572Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
07/pr/UNK/54b2009/github-Extensions-6130-success/6_test (datalad-next).txt:2023-02-07T03:30:53.2093499Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
07/pr/UNK/54b2009/github-Extensions-6130-success/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-07T03:30:53.2093140Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
07/push/maint/451d402/github-Extensions-6133-success/6_test (datalad-next).txt:2023-02-07T03:59:00.7867688Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
07/push/maint/451d402/github-Extensions-6133-success/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-07T03:59:00.7867294Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
17/pr/7289/6674407/github-Extensions-6135-failed/6_test (datalad-next).txt:2023-02-17T18:34:20.3762825Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
17/pr/7289/6674407/github-Extensions-6135-failed/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-17T18:34:20.3762372Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
21/pr/7271/2340b4c/github-Extensions-6140-failed/6_test (datalad-next).txt:2023-02-21T20:43:52.9272196Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
21/pr/7271/2340b4c/github-Extensions-6140-failed/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-21T20:43:52.9271818Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
21/pr/7292/a6d04eb/github-Extensions-6138-failed/6_test (datalad-next).txt:2023-02-21T17:51:54.5743063Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
21/pr/7292/a6d04eb/github-Extensions-6138-failed/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-21T17:51:54.5742680Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
21/pr/7292/db062bb/github-Extensions-6139-failed/6_test (datalad-next).txt:2023-02-21T17:52:30.0783368Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
21/pr/7292/db062bb/github-Extensions-6139-failed/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-21T17:52:30.0782885Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
21/pr/7293/5983e07/github-Extensions-6143-failed/6_test (datalad-next).txt:2023-02-21T21:00:05.3347304Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
21/pr/7293/5983e07/github-Extensions-6143-failed/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-21T21:00:05.3346843Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
21/push/maint/3cca0db/github-Extensions-6141-failed/6_test (datalad-next).txt:2023-02-21T20:50:20.3239495Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
21/push/maint/3cca0db/github-Extensions-6141-failed/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-21T20:50:20.3238901Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
21/push/maint/6c2a573/github-Extensions-6142-failed/6_test (datalad-next).txt:2023-02-21T20:53:58.8893521Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
21/push/maint/6c2a573/github-Extensions-6142-failed/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-21T20:53:58.8893136Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
22/push/maint/589dd65/github-Extensions-6144-failed/6_test (datalad-next).txt:2023-02-22T14:17:14.2291479Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
22/push/maint/589dd65/github-Extensions-6144-failed/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-22T14:17:14.2290836Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
23/pr/7298/cb86f33/github-Extensions-6145-failed/6_test (datalad-next).txt:2023-02-23T08:21:03.3728395Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
23/pr/7298/cb86f33/github-Extensions-6145-failed/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-23T08:21:03.3728159Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
23/pr/7301/f884340/github-Extensions-6147-failed/6_test (datalad-next).txt:2023-02-23T13:40:54.7131815Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED
23/pr/7301/f884340/github-Extensions-6147-failed/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-23T13:40:54.7131442Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl PASSED

$> GIT_TERMINAL_PROMPT=0 datalad foreach-dataset --o-s relpath -r -J10 'git grep "annexremotes/tests/test_uncurl.py::test_uncurl_checkurl"' | grep FAIL
23/pr/7301/95fd21c/github-Extensions-6146-failed/6_test (datalad-next).txt:2023-02-23T13:37:13.0638685Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl FAILED
23/pr/7301/95fd21c/github-Extensions-6146-failed/6_test (datalad-next).txt:2023-02-23T13:50:18.1610897Z FAILED annexremotes/tests/test_uncurl.py::test_uncurl_checkurl - AssertionError: assert False
23/pr/7301/95fd21c/github-Extensions-6146-failed/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-23T13:37:13.0638162Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl FAILED
23/pr/7301/95fd21c/github-Extensions-6146-failed/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-23T13:50:18.1610893Z FAILED annexremotes/tests/test_uncurl.py::test_uncurl_checkurl - AssertionError: assert False
24/pr/7303/9f6003a/github-Extensions-6148-failed/6_test (datalad-next).txt:2023-02-24T03:38:23.8123241Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl FAILED
24/pr/7303/9f6003a/github-Extensions-6148-failed/6_test (datalad-next).txt:2023-02-24T03:49:08.6291997Z FAILED annexremotes/tests/test_uncurl.py::test_uncurl_checkurl - ValueError: unsupported URL 'garbledhttps://httpbin.org/status/404'
24/pr/7303/9f6003a/github-Extensions-6148-failed/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-24T03:38:23.8122755Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl FAILED
24/pr/7303/9f6003a/github-Extensions-6148-failed/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-24T03:49:08.6291991Z FAILED annexremotes/tests/test_uncurl.py::test_uncurl_checkurl - ValueError: unsupported URL 'garbledhttps://httpbin.org/status/404'
25/pr/7305/8a145bf/github-Extensions-6149-failed/6_test (datalad-next).txt:2023-02-25T02:27:15.6902746Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl FAILED
25/pr/7305/8a145bf/github-Extensions-6149-failed/6_test (datalad-next).txt:2023-02-25T02:35:10.6044091Z FAILED annexremotes/tests/test_uncurl.py::test_uncurl_checkurl - ValueError: unsupported URL 'garbledhttps://httpbin.org/status/404'
25/pr/7305/8a145bf/github-Extensions-6149-failed/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-25T02:27:15.6902293Z annexremotes/tests/test_uncurl.py::test_uncurl_checkurl FAILED
25/pr/7305/8a145bf/github-Extensions-6149-failed/test (datalad-next)/13_datalad-next tests using pytest.txt:2023-02-25T02:35:10.6044073Z FAILED annexremotes/tests/test_uncurl.py::test_uncurl_checkurl - ValueError: unsupported URL 'garbledhttps://httpbin.org/status/404'

[mih]

I will wait and see what test runs in -core do now. I have made a wide range of updates that also cover these tests. One issue is/was that the public httpbin instance is not reliable enough, and when it breaks the tests fail -- in sometimes not so obvious ways.

Within -next I have switched most test runs (incl. github actions) to a docker-based httpbin instance: #236

I guess it would make sense to perform similar actions in the datalad-core test action. If that is the case, this issue should be transferred to datalad-core.

[yarikoptic]

I don't think that datalad-core (or anything else e.g. datalad-extensions, later debian package and any other package) should require special knowledge on setting up test infrastructure for datalad-next.

Here you already needed to duplicate that setup among multiple CIs because of having such knowledge not centralized:

.appveyor.yml:  # cache the docker image for httpbin. in 2023 it has not changed in
.appveyor.yml:  - /home/appveyor/cache/httpbin.dockerimg
.appveyor.yml:  - C:\Users\appveyor\httpbin.dockerimg
.appveyor.yml:  - sh: "[ -n \"$DEPLOY_HTTPBIN\" ] && tools/appveyor/docker-load-httpbin && docker run -p 8765:80 kennethreitz/httpbin &"
.github/workflows/test_crippledfs.yml:    - name: Cache httpbin docker image
.github/workflows/test_crippledfs.yml:        key: httpbin.dockerimg
.github/workflows/test_crippledfs.yml:          ~/cache/httpbin.dockerimg
.github/workflows/test_crippledfs.yml:        # deploy the httpbin image
.github/workflows/test_crippledfs.yml:        tools/appveyor/docker-load-httpbin
.github/workflows/test_crippledfs.yml:        # run httpbin container
.github/workflows/test_crippledfs.yml:        docker run -p 8765:80 kennethreitz/httpbin &

I think that beyond caching of images (good to leave to CI), if desired to make it "non-optional" (e.g. skipped if no container running or some special env var set like we do for ssh testing in core) the actual running of the container should be done by a fixture within datalad-next itself. For an example of that see https://github.com/dandi/dandi-cli/blob/HEAD/dandi/tests/fixtures.py#L342 which does nearly exactly the same for docker-compose setup. Helper env var DANDI_TESTS_PERSIST_DOCKER_COMPOSE also was added to allow for that instance to persist across runs of pytest locally to speed things up, but I guess might not even be really needed in your case since likely you container startup time is small.

[jwodder]

@yarikoptic Should the caching of the Docker image be preserved or not? If so, how should that work when running the tests locally?

[jwodder]

@yarikoptic Also, the tests in datalad-next can run with or without the Docker container running; what should determine whether the tests use the container?

[yarikoptic]

@yarikoptic Should the caching of the Docker image be preserved or not?

yes

If so, how should that work when running the tests locally?

pretty much just move docker run -p 8765:80 kennethreitz/httpbin invocation into the fixture. So if ran locally, if image is present - it would use it, if not - fetch it. CIs would do their dance with caching.

@yarikoptic Also, the tests in datalad-next can run with or without the Docker container running; what should determine whether the tests use the container?

that container service is used within get_httpbin_urls. So I think just RF that into httpbin fixture which would return that dictionary of get_httpbin_urls?

[jwodder]

@yarikoptic

• Do you mean that downloading the image to ~/cache/httpbin.dockerimg should happen solely as part of the CI rather than the fixture?
• get_httpbin_urls currently tests whether the Docker container is up and returns a different set of URLs depending on the result; hence, the tests can run either with the container or without the container. When should the fixture use a container and when not?

[yarikoptic]

• Do you mean that downloading the image to ~/cache/httpbin.dockerimg should happen solely as part of the CI rather than the fixture?

yes.

• get_httpbin_urls currently tests whether the Docker container is up and returns a different set of URLs depending on the result; hence, the tests can run either with the container or without the container. When should the fixture use a container and when not?

ah, missed that. I think @mih would not mind consistency:

• always try to start using docker
• if docker one doesn't work (e.g. no docker or just fails to start) -- issue a warning and continue with public instance.

[yarikoptic]

with that in mind -- going back to original issue, @mih it is not clear to me why docker vs public instance matters here?

it passes on docker but not on public instance

❯ python -m pytest --tb=line --disable-warnings datalad_next/annexremotes/tests/test_uncurl.py::test_uncurl_checkurl
================================================================================ test session starts ================================================================================
platform linux -- Python 3.11.1, pytest-7.2.2, pluggy-1.0.0
rootdir: /home/yoh/proj/datalad/datalad-next
plugins: cov-4.0.0
collected 1 item                                                                                                                                                                    

datalad_next/annexremotes/tests/test_uncurl.py F                                                                                                                              [100%]

===================================================================================== FAILURES ======================================================================================
/home/yoh/proj/datalad/datalad-next/datalad_next/url_operations/any.py:136: ValueError: unsupported URL 'garbledhttps://httpbin.org/status/404'
============================================================================== short test summary info ==============================================================================
FAILED datalad_next/annexremotes/tests/test_uncurl.py::test_uncurl_checkurl - ValueError: unsupported URL 'garbledhttps://httpbin.org/status/404'
=========================================================================== 1 failed, 3 warnings in 0.12s ===========================================================================
❯ docker run -p 8765:80 kennethreitz/httpbin &
[2] 631147
 ~datalad/datalad-next  main ?3 ▓▒░──────────────────────────────────────────────────────────────────────────────────────────────────────────────────░▒▓ ≡  dev3.11 Py  02:48:21 PM 
❯ [2023-03-10 19:48:21 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2023-03-10 19:48:21 +0000] [1] [INFO] Listening at: http://0.0.0.0:80 (1)
[2023-03-10 19:48:21 +0000] [1] [INFO] Using worker: gevent
❯ python -m pytest --tb=line --disable-warnings datalad_next/annexremotes/tests/test_uncurl.py::test_uncurl_checkurl
================================================================================ test session starts ================================================================================
platform linux -- Python 3.11.1, pytest-7.2.2, pluggy-1.0.0
rootdir: /home/yoh/proj/datalad/datalad-next
plugins: cov-4.0.0
collected 1 item                                                                                                                                                                    

datalad_next/annexremotes/tests/test_uncurl.py .                                                                                                                              [100%]

=========================================================================== 1 passed, 3 warnings in 0.17s ===========================================================================
❯ git grep garbled
datalad_next/annexremotes/tests/test_uncurl.py:    assert not r.checkurl(f'garbled{hbsurl}/status/404')
datalad_next/annexremotes/tests/test_uncurl.py:    assert r.checkurl(f'garbled{hbsurl}/bytes/24')
datalad_next/annexremotes/tests/test_uncurl.py:    testurl = f'garbled{hbsurl}/bytes/24'

ok -- it is because with

# this is the URL against which the httpbin calls will be made
hbsurl = get_httpbin_urls()['standard']

you get

standard=ciurl if ci_httpbin else hbsurl,

so https for public and http for local. Proposing fix in #312

I still think getting fixture to start docker instance would be good since would harmonize testing here and concentrate logic on deciding which port in that fixture.