Increase visibility of allowed failures (#4584)

* try to warn on allowed failure

* allow failures on upstream-dev

* install pynio to intentionally fail the upstream-dev CI

* install pynio before uninstalling other packages

* fail a flaky test instead of installing a incompatible library

* try changing to error

* try setting the result

* Update ci/azure/unit-tests.yml

Co-authored-by: Mathias Hauser <mathause@users.noreply.github.com>

* remove the intentionally failed test

* don't use the escape code for '#'

Co-authored-by: Mathias Hauser <mathause@users.noreply.github.com>

Author: keewis

azure-pipelines.yml

@@ -28,6 +28,7 @@ jobs:
       py38-upstream-dev:
         conda_env: py38
         upstream_dev: true
+        allow_failure: true
       py38-flaky:
         conda_env: py38
         pytest_extra_flags: --run-flaky --run-network-tests

ci/azure/unit-tests.yml

@@ -15,7 +15,12 @@ steps:
     --junitxml=junit/test-results.xml \
     --cov=xarray \
     --cov-report=xml \
-    $(pytest_extra_flags) || [ "$ALLOW_FAILURE" = "true" ]
+    $(pytest_extra_flags) \
+    || ( \
+      [ "$ALLOW_FAILURE" = "true" ] \
+      && echo "##vso[task.logissue type=warning]Allowed CI failure!!" \
+      && echo "##vso[task.complete result=SucceededWithIssues]" \
+    )
   displayName: Run tests
 
 - bash: |