flaky API tests

[individual-it]

• [ ✔️ ] apiSharingNgShareInvitation/shareInvitations.feature:186

  • Scenario Outline: send share invitation to group with different roles
  • Error: Share mountpoint 'FolderToShare' was not found in the drives list.
  • CI: https://ci.opencloud.eu/repos/3/pipeline/1030/230#L494
  • Fixed: apiTest-flaky: sync share before checking opencloud#1550

• [ ✔️ ] apiSharingNgShareInvitation/shareInvitations.feature:3111

  • Scenario: share a file to user and group having same name (Personal space)
  • Error: Share mountpoint 'textfile.txt' was not found in the drives list.
    Failed asserting that false is identical to true.
  • CI: https://ci.opencloud.eu/repos/3/pipeline/181/258
  • Fixed: apiTest-flaky: sync share before checking opencloud#1550

• [ ✔️ ] coreApiWebdavUploadTUS/lowLevelUpload.feature:58

  • Scenario Outline: send last chunk twice
  • Error: HTTP status code 409 is not the expected value 404
  • CI: https://ci.opencloud.eu/repos/3/pipeline/2944/396
  • Fixed: chore: bump reva opencloud#1701, [full-ci][tests-only] test: add test to check mismatch offset during TUS upload opencloud#1993

• [ ✔️ ] coreApiShareOperationsToShares1/accessToShare.feature:103

  • Scenario Outline: sharee can see the group share
  • Error: Share id 4abc416d-eb9d-42d5-bd7b-9b8b026402d5:f8da51ef-725a-4372-bef1-04ccfa1e2cd8:df783cac-ec0f-4408-8f89-57c67f7ff0b4 not found in response
  • CI: https://ci.opencloud.eu/repos/3/pipeline/4/398
  • Fixed: [full-ci] deleted unused step definitions opencloud#1639

• [ ✔️ ] coreApiShareOperationsToShares2/uploadToShare.feature:28

  • Scenario Outline: uploading file to a user read-only share folder does not work
  • Error: HTTP status code 409 is not the expected value 403
  • CI: https://ci.opencloud.eu/repos/3/pipeline/117/396
  • Fixed: reva-bump-2.39.2. update opencloud 4.0.0-rc.1 opencloud#1849

• [ ✔️ ] apiAntivirus/antivirus.feature:227

  • Scenario Outline: upload a file with virus to a project space
  • Error: Response doesn't contain notification:
  • CI: https://ci.opencloud.rocks/repos/3/pipeline/265/70
  • Fixed: [full-ci][tests-only] test: fix some test flakiness opencloud#2003

• [ ❌ ] apiSharingNg1/sharedWithMe.feature:5035

  • Scenarios:
    • sharee lists the shares after hiding (Personal space)
    • sharee lists the shares after hiding (Project space)
  • Error:

    JSON Schema validation failed:
      1. properties.value.items.0.properties.@UI.Hidden:
      	 - Const failed
      	   Expected: 1
      	   Received:
    

  • CI: https://ci.opencloud.rocks/repos/3/pipeline/266/155

• [ ✔️ ] apiSearch2/tagSearch.feature:254

  • Scenario Outline: search resources using different search patterns (KQL feature)
  • Error: Response doesn't contain notification:
  • CI: https://ci.opencloud.rocks/repos/3/pipeline/322/133

    Failed step: Given user "Alice" has been created with default attributes
        Provisioning::userHasBeenCreated cannot create user 'Alice'.
        Response:[]
        Failed asserting that 401 matches expected 201.

  • Fixed: [full-ci][tests-only] test: fix some test flakiness opencloud#2003

• collaborativePosix: 🔥

  • [ ❌ ] collaborativePosixFS.feature:55, collaborativePosixFS.feature:103
    • Scenario: edit file, move file to folder
    • Step: Then the content of file "/test.txt" for user "Alice" should be "contentnew"
    • Error: The content was expected to be 'contentnew', but actually is 'new'. HTTP status was 200
      • Initial content upload fails. see comment
    • CI:
      • https://ci.opencloud.rocks/repos/3/pipeline/183/89
      • https://ci.opencloud.rocks/repos/3/pipeline/179/90
      • https://ci.opencloud.rocks/repos/3/pipeline/265/88
  • [ ✔️ ] collaborativePosixFS.feature:136
    • Scenario: user doesn't lose file versions after renaming the file
    • Step: And the number of versions should be "2"
    • Error: Expected number of versions was '2', but got '3'
    • CI: https://ci.opencloud.rocks/repos/3/pipeline/176/89
    • Fixed: [full-ci][tests-only] test: fix some test flakiness opencloud#2003

[butonic]

I ran into these flaky tests whn rebasing opencloud-eu/opencloud#1622 today in https://ci.opencloud.eu/repos/3/pipeline/79/88:

collaborativePosix/collaborativePosixFS.feature:23
collaborativePosix/collaborativePosixFS.feature:144

  Scenario: create nested folder                                                                                        # /woodpecker/src/github.com/opencloud-eu/opencloud/tests/acceptance/features/collaborativePosix/collaborativePosixFS.feature:23
    When the administrator creates the folder "deep/nested/structure/myFolder" for user "Alice" on the POSIX filesystem # CliContext::theAdministratorCreatesFolder()
    Then the command should be successful                                                                               # CliContext::theCommandShouldBeSuccessful()
    And as "Alice" folder "/deep/nested/structure/myFolder" should exist                                                # FeatureContext::asFileOrFolderShouldExist()
      folder '/deep/nested/structure/myFolder' expected to exist for user Alice but not found
      Failed asserting that false is true.

and

  Scenario: user doesn't lose share and public link after renaming the file                                                                             # /woodpecker/src/github.com/opencloud-eu/opencloud/tests/acceptance/features/collaborativePosix/collaborativePosixFS.feature:144
    Given user "Brian" has been created with default attributes                                                                                         # FeatureContext::userHasBeenCreatedWithDefaultAttributes()
    And user "Alice" has uploaded file with content "content" to "textfile.txt"                                                                         # FeatureContext::userHasUploadedAFileWithContentTo()
    And user "Alice" has created the following resource link share:                                                                                     # SharingNgContext::userHasCreatedTheFollowingResourceLinkShare()
      | resource        | textfile.txt |
      | space           | Personal     |
      | permissionsRole | view         |
      | password        | %public%     |
    And user "Alice" has sent the following resource share invitation:                                                                                  # SharingNgContext::userHasSentTheFollowingResourceShareInvitation()
      | resource        | textfile.txt |
      | space           | Personal     |
      | sharee          | Brian        |
      | shareType       | user         |
      | permissionsRole | Viewer       |
    When the administrator renames the file "textfile.txt" to "new-name.txt" for user "Alice" on the POSIX filesystem                                   # CliContext::theAdministratorRenamesFile()
    Then the command should be successful                                                                                                               # CliContext::theCommandShouldBeSuccessful()
    And user "Brian" should have a share "textfile.txt" shared by user "Alice" from space "Personal"                                                    # SharingNgContext::userShouldHaveShareSharedByUserFromSpace()
      Share mountpoint 'textfile.txt' was not found in the drives list.
      Failed asserting that false is identical to true.
    And the public should be able to download file "textfile.txt" from the last link share with password "%public%" and the content should be "content" # PublicWebDavContext::shouldBeAbleToDownloadFileInsidePublicSharedFolderWithPasswordForSharingNGAndContentShouldBe()

in the second run the passed: https://ci.opencloud.eu/repos/3/pipeline/81/88

[butonic]

and:

runsh: Total unexpected failed scenarios throughout the test run:
coreApiWebdavEtagPropagation2/createFolder.feature:44
coreApiWebdavEtagPropagation2/createFolder.feature:46

[rhafer]

Theses seem to be flaky as well:

coreApiShareManagementBasicToShares/deleteShareFromShares.feature:102
coreApiShareManagementBasicToShares/deleteShareFromShares.feature:129

https://ci.opencloud.eu/repos/3/pipeline/112/218
https://ci.opencloud.eu/repos/3/pipeline/115/218

Similar to the above these are currently failing a lot:

coreApiShareManagementBasicToShares/deleteShareFromShares.feature:175
coreApiShareManagementBasicToShares/deleteShareFromShares.feature:176
coreApiShareManagementBasicToShares/deleteShareFromShares.feature:220

And there was: https://ci.opencloud.eu/repos/3/pipeline/341/248

coreApiWebdavUploadTUS/uploadToShare.feature:298

All of these seem to be pretty similar in how they fail. A share is being created for a file just uploaded file (or it's parent folder) and the share receiver gets a 404 when trying to access that file.

[rhafer]

Theses seem to be flaky as well:

coreApiShareManagementBasicToShares/deleteShareFromShares.feature:102
coreApiShareManagementBasicToShares/deleteShareFromShares.feature:129

https://ci.opencloud.eu/repos/3/pipeline/112/218 https://ci.opencloud.eu/repos/3/pipeline/115/218

Similar to the above these are currently failing a lot:

coreApiShareManagementBasicToShares/deleteShareFromShares.feature:175 coreApiShareManagementBasicToShares/deleteShareFromShares.feature:176 coreApiShareManagementBasicToShares/deleteShareFromShares.feature:220

And there was: https://ci.opencloud.eu/repos/3/pipeline/341/248

coreApiWebdavUploadTUS/uploadToShare.feature:298

All of these seem to be pretty similar in how they fail. A share is being created for a file just uploaded file (or it's parent folder) and the share receiver gets a 404 when trying to access that file.

Looking at the tests and the code I think the problem here is that the test are trying to access the shares before that have been accepted. Even though FRONTEND_AUTO_ACCEPT_SHARES is true by default. Accepting the share happens asynchronously and the test to wait for the share to be ACCEPT before e.g. trying to access something in that share.

Accessing an un-accepted share via the share-jail (e.g. via DELETE /remote.php/dav/spaces/a0ca6a90-a365-4782-871e-d44447bbc668$a0ca6a90-a365-4782-871e-d44447bbc668/shared/shared_file.txt) will result in a 404 error.

[ScharfViktor]

We have already put this step And user "Brian" has a share "resource" synced everywhere to prevent flakes.

[rhafer]

https://ci.opencloud.eu/repos/3/pipeline/360/70

apiAntivirus/antivirus.feature:44
apiAntivirus/antivirus.feature:200

Looking at the tests and the logs. It seems that the files havn't finished postprocessing.

For apiAntivirus/antivirus.feature:44 there is:

       ### And as "Alice" file "virusFile1.txt" should not exist
                _______________________________________________________________________

                ==> REQUEST
                PROPFIND /remote.php/dav/spaces/ed1f948c-8fe8-4fe4-9332-cea2821de6ba$942b4443-d0be-48f1-a999-655a342ad5b1/virusFile1.txt
                X-Request-ID: apiAntivirus/antivirus.feature:44-37
                ==> REQ BODY
                <?xml version="1.0"?>
                                <d:propfind
                                   xmlns:d="DAV:"
                                   xmlns:oc="http://owncloud.org/ns"
                                   xmlns:ocs="http://open-collaboration-services.org/ns"
                                   >
                                    <d:prop><d:getetag/><d:resourcetype/></d:prop>
                                </d:propfind>

                <== RESPONSE
                207 Multi-Status
                X-Xss-Protection: 1; mode=block
                <== RES BODY
                <d:multistatus xmlns:s="http://sabredav.org/ns" xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns"><d:response><d:href>/remote.php/dav/spaces/ed1f948c-8fe8-4fe4-9332-cea2821de6ba$942b4443-d0be-48f1-a999-655a342ad5b1/virusFile1.txt</d:href><d:propstat><d:prop><d:getetag>"86430f675e78c34ab77f67251bb8c31b"</d:getetag><d:resourcetype></d:resourcetype></d:prop><d:status>HTTP/1.1 425 TOO EARLY</d:status></d:propstat></d:response></d:multistatus>

And for apiAntivirus/antivirus.feature:200 the ### And user "Brian" should get a notification with subject "Virus found" and message seems to fail.

But I think in both cases we need to wait for the post-processing on that file to finish before checking the notifications. After all the notifications are send during post-processing so checking them before it finished can't work reliably.

[rhafer]

https://ci.opencloud.eu/repos/3/pipeline/366/140

apiSearchContent/contentSearch.feature:129

It might be that the 5s sleep before the search just isn't enough for the indexes to be updated.

[individual-it]

We wait for max 5 sec for the notification in https://github.com/opencloud-eu/opencloud/blob/72fd765b93dc9abe65e4143e6293a2dfd58bb889/tests/acceptance/bootstrap/NotificationContext.php#L402-L401
that could be increased
The other problem is that @Then as :user :entry :path should not exist will accept any 4xx return code as valid see https://github.com/opencloud-eu/opencloud/blob/fb8af22073cee757a1aeed54cd30aa2fda9615eb/tests/acceptance/bootstrap/WebDav.php#L1267
I thin 425 should not be accepted, but in that case it should retry

[individual-it]

also the search sleep can be increased (done in opencloud-eu/opencloud#1802) but the real fix would be to redo the search if the propfind response has any 425

[individual-it]

Here we got a 500
https://ci.opencloud.eu/repos/3/pipeline/382/64
in apiAntivirus/antivirus.feature:334