Fixed failing location query robot test. [5.1]

[mauritsvanrees]

Well, I hope this fixes it anyway, similar to what Johannes has already done. Locally, with Firefox 46 (or chrome) this change is not needed, at least not when running the location query test individually. But Jenkins has been failing for a while, for example http://jenkins.plone.org/job/plone-5.1-python-2.7-robot/558/robot/

[mauritsvanrees]

The real problem is in Wait until keyword succeeds. When this fails the first time and succeeds on a retry, selenium/robotframework thinks it has failed. Locally everything may seem fine. But on Jenkins, the tests are run with ROBOT_SELENIUM_RUN_ON_FAILURE='Capture Page Screenshot'. You can try this locally too:

export ROBOT_SELENIUM_RUN_ON_FAILURE='Capture Page Screenshot'
bin/test --all -s Products.CMFPlone -m test_robot -t test_linkintegrity

With CMFPlone master the tests will pass. But then open parts/test/robot_report.html, go to the link integrity tests and click them open. It will show one failure for Page should not contain element followed by a success for the second try:

Zoom in and you see that a screen shot was generated because of the initial failure:

What is happening in this specific case is:

• go to folder contents
• the delete button is disabled
• select an item
• the delete button is enabled, but this takes a short while so we retry it a couple of times.

You can see the same when you start the tests with ROBOT_SELENIUM_RUN_ON_FAILURE=Debug, which will open a debug console and when you exit this, all will seem fine. I sometimes get an error when I try this. This probably depends on how long I take before exiting the debug console. [Edit: yes, when I wait longer than the amount of seconds that is specified in Wait until keyword succeeds, then I get a failure. ]

This pull request addresses the symptom by doing a Sleep 1 before Wait until keyword succeeds.

The real fix would be to fix (p.a.)robotframework or selenium or jenkins or whatever to not treat these initial failures as real failures. I am happy if someone else dives into that...

[mauritsvanrees]

cc @thet @jensens @gforcada .
I think the above is why we had so many problems with robot tests lately.
Apart from that (though possibly ultimately caused by it), there are several problems on the jenkins nodes: plone/jenkins.plone.org#180 plone/jenkins.plone.org#182 plone/jenkins.plone.org#183

[mauritsvanrees]

Alternatively, we could switch off the creation of screen shots on failure. That would also be a workaround, but currently the html reports are not visible on jenkins, so no screen shots are visible. (Well, I think I did see some screen shots in a list somewhere, but I cannot find them.) If the absence of screen shots means the tests become green again, then I am happy.

I don't know where this could be changed without affecting other users of plone.app.robotframework. This seems to be where the default is set: https://github.com/plone/plone.app.robotframework/blob/master/src/plone/app/robotframework/selenium.robot#L14

[jensens]

I think plone.app.robotframework is the wrong place to set this variable anyway, To screenshot or not is more a feature to enable/disable from the outside, so i.e. as an environment variable.

If this is not possible, may we unset/blank this variable before a Wait until keyword succeeds and switch back on afterwards?

The real fix would be to fix robotframeworks selenium to detect failures correctly, right?

[mauritsvanrees]

Some notes.

We pass Capture Page Screenshot as run_on_failure to Selenium2Library here:
https://github.com/plone/plone.app.robotframework/blob/0.9.16/src/plone/app/robotframework/selenium.robot#L5
We can override this on the command line with for example ROBOT_SELENIUM_RUN_ON_FAILURE=Debug.

Capture Page Screenshot is already the default in Selenium2Library:
https://github.com/robotframework/Selenium2Library/blob/v1.7.4/src/Selenium2Library/__init__.py#L159
There we can see that we can use Nothing to disable this. I propose we do that in plone.app.robotframework.

When we have a keyword for running on failure, this is run using a decorator in Selenium2Library:
https://github.com/robotframework/Selenium2Library/blob/v1.7.4/src/Selenium2Library/keywords/keywordgroup.py#L17

The retrying for Wait until keyword succeeds happens here in robotframework:
https://github.com/robotframework/robotframework/blob/2.9.2/src/robot/libraries/BuiltIn.py#L1736
But that is a try/except around the already decorated test function. So the page screenshot will already have been made, and it seems this leads to Jenkins thinking that the test has failed. (Locally the test returns error code zero, so I am not sure why that is.)

There is an issue in Selenium2Library which suggests to remove run_on_failure because this can be done differently, using core robotframework functionality. This may not have been available at the time, I'm not sure. The idea is to not use run_on_failure, but to use the test teardown. We can create a keyword like this (but cleaned up):

Plone Test Teardown
    Log  Doing custom plone teardown
    Run Keyword If Test Passed  Log  Test passed
    Run Keyword If Test Failed  Log  Test failed
    Run Keyword If Test Failed  ${SELENIUM_RUN_ON_FAILURE}
#    Run Keyword If Test Failed  Capture Page Screenshot
    Report test status
    Close all browsers

So then you can still override what happens on failure by passing ROBOT_SELENIUM_RUN_ON_FAILURE=Debug on the command line. But when we initialize the Selenium2Library, we should pass run_on_failure=Nothing. That seems to do the trick locally, though we will have to see what happens on Jenkins. I will make a pull request.

[mauritsvanrees]

Sigh, the 5.1 robot tests fail again. This pull request is almost finished, but is failing too:

StaleElementReferenceException: Message: Element not found in the cache
 - perhaps the page has changed since it was looked up

This pull request should no longer be relevant:

• I already took over the Click Element css=body in a different pull request that was merged.
• The extra Sleep 1 should no longer be necessary after the merged p.a.robotframework fix.

[mauritsvanrees]

I am rebuilding http://jenkins.plone.org/job/plone-5.1-python-2.7-robot/562/consoleFull just for good measure.