Espresso test failing

[hannesa2]

sometimes, the BitRise fails due to some timeouts or performance problems in its internal emulators. We are looking for the best solutions.

Originally posted by @jesmrec in #2689 (comment)

I took a look into Espresso test healthiness but the result is sobering.
I run locally ./gradlew :owncloudApp:connectedDebugAndroidTest on master c2d0e86 and see this

Does is only run on specific Emulators ? I use

Or is there some magic to to on my machine, to have a constant 100% success result ?

---

Edited by @davigonz

After writing down the tests which fail with more frequency, we can determine that secondTryIncorrect is the test that fail at most times. @jesmrec I think you wrote this test, is there anything that can be improved there?

[jesmrec]

no, the magic is not needed. Let me tell you what the problem is.

Before the current re-architecting of the app, there was not posible to develop test in every layer. So, the Espresso tests cover the whole stack (including the server). So, these tests were removed from CI, to be executed only locally. We should move then (not remove), in order to have a cleaner execution, avoiding the thing you noticed. I will take care of this.

[hannesa2]

Nice, when you take care about it. 👍
Btw, I expected a constant failure rate, but the percentage is changing, that means there are flaky tests too

[davigonz]

Hi @hannesa2 , thanks for these reports. UI tests in Android, as you probably already know, are highly dependent on the device in which are executed: different screens resolution, size, performance, animations. It seems that the Emulator you chose is a default one with everything enabled I guess, including the animations, so your results are what I would expect.

We are using Virtual Device Testing for Android with the Bitrise CLI but there's one or two tests that fail from time to time and are not always the same so we suppose that the emulator gets stuck at some point because it's not optimised (enabled animations and so on) so far, we already asked Bitrise guys and they can not do too much, at least, for the moment.

So what did we do there? We created a script step, trying to run the emulator we wanted, disabling the animations but something was wrong as well, the Bitrise jobs started to get aborted due to timeouts, it's also reported to Bitfire guys.

What I'm doing now is having a look at the failing tests every time we run the tests and I'm writing down the tests that fail the most so we can take a deeper look at them, in case we can improve something there.

Anyway, JFYI, we're currently focused on modularizing the app and one of the most important tasks is adapting the tests to those layers and even create new ones, so as soon as we have that task finished and the tests running again (locally, at least), we will find a solution for this, no worries.

[hannesa2]

I run on other places tests with animation, almost 100 % error free (around 300 tests). For the flakyness I have/prepare a workaround for myself

To know what's wrong on Espresso tests, I collect failing test with Google Analytics, then you have a clear and exact overview, which tests needs the most attention

eg:

#!/usr/bin/env bash

TID="UA-<YOUR NUMBER>-1"

while getopts ":a:c:l:" opt; do
	case $opt in
		a)
			ACTION="$OPTARG"
			;;
		c)
			CATEGORY="$OPTARG"
			;;
		l)
			LABEL="$OPTARG"
			;;
		\?)
			echo "Invalid option: -$OPTARG" >&2
			exit 1
			;;
		:)
			echo "Option -$OPTARG requires an argument." >&2
			exit 1
			;;
	esac
done

#echo "Action = ${ACTION}"
#echo "Category = ${CATEGORY}"
#echo "Label = ${LABEL}"

# For improvement https://www.optimizesmart.com/understanding-universal-analytics-measurement-protocol/

CID=$(uuidgen);
curl \
	--silent \
	--data-urlencode "v=1" \
	--data-urlencode "t=event" \
	--data-urlencode "tid=${TID}" \
	--data-urlencode "cid=${CID}" \
	--data-urlencode "ec=${CATEGORY}" \
	--data-urlencode "ea=${ACTION}" \
	--data-urlencode "el=${LABEL}" \
	http://www.google-analytics.com/collect > /dev/null

[davigonz]

I run on other places tests with animation, almost 100 % error free (around 300 tests). For the flakyness I have/prepare a workaround for myself

A description about what you are trying to do in https://github.com/hannesa2/Moka would be appreciated 😉 , I just see No description, website, or topics provided.

To know what's wrong on Espresso tests, I collect failing test with Google Analytics, then you have a clear and exact overview, which tests needs the most attention

Which information do you want to send to Google Analytics? The logs when executing the tests and which tests are failing? If so, Bitrise is already telling us in a very complete UI the tests that are failing, logs and so on, and I just need to know which tests fail the most.

Google Analytics is a good option for handling larger amounts of information, using it does not worth it for what I need, in my humble opinion. But well, if you figure out something else, let us know. Thanks

[jesmrec]

Let's close this since #2711 was merged. Please, reopen if it is needed.

[hannesa2]

On merged master 9d123cc I can't confirm this 100% error free behavior

The question is still what's different on your side ?

[jesmrec]

@hannesa2 let's check what happens with #2737