[tizen_app_control] Integration test is failing

[seungsoo47]

@swift-kim Integration test for tizen_app_control is failing like following. Could you check the problems?

$ ./tools/run_command.py test --plugins tizen_app_control --platforms mobile-6.5
============= Testing for tizen_app_control (1/1) =============

00:00 +0: loading /tmp/flutter_tools.KGWRDJ/KLIKGX/tizen_app_control_test.dart Building a Tizen application in debug mode...
The project type of messageport_tizen plugin is "sharedLib", which is deprecated in favor of "staticLib".

00:21 +0: loading /tmp/flutter_tools.KGWRDJ/KLIKGX/tizen_app_control_test.dart
00:22 +0: loading /tmp/flutter_tools.KGWRDJ/KLIKGX/tizen_app_control_test.dart The public profile is used for signing.

00:34 +0: loading /tmp/flutter_tools.KGWRDJ/KLIKGX/tizen_app_control_test.dart
00:35 +0: loading /tmp/flutter_tools.KGWRDJ/KLIKGX/tizen_app_control_test.dart Building a Tizen application in debug mode... 36.3s
✓ Built build/tizen/tpk/org.tizen.tizen_app_control_example-1.0.0.tpk (26.5MB).

00:37 +0: loading /tmp/flutter_tools.KGWRDJ/KLIKGX/tizen_app_control_test.dart Installing build/tizen/tpk/org.tizen.tizen_app_control_example-1.0.0.tpk...
01:03 +0: loading /tmp/flutter_tools.KGWRDJ/KLIKGX/tizen_app_control_test.dart
01:04 +0: loading /tmp/flutter_tools.KGWRDJ/KLIKGX/tizen_app_control_test.dart 27.0s

01:09 +0: loading /tmp/flutter_tools.KGWRDJ/KLIKGX/tizen_app_control_test.dart
01:11 +1: Can receive request from platform
01:11 +2: Can find matching applications
01:11 +3: Can send and receive request
01:11 +4: Omit invalid extra data
01:22 +4: Can send and receive reply
01:22 +4 -1: Can send and receive reply [E]
TimeoutException after 0:00:10.000000: Test timed out after 10 seconds.
dart:isolate _RawReceivePortImpl._handleMessage

01:22 +5 -1: Cannot find target applications
01:23 +5 -1: Can terminate service application
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following TestFailure object was thrown running a test:
Expected: false
Actual:

When the exception was thrown, this was the stack:
#4 main. (file:///home/leess/11_HDD/01_SRC/2020_Flutter/github/plugins/packages/tizen_app_control/example/integration_test/tizen_app_control_test.dart:111:5)
#5 main. (file:///home/leess/11_HDD/01_SRC/2020_Flutter/github/plugins/packages/tizen_app_control/example/integration_test/tizen_app_control_test.dart:110:52)
#6 testWidgets.. (package:flutter_test/src/widget_tester.dart:176:29)


(elided one frame from package:stack_trace)

This was caught by the test expectation on the following line:
file:///home/leess/11_HDD/01_SRC/2020_Flutter/github/plugins/packages/tizen_app_control/example/integration_test/tizen_app_control_test.dart line 111
The test description was:
Can terminate service application
════════════════════════════════════════════════════════════════════════════════════════════════════

01:23 +5 -2: Can terminate service application [E]
Test failed. See exception logs above.
The test description was: Can terminate service application

01:24 +5 -2: (tearDownAll)
01:24 +5 -2: Some tests failed.
============= TEST RESULT =============
FAILED: tizen_app_control ('mobile-6.5', 'TM1', '0000d82c00006200')
flutter-tizen test integration_test failed, see the output above for details.
Some tests failed.

[swift-kim]

It's a known issue and for this reason tizen_app_control is being excluded from CI integration testing.

plugins/.github/workflows/integration_test.yml

Line 46 in 656fbab

google_maps_flutter tizen_app_control url_launcher network_info_plus

The issue will not be fixed due to the fundamental limitation of flutter-tizen's test command. You can use the drive command to run the integration test instead.

@HakkyuKim By the way, why do we manage two separate lists of packages to include/exclude packages to/from integration testing? Can we use only one?

• https://github.com/flutter-tizen/plugins/blob/master/tools/recipe.yaml

• plugins/.github/workflows/integration_test.yml

  Lines 43 to 46 in 656fbab

  	--exclude wearable_rotary image_picker camera webview_flutter \
  	video_player permission_handler geolocator battery connectivity \
  	device_info package_info sensors share wifi_info_flutter \
  	google_maps_flutter tizen_app_control url_launcher network_info_plus

[HakkyuKim]

@swift-kim
The recipe file does not include/exclude plugins to test, it specifies which target platforms to test on and the tool refers to the file to create specific emulator profiles and Tizen versions.

plugins/tools/commands/integration_test.py

Lines 371 to 385 in 45896bd

	group.add_argument('--recipe',
	type=str,
	default='',
	help='''The recipe file path. A recipe refers to a
	yaml file that defines a list of target platforms to test for each plugin.
	Pass this file if you want to select specific target platform for different
	plugins. Note that recipe does not select which plugins to test(that is covered
	by the --plugins option), it only defines which target platform to test
	for certain plugins if those plugins are being tested.
	(
	plugins:
	a: [wearable-5.5, tv-6.0]
	b: [mobile-6.0]
	c: [wearable-4.0]
	)''')

A similar proposal was made by @bbrto21 here, I haven't got a chance to work on it yet. When I do, I'll design towards listing all info in recipe.

[swift-kim]

@HakkyuKim But omitting a package or providing an empty list for a package in the recipe file effectively does the same thing as the --exclude option, right? I was thinking of eliminating this redundancy.

[swift-kim]

I read your comment about an empty list in the recipe file but I think you need to be more explicit about the "non specified" case if you really need it. For example, the following syntax may be used.

plugins:
  audioplayers: any
  battery_plus: []
  camera: ["mobile-6.0"]

[HakkyuKim]

The "non-specified" case(empty list) doesn't have any practical meaning at this point, I remembered that I copied the entries from device limitation table. At the time I wanted to point out that plugins with empty list cannot be tested on emulators, but I guess we're already filtering those out in the CI script.
We could just remove the entries with empty list, the result would be the same:

// recipe.yaml:
// plugins:
//   battery_plus: []

./tools/run_command.py test --plugins battery_plus --recipe recipe.yaml
FAILED: battery_plus 
Cannot find any testable targets.
Some tests failed.

// recipe.yaml:
// plugins:
//   nothing:

./tools/run_command.py test --plugins battery_plus --recipe recipe.yaml
FAILED: battery_plus 
Cannot find any testable targets.
Some tests failed.

[swift-kim]

@HakkyuKim I don't quite get what you want to say.

[HakkyuKim]

@swift-kim

By the way, why do we manage two separate lists of packages to include/exclude packages to/from integration testing? Can we use only one?

Right now, no because

But omitting a package or providing an empty list for a package in the recipe file effectively does the same thing as the --exclude option, right? I was thinking of eliminating this redundancy.

is not true.

# recipe.yaml
plugins:
  battery_plus: ["wearable-5.5"]
  camera: []
  device_info_plus: []

./tools/run_command.py test --exclude device_info_plus --recipe recipe.yaml

means:

• test battery_plus on wearable-5.5 emulator. -> ok
• test camera on None. -> error because no testing platforms are given.
• exclude device_info_plus -> ok because intentionally exclude testing. (The tool will not look for the entry in recipe.yaml)

I haven't got a chance to work on it yet. When I do, I'll design towards listing all info in recipe.

Meant that maybe later recipe file can support something like:

# recipe.yaml
plugins:
  battery_plus: ["wearable-5.5"]
  camera: []
  device_info_plus: exclude

As for now, we can remove all empty list entries in the recipe file because we're specifying them with --exclude(Not because empty list will exclude them from tests but because the tool will not look for the entry in recipe.yaml).

Does this answer your question?

[swift-kim]

Meant that maybe later ...

I know the current status. What we're talking about here is that "later" design and I wanted to hear your opinion on my suggestion.

[HakkyuKim]

@swift-kim and I talked about the subject on company chat and concluded that it's simpler to just not have empty list/any. I'll remove them from recipe.yaml.