CLI tests with dummy core pass unexpectedly on Windows

[dail8859]

The CLI tests only validate that the executable returned a 0 exit code, it does not validate the textual output of the command.

My app output something such as "dummy output" and it still passes the two version switch tests. I do not know much about CMake, however it looks like this function needs to accept parameters.

editorconfig-core-test/CMakeLists.txt

Lines 105 to 116 in 48610d4

	function(new_ec_cli_test)
	# Parse args
	set(one_value_keywords NAME MATCH)
	set(multi_value_keywords ARGS)
	cmake_parse_arguments(P
	"" "${one_value_keywords}" "${multi_value_keywords}" ${ARGN})
	# Add test
	add_test(NAME ${P_NAME} COMMAND ${EDITORCONFIG_CMD} ${P_ARGS})
	set_tests_properties(${name} PROPERTIES PASS_REGULAR_EXPRESSION "${P_MATCH}")
	endfunction()

[cxw42]

Thanks for the report! Would you please attach or point us to your test code, so we can reproduce the problem? The set_tests_properties line sets the regex to check in the output. Much appreciated!

[dail8859]

I do not have any easy shareable example currently but it should not be hard to verify with any of the other cores.

Shouldn't new_ec_cli_test accept parameters like the other new_xxx functions?

[cxw42]

It does! cmake_parse_arguments is the function that extracts the args. CMake syntax is very strange.

What project are you working on? You have opened a number of issues recently, and we could better assist if we knew the big picture.

In the mean time, I am closing this issue in the absence of a test case.

[dail8859]

CMake syntax is very strange.

Indeed it is!

What project are you working on?

I'm writing an editorconfig core for Qt5

in the absence of a test case.

I'm using Windows for this test...

1. mkdir mytest
2. cd mytest
3. git clone https://github.com/editorconfig/editorconfig-core-test
4. Create CMakeLists.txt with this text:
   cmake_minimum_required(VERSION 3.17)
   project(mytest NONE)
   enable_testing()
   set(EDITORCONFIG_CMD ${CMAKE_CURRENT_SOURCE_DIR}/test.bat )
   add_subdirectory(editorconfig-core-test)
   message( ${CMAKE_CURRENT_SOURCE_DIR} )
5. Create test.bat with this text
   echo "nothing"
6. cmake .
7. ctest -C Release -R switch (no clue why I have to include the -C Release on my system but it complains if it is missing...the -R will just run the two version switch tests)
8. The two tests incorrectly pass

[cxw42]

editorconfig/editorconfig-qtcreator#3

[dail8859]

Yep. Very aware of that issue. Even commented in it :)

[dail8859]

Oh never mind. You are probably just linking the issue and not pointing it out to me specifically :)