re-add bison + flex tests

[GitMensch]

The first step would be to integrate the test suites back and everything that is necessary to execute them.
With bison this should be quite easy: just add the testsuite and a created atlocal script (with bison script incorporated if needed). Then all that is likely needed is a minimal MinGW/Cygwin installation + diff to execute it.

Flex seems to be a little bit more work as it doesn't use a autotest testsuite script but a self-made one based on automake. But this may also allow to run the tests with VS nmake (no unix shell interpreter needed).

With something important as the parser/tokenizer core of many systems I actually do wander why there are no tests (or did I missed something other that "the un-modified versions are tested already")...

[lexxmark]

With something important as the parser/tokenizer core of many systems I actually do wander why there are no tests (or did I missed something other that "the un-modified versions are tested already")...

It's all about time and efforts vs results. I only had time to refactor flex/bison code to windows environment. As windows supports linux subsystem I hope this project will be less demanded in the future. So I don't see any reason to make this project better as it's now.

But if you prefer to invest in this I think it would be valuable for the project.

Then all that is likely needed is a minimal MinGW/Cygwin installation + diff to execute it.

Is any chance to stay within VS C++ compiler?

[GitMensch]

As windows supports linux subsystem I hope this project will be less demanded in the future.

I don't see this. I guess you speak about the so-called "bash on windows" but this means to install some GB just to translate bison/flex sources to C.

Then all that is likely needed is a minimal MinGW/Cygwin installation + diff to execute it.

Is any chance to stay within VS C++ compiler?

Depends on what you understand as "stay within". Doing the tests from VS would mean to rewrite the complete testsuite script as VS test (the "sources" for the testsuite are available, therefore this would be possible but likely time intensive [remark: I don't know very much about the VS internal test option]).
Otherwise you need something that can execute the test script (= a shell interpreter and some standard tools like diff) or [which would be the best option but I haven't heard of any general purpose tool that works] create/use a test runner that directly executes the test definitions *.a (or an automated tool/script that change these to VS tests)).

Note: Even with a MinGW/cygwin shell executing the tests these would still be executed using the VS-generated win_flex and win_bison executables and would use cl.exe to compile their results.

[lexxmark]

I don't see this. I guess you speak about the so-called "bash on windows" but this means to install some GB just to translate bison/flex sources to C.

I meant Windows_Subsystem_for_Linux

Doing the tests from VS would mean to rewrite the complete testsuite script as VS test (the "sources" for the testsuite are available, therefore this would be possible but likely time intensive [remark: I don't know very much about the VS internal test option]).

I think it's not an option. As I understand Windows_Subsystem_for_Linux is pretty similar to MinGW/cygwin. So we could adopt tests to run using MinGW/cygwin and get possibility to run test under Windows_Subsystem_for_Linux for free. So user will have two options for tests.

Note: Even with a MinGW/cygwin shell executing the tests these would still be executed using the VS-generated win_flex and win_bison executables and would use cl.exe to compile their results.

I understand that.

[GitMensch]

I meant Windows_Subsystem_for_Linux

That is exactly what I meant - GNU/kWindows. As it is now available via Windows Store and the initial size gone down with the Debian "app" to 76MB this would be an option for people. Adding bison and flex brings in some additional MBs...

Note: Your VS build rules could be adjusted to run these tools from the WSL environment (using wsl.exe flex [...]) - Did you already tried this?

[lexxmark]

Note: Your VS build rules could be adjusted to run these tools from the WSL environment (using wsl.exe flex [...]) - Did you already tried this?

No I haven't tried it. I'm kinda lazy man :)
But it's a good idea.

[GitMensch]

If it isn't possible to add the full tests soon we should at least add an example compilation.
Thanks to @Croydon we have a working CI build now which would do the checks, too.

Actually the bug I've fixed with 6cfd835 is now many years old and would have raised a broken test in the debug environments (it likely had no actual bad result in production but could be seen directly after the commit).

I should investigate adding the original test suites with the goal to run them on CI. If this isn't possible within a reasonable time frame I go with extracting bison/flex sources and all needed files to compile them from GnuCOBOL, run bison/flex and try to use the C compiler to compile to object files. This would at least provide some general checks. @lexxmark any thoughts about adding a test subfolder containing these files?

[lexxmark]

Yes, please.
I would be great to have tests.

[GitMensch]

The first approach to just use the 2.6.4 tests with a hand-revised Makefile and testsuite runner did not work out completely (compilation of the test programs [in Release mode] is fine but the tests aren't all run) and showed some errors. Just tested using a full-configured Cygwin environment + westes/flex@4e2a81a, this time everything compiles and runs but the same test failures are in:

[...]
PASS: tableopts_opt_nr-Ca.opt.exe
PASS: tableopts_opt_nr-Ce.opt.exe
PASS: tableopts_opt_nr-Cf.opt.exe
PASS: tableopts_opt_nr-Cm.opt.exe
PASS: tableopts_opt_nr-Cem.opt.exe
PASS: tableopts_opt_nr-Cae.opt.exe
PASS: tableopts_opt_nr-Caef.opt.exe
PASS: tableopts_opt_nr-Cam.opt.exe
PASS: tableopts_opt_nr-Caem.opt.exe
PASS: tableopts_opt_r-Ca.opt.exe
PASS: tableopts_opt_r-Ce.opt.exe
PASS: tableopts_opt_r-Cf.opt.exe
PASS: tableopts_opt_r-Cm.opt.exe
PASS: tableopts_opt_r-Cem.opt.exe
PASS: tableopts_opt_r-Cae.opt.exe
PASS: tableopts_opt_r-Caef.opt.exe
PASS: tableopts_opt_r-Cam.opt.exe
PASS: tableopts_opt_r-Caem.opt.exe
FAIL: tableopts_ser_nr-Ca.ser.exe
FAIL: tableopts_ser_nr-Ca.ser.exe
FAIL: tableopts_ser_nr-Ce.ser.exe
FAIL: tableopts_ser_nr-Cf.ser.exe
FAIL: tableopts_ser_nr-Cm.ser.exe
FAIL: tableopts_ser_nr-Cem.ser.exe
FAIL: tableopts_ser_nr-Cae.ser.exe
FAIL: tableopts_ser_nr-Caef.ser.exe
FAIL: tableopts_ser_nr-Cam.ser.exe
FAIL: tableopts_ser_nr-Caem.ser.exe
FAIL: tableopts_ser_r-Ca.ser.exe
FAIL: tableopts_ser_r-Ce.ser.exe
FAIL: tableopts_ser_r-Cf.ser.exe
FAIL: tableopts_ser_r-Cm.ser.exe
FAIL: tableopts_ser_r-Cem.ser.exe
FAIL: tableopts_ser_r-Cae.ser.exe
FAIL: tableopts_ser_r-Caef.ser.exe
FAIL: tableopts_ser_r-Cam.ser.exe
FAIL: tableopts_ser_r-Caem.ser.exe
FAIL: tableopts_ver_nr-Ca.ver.exe
FAIL: tableopts_ver_nr-Ce.ver.exe
FAIL: tableopts_ver_nr-Cf.ver.exe
FAIL: tableopts_ver_nr-Cm.ver.exe
FAIL: tableopts_ver_nr-Cem.ver.exe
FAIL: tableopts_ver_nr-Cae.ver.exe
FAIL: tableopts_ver_nr-Caef.ver.exe
FAIL: tableopts_ver_nr-Cam.ver.exe
FAIL: tableopts_ver_nr-Caem.ver.exe
FAIL: tableopts_ver_r-Ca.ver.exe
FAIL: tableopts_ver_r-Ce.ver.exe
FAIL: tableopts_ver_r-Cf.ver.exe
FAIL: tableopts_ver_r-Cm.ver.exe
FAIL: tableopts_ver_r-Cem.ver.exe
FAIL: tableopts_ver_r-Cae.ver.exe
FAIL: tableopts_ver_r-Caef.ver.exe
FAIL: tableopts_ver_r-Cam.ver.exe
FAIL: tableopts_ver_r-Caem.ver.exe
[...]
# TOTAL: 113
# PASS:  69
# SKIP:  0
# XFAIL: 0
# FAIL:  44
# XPASS: 0
# ERROR: 0

test-suite.log

As cygwin64 is part of the Appveyor environment I think about just adding clwrap.sh to the repo, then after building do something like:

FLEXVER=2.6.4
wget https://github.com/westes/flex/releases/download/v$FLEXVER/flex-$FLEXVER.tar.gz
tar -xvf flex-$FLEXVER.tar.gz
cd flex-$FLEXVER
./configure
cd tests
sed -i -e 's|/dev/null|/NUL|' options.cn
make check "CC=./clwrap.sh" "CXX=./clwrap.sh -EHsc" "OBJEXT=obj" "FLEX=/cygdrive/c/projects/winflexbison/build/win_flex_bison-$APPVEYOR_REPO_TAG_NAME/win_flex.exe" "YACC=/cygdrive/c/projects/winflexbison/build/win_flex_bison-$APPVEYOR_REPO_TAG_NAME/win_bison.exe -y"

Opinions?

@lexxmark Can you please have a look at the currently failing tableopts tests (inspecting the posted test-suite.log)?

[lexxmark]

Opinions?

Sounds good.

Can you please have a look at the currently failing tableopts tests (inspecting the posted test-suite.log)?

I'm not familiar with clwrap.sh. Could you point me where in test-suite.log file I can get command that failed and expected result.
I will really appreciate you if you give me flex file to debug.

[GitMensch]

I'm not familiar with clwrap.sh

That's just a quick hack to translate the commands from the configured CC=gcc to the actual used cl.exe to use one for configuration (the normal flex environment won't configure with cl.exe) and the other one as test.

Let's inspect the test-suite.log together (I'm not familiar with the flex test-suite) and try to decipher it:

=========================

set -euvx
+ set -euvx

# testwrapper.sh: run a flex test, typically called by a Makefile

# Each test will exercise some feature or aspect of flex. Run the test with any
# input it may need.

INPUT_DIRECTORY=""
+ INPUT_DIRECTORY=
INPUT_NAME=""
+ INPUT_NAME=
INPUT_COUNT=0
+ INPUT_COUNT=0
USE_REDIRECT=0
+ USE_REDIRECT=0
DO_COMPARISON=0
+ DO_COMPARISON=0

while getopts :d:i:rt1 OPTION ; do
    case $OPTION in
        d) INPUT_DIRECTORY=$OPTARG ;;
        i)
            if [ "$INPUT_NAME" = "" ] ; then
                INPUT_NAME="$OPTARG"
            else
                INPUT_NAME="$INPUT_NAME $OPTARG"
            fi
            INPUT_COUNT=$(($INPUT_COUNT+1))
            ;;
        r) USE_REDIRECT=1 ;;
        t) USE_TABLES=1 ;;
        1) DO_COMPARISON=1 ;;
    esac
done
+ getopts :d:i:rt1 OPTION
+ case $OPTION in
+ INPUT_DIRECTORY=.
+ getopts :d:i:rt1 OPTION
+ case $OPTION in
+ '[' '' = '' ']'
+ INPUT_NAME=../../tests/tableopts.txt
+ INPUT_COUNT=1
+ getopts :d:i:rt1 OPTION
+ case $OPTION in
+ USE_REDIRECT=1
+ getopts :d:i:rt1 OPTION
+ case $OPTION in
+ USE_TABLES=1
+ getopts :d:i:rt1 OPTION

shift $(($OPTIND-1))
+ shift 6
TESTNAME=$1
+ TESTNAME=./tableopts_ser_nr-Ca.ser.exe

INPUT_NAME=${INPUT_NAME:-$INPUT_DIRECTORY/`basename "${TESTNAME%.exe}"`.txt}
+ INPUT_NAME=../../tests/tableopts.txt

if [ $DO_COMPARISON = 1 ] ; then
    TEST_OUTPUT=`$TESTNAME < $INPUT_NAME | tr -d "\r"`
    REF_OUTPUT=`$TESTNAME 1 < $INPUT_NAME | tr -d "\r"`
    test "$TEST_OUTPUT" -eq "$REF_OUTPUT"
    exit $?
fi
+ '[' 0 = 1 ']'

if [ $INPUT_COUNT -gt 1 ] ; then
    $TESTNAME ${USE_TABLES:+${INPUT_DIRECTORY}/${TESTNAME%.exe}.tables} ${INPUT_NAME}
    exit $?
fi
+ '[' 1 -gt 1 ']'

if [ -f ${INPUT_NAME} ] ; then
    if [ $USE_REDIRECT = 1 ] ; then
        $TESTNAME ${USE_TABLES:+${INPUT_DIRECTORY}/${TESTNAME%.exe}.tables} < $INPUT_NAME
    else
        $TESTNAME ${USE_TABLES:+${INPUT_DIRECTORY}/${TESTNAME%.exe}.tables} $INPUT_NAME
    fi
else
    $TESTNAME
fi
+ '[' -f ../../tests/tableopts.txt ']'
+ '[' 1 = 1 ']'
+ ./tableopts_ser_nr-Ca.ser.exe ././tableopts_ser_nr-Ca.ser.tables
table id not found in map.
FAIL tableopts_ser_nr-Ca.ser.exe (exit status: 2)

This is what I guess:

• the un-prefixed parts are what the testwrapper.sh has
• the parts that are effective for the specific test are marked with +

This leads us to:

• TESTNAME=./tableopts_ser_nr-Ca.ser.exe
• INPUT_NAME=../../tests/tableopts.txt
• USE_TABLES=1
• DO_COMPARISON=0 (leading to the shell script not using the comparison branch)
• INPUT_COUNT=1 (leading to the shell script not using the direct call branch)
• USE_REDIRECT=1
• final call: ./tableopts_ser_nr-Ca.ser.exe ././tableopts_ser_nr-Ca.ser.tables
• result "table id not found in map.", exit code 2

To generate tableopts_ser_nr-Ca.ser.exe we have to check the makefile.

• Input: https://github.com/westes/flex/blob/master/tests/tableopts.l4
• C file generated with: win_flex.exe --unsafe-no-m4-sect3-escape -P $(subst -,_,$(basename $(*F))) --tables-file="tableopts_ser_nr-Ca.ser.tables" --tables-verify $(subst _F,F,$*) -o tableopts_ser_nr-Ca.ser.c tableopts.l4
• compilation done with -DTEST_HAS_TABLES_EXTERNAL (and, via clwrap.sh with -DYY_NO_UNISTD_H

This is what I've just tried on a "normal" environment with the tableopts.l4 locally available:

win_flex.exe --unsafe-no-m4-sect3-escape  --tables-file="tableopts_ser_nr-Ca.ser.tables" --tables-verify -o tableopts_ser_nr-Ca.ser.c tableopts.l4
rem change generated C file: comment config.h, replace <netinet/in.h> with <Winsock2.h>
cl.exe -DTEST_HAS_TABLES_EXTERNAL -DYY_NO_UNISTD_H tableopts_ser_nr-Ca.ser.c Ws2_32.lib
.\tableopts_ser_nr-Ca.ser.exe tableopts_ser_nr-Ca.ser.tables
   tables verification failed at flex_int8_t

I have not used generation of a tables file before this one and I have no clue how they should work, but I guess the testsuite checks that the result has an errorlevel of zero.

If something is unclear or if you need more information I'd recommend to ask in the flex mailing list / flex Github issues.

[lexxmark]

It seems like this issue related to windows EOL as well.
I changed EOL in win_flex and test has passed. See last commit

[GitMensch]

EOL is not fixed for stdout/stderr, the tests are still failing.
You can also see this with win_bison --help > help.txt where help.txt should contain unix-lf.

I assume you've missed something like

		(void)_setmode (_fileno (stdin), _O_BINARY);
		(void)_setmode (_fileno (stdout), _O_BINARY);
		(void)_setmode (_fileno (stderr), _O_BINARY);

The part that I've previously requested was to not use binary mode by default, but only if explicit requested. Otherwise you write files with \n only which may confuse some Windows editors and the stdout/stderr may occur broken on cmd.exe.
Something along the following code near the top of main() for both bison and flex:

#ifdef	_WIN32
	/* Allows running tests under woe32 */
	char *p = getenv ("USE_UNIX_LF");
	if (p && (*p == 'Y' || *p == 'y' || *p == '1')) {
		use_unix_lf = 1;  // extern int, referenced in all places to decide for "wb" or "w"
		(void)_setmode (_fileno (stdin), _O_BINARY);
		(void)_setmode (_fileno (stdout), _O_BINARY);
		(void)_setmode (_fileno (stderr), _O_BINARY);
	}
#endif

[lexxmark]

I have added development branch with
(void)_setmode (_fileno (stdout), _O_BINARY);
(void)_setmode (_fileno (stderr), _O_BINARY);
code added. please check

You can modify my changes or add needed code to get tests pass.
If something unclear - let me know I will try to help you.

When we finish with tests we will cleanup code (add macro guard etc) and merge it to master