Add a utility to assist in manually testing an exercise

Instead of having to manually enable tests, copy the examples, and
clean up, this script automates all that.

- Take optional exercise path argument
- Express all paths in terms of exercise path
- Pass further arguments through to cargo
- Use arrays to simplify file/dir preservation and reset
- Use trap to ensure reset occurs on ctrl-c
- Express check-exercises.sh in terms of bin/test-exercise
- Add debug tests to check-exercises in case of remote execution failure
- Move $![deny(warnings)] insertion into test-exercise.
   - Give the feature to people testing the exercise on its own
   - Clean up properly after
- Return the bitwise OR of individual exercises' return values

That last one wants some commenting. If a bunch of values fail in
different ways, this is of limited utility; your return code won't
be 0, and Travis will fail, but you won't get the info you expected
from the return codes. However, this does have some useful properties:
   - if a bunch of things fail in the same way, or only one thing
     fails, you get Cargo's return code back directly
   - you discard less overall information than the naïve implementation
     ( if [ $? != 0 ]; then return_code=1; fi )
   - always returns zero or non-zero appropriately without branching

Author: coriolinus

_test/check-exercises.sh

@@ -1,73 +1,55 @@
 #!/bin/bash
 
+# test for existence and executability of the test-exercise script
+# this depends on that
+if [ ! -f "./bin/test-exercise" ]; then
+   echo "bin/test-exercise does not exist"
+   exit 1
+fi
+if [ ! -x "./bin/test-exercise" ]; then
+   echo "bin/test-exercise does not have its executable bit set"
+   exit 1
+fi
+
 # In DENYWARNINGS mode, do not set -e so that we run all tests.
 # This allows us to see all warnings.
 if [ -z "$DENYWARNINGS" ]; then
     set -e
 fi
 
+# can't benchmark with a stable compiler; to bench, use
+# $ BENCHMARK=1 rustup run nightly _test/check-exercises.sh
 if [ -n "$BENCHMARK" ]; then
     files=exercises/*/benches
 else
     files=exercises/*/tests
 fi
 
-tmp=${TMPDIR:-/tmp/}
-mkdir "${tmp}exercises"
-
-exitcode=0
-
+return_code=0
 # An exercise worth testing is defined here as any top level directory with
 # a 'tests' directory
 for exercise in $files; do
-    # This assumes that exercises are only one directory deep
-    # and that the primary module is named the same as the directory
-    directory=$(dirname "${exercise}");
-
-    workdir=$(mktemp -d "${tmp}${directory}.XXXXXXXXXX")
-
-    cp -R -T $directory $workdir
-
-    # Run in subshell to change workdir without affecting current workdir.
-    (
-        cd $workdir
-
-        cp example.rs src/lib.rs
-
-        # Overwrite empty Cargo.toml if an example specific file exists
-        if [ -f Cargo-example.toml ]; then
-            cp Cargo-example.toml Cargo.toml
-        fi
-
-        # Forcibly strip all "ignore" statements from the testing files
-        for test in tests/*.rs; do
-            sed -i '/\[ignore\]/d' $test
-        done
-
-        # Run benchmarks instead of tests when enabled.
-        if [ -n "$BENCHMARK" ]; then
-            cargo bench
-        elif [ -n "$DENYWARNINGS" ]; then
-            sed -i -e '1i #![deny(warnings)]' src/lib.rs
-
-            # No-run mode so we see no test output.
-            # Quiet mode so we see no compile output
-            # (such as "Compiling"/"Downloading").
-            # Compiler errors will still be shown though.
-            # Both flags are necessary to keep things quiet.
-            cargo test --quiet --no-run
-        else
-            # Run the test and get the status
-            cargo test
-        fi
-    )
-
-    status=$?
-
-    if [ $status -ne 0 ]
-    then
-        exitcode=1
-    fi
+   # This assumes that exercises are only one directory deep
+   # and that the primary module is named the same as the directory
+   directory=$(dirname "${exercise}");
+
+   if [ -n "$DENYWARNINGS" ]; then
+      # No-run mode so we see no test output.
+      # Quiet mode so we see no compile output
+      # (such as "Compiling"/"Downloading").
+      # Compiler errors will still be shown though.
+      # Both flags are necessary to keep things quiet.
+      ./bin/test-exercise $directory --quiet --no-run
+      return_code=$(($return_code | $?))
+   else
+      # Run the test and get the status
+      # We use release mode here because, while it somewhat increases
+      # the compile time for all exercises, it substantially improves
+      # the runtime for certain exercises such as alphametics.
+      # Overall this should be an improvement.
+      ./bin/test-exercise $directory
+      return_code=$(($return_code | $?))
+   fi
 done
 
-exit $exitcode
+exit $return_code

bin/test-exercise

@@ -0,0 +1,84 @@
+#!/bin/bash
+# Test an exercise
+
+# which exercise are we testing right now?
+# if we were passed an argument, that should be the
+# exercise directory. Otherwise, assume we're in
+# it currently.
+if [ $# -ge 1 ]; then
+   exercise=$1
+   # if this script is called with arguments, it will pass through
+   # any beyond the first to cargo. Note that we can only get a
+   # free default argument if no arguments at all were passed,
+   # so if you are in the exercise directory and want to pass any
+   # arguments to cargo, you need to include the local path first.
+   # I.e. to test in release mode:
+   # $ test-exercise . --release
+   shift 1
+else
+   exercise='.'
+fi
+
+# what cargo command will we use?
+if [ -n "$BENCHMARK" ]; then
+   command="bench"
+else
+   command="test"
+fi
+
+declare -a preserve_files=("src/lib.rs" "Cargo.toml" "Cargo.lock")
+declare -a preserve_dirs=("tests")
+
+# reset instructions
+reset () {
+   for file in ${preserve_files[@]}; do
+      if [ -f "$exercise/$file.orig" ]; then
+         mv -f "$exercise/$file.orig" "$exercise/$file"
+      fi
+   done
+   for dir in ${preserve_dirs[@]}; do
+      if [ -d "$exercise/$dir.orig" ]; then
+         rm -rf "$exercise/$dir"
+         mv "$exercise/$dir.orig" "$exercise/$dir"
+      fi
+   done
+}
+
+# cause the reset to execute when the script exits normally or is killed
+trap reset EXIT INT TERM
+
+# preserve the files and directories we care about
+for file in ${preserve_files[@]}; do
+   if [ -f "$exercise/$file" ]; then
+      cp "$exercise/$file" "$exercise/$file.orig"
+   fi
+done
+for dir in ${preserve_dirs[@]}; do
+   if [ -d "$exercise/$dir" ]; then
+      cp -r "$exercise/$dir" "$exercise/$dir.orig"
+   fi
+done
+
+# Move example files to where Cargo expects them
+cp -f "$exercise/example.rs" "$exercise/src/lib.rs"
+if [ -f "$exercise/Cargo-example.toml" ]; then
+   cp -f "$exercise/Cargo-example.toml" "$exercise/Cargo.toml"
+fi
+
+# If deny warnings, insert a deny warnings compiler directive in the header
+if [ -n "$DENYWARNINGS" ]; then
+   sed -i -e '1i #![deny(warnings)]' "$exercise/src/lib.rs"
+fi
+
+# eliminate #[ignore] lines from tests
+for test in "$exercise/tests/*.rs"; do
+    sed -i '/\[ignore\]/d' $test
+done
+
+# run tests from within exercise directory
+# (use subshell so we auto-reset to current pwd after)
+(
+   cd $exercise
+   # this is the last command; its exit code is what's passed on
+   cargo $command "$@"
+)