Skip to content

Commit

Permalink
Add handy message for how to diff manually instead of using meld since (
Browse files Browse the repository at this point in the history
apache#18)

it's hard to install meld on mac. Also, wget may be missing on mac so
auto-install that during automatic spark-download.

Fix the first-regtest-setup case by updating SPARK_HOME after download.
  • Loading branch information
sfc-gh-dhuo committed Apr 30, 2024
1 parent 8775ff1 commit 6ae2eed
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 1 deletion.
3 changes: 2 additions & 1 deletion regtests/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,8 @@ for TEST_FILE in ${TEST_LIST}; do
echo '#!/bin/bash' > ${TEST_TMPDIR}/${TEST_SHORTNAME}.fixdiffs.sh
echo "meld ${TEST_STDOUT} ${TEST_REF}" >> ${TEST_TMPDIR}/${TEST_SHORTNAME}.fixdiffs.sh
chmod 750 ${TEST_TMPDIR}/${TEST_SHORTNAME}.fixdiffs.sh
logred "To compare and fix diffs: ${TEST_TMPDIR}/${TEST_SHORTNAME}.fixdiffs.sh"
logred "To compare and fix diffs (if 'meld' installed): ${TEST_TMPDIR}/${TEST_SHORTNAME}.fixdiffs.sh"
logred "Or manually diff: diff ${TEST_STDOUT} ${TEST_REF}"
logred "See stderr from test run for additional diagnostics: ${TEST_STDERR}"
NUM_FAILURES=$(( NUM_FAILURES + 1 ))
fi
Expand Down
10 changes: 10 additions & 0 deletions regtests/setup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,18 +10,28 @@ set -x
SPARK_HOME=$(realpath ~/spark-3.5.1-bin-hadoop3-scala2.13)
SPARK_CONF="${SPARK_HOME}/conf/spark-defaults.conf"


# Ensure binaries are downloaded locally
echo 'Verifying Spark binaries...'
if ! [ -f ${SPARK_HOME}/bin/spark-sql ]; then
echo 'Setting up Spark...'
if ! [ -f ~/spark-3.5.1-bin-hadoop3-scala2.13.tgz ]; then
echo 'Downloading spark distro...'
wget -O ~/spark-3.5.1-bin-hadoop3-scala2.13.tgz https://dlcdn.apache.org/spark/spark-3.5.1/spark-3.5.1-bin-hadoop3-scala2.13.tgz
if ! [ -f ~/spark-3.5.1-bin-hadoop3-scala2.13.tgz ]; then
if [[ "${OSTYPE}" == "darwin"* ]]; then
echo "Detected OS: mac. Running 'brew install wget' to try again."
brew install wget
wget -O ~/spark-3.5.1-bin-hadoop3-scala2.13.tgz https://dlcdn.apache.org/spark/spark-3.5.1/spark-3.5.1-bin-hadoop3-scala2.13.tgz
fi
fi
else
echo 'Found existing Spark tarball'
fi
tar xzvf ~/spark-3.5.1-bin-hadoop3-scala2.13.tgz -C ~
echo 'Done!'
SPARK_HOME=$(realpath ~/spark-3.5.1-bin-hadoop3-scala2.13)
SPARK_CONF="${SPARK_HOME}/conf/spark-defaults.conf"
else
echo 'Verified Spark distro already installed.'
fi
Expand Down

0 comments on commit 6ae2eed

Please sign in to comment.