Wide-deep example shutting down

[ksy3395]

Hello,
I am trying to run Wide-deep example with Spark standalone cluster on my Mac and also on a ubuntu machine separately but it keeps shutting down the spark job, so I can't check anything since there is no error or log. I tried mnist stanalone example, but I didn't have any problem on these machines. Is there anything else I can check if there is anything wrong?

Command:
${SPARK_HOME}/bin/spark-submit --master spark://192.168.1.9:7077 --py-files census_dataset.py, wide_deep_run_loop.py --conf spark.cores.max=3 --conf spark.task.cpus=1 --conf spark.executorEnv.JAVA_HOME="$JAVA_HOME" --conf spark.task.maxFailures=1 --conf spark.stage.maxConsecutiveAttempts=1 census_main.py --cluster_size 3

2019-02-18 01:55:26 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-02-18 01:55:28 INFO ShutdownHookManager:54 - Shutdown hook called
2019-02-18 01:55:28 INFO ShutdownHookManager:54 - Deleting directory /private/var/folders/98/34ltsy7x4f5btv6bpt6d75xm0000gn/T/spark-7cf40b10-9420-4680-8059-fed7d4ae4930
2019-02-18 01:55:28 INFO ShutdownHookManager:54 - Deleting directory /private/var/folders/98/34ltsy7x4f5btv6bpt6d75xm0000gn/T/localPyFiles-75adab00-d652-47e4-92c5-b12dcb07ca99

[leewyang]

Try reducing your cluster_size to 2 or maybe even 1... (I'm wondering if it's just a memory/resource issue). Also, see this stack overflow about how to view/retrieve logs for spark standalone clusters...

[ksy3395]

I have tried to reduce the cluster size to 2 and 1 and also tried to reduce conf spark.cores.max. to 2 and 1. And there is no log on $SPARK_HOME/logs as well as on the webUI. There is no application created on WebUI.

I tested with two Ubuntu machines workers with hdfs (each got two cores and 2.9 gb memory on Spark WebUI), and I also tried it only on a Mac machine worker which has 4 cores and 7gb memory on Spark Web UI, but I got the same problem.

[leewyang]

Try this: ls -ltr ${SPARK_HOME}/work... you should see folders for each of your application attempts, with the latest attempt at the bottom.

[ksy3395]

Thank you for your reply. I tried it but there was no log application created or application attempt.

[leewyang]

@ksy3395 not sure what's going on, but here are a couple suggestions:

1. I've updated the README for this example to add some instructions from the tensorflow/models repo that I had left out before... these are important to ensure that some of the tensorflow/models/official library functions are available to your Spark job. Please make sure you have run these steps BEFORE starting your local Spark Standalone cluster.

2. Retry the mnist standalone example, and see if you can A) view the executor logs in the Spark UI while the job is running, and B) find the executor logs on disk after the job has completed. If you can see and find these logs, then the wide-deep example logs should be at a similar location.

3. The fact that your ${SPARK_HOME}/work directory is empty is a bit unusual, since that's the standard location for log files. You may want to try re-installing Spark (possibly on another machine).

4. Last of all, if nothing else works... you can send me your Spark driver logs (to leewyang@gmail.com, if you don't want to post publicly), and I can see if there's any useful info there...

[leewyang]

Closing due to inactivity. Feel free to reopen if this is still an issue.