Skip to content

Commit

Permalink
Cloudera GCS connector Parcel: copy current folder content into the p…
Browse files Browse the repository at this point in the history
…arcel (GoogleCloudPlatform#210)

* Copy all current folder content to the parcel

Fixes GoogleCloudPlatform#209 by copying not just the jar file, but all the files in the current directory. As per [the official guide](https://cloud.google.com/blog/products/storage-data-transfer/how-to-connect-clouderas-cdh-to-cloud-storage), the JSON file also has to be copied over to the cluster

* Update create_parcel.sh

* Copy just the json key file and JAR
  • Loading branch information
syvlabs authored and Jacob Ferriero committed Apr 2, 2019
1 parent 54c8d0b commit dc14b6f
Showing 1 changed file with 5 additions and 0 deletions.
5 changes: 5 additions & 0 deletions tools/cloudera-parcel-gcsconnector/create_parcel.sh
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ function main() {
mkdir -p ${BASEDIR}
LOGDIR=${HOME}/parcel-logs
mkdir -p ${LOGDIR}/
cp * ${BASEDIR}/
cd ${BASEDIR}
touch ${LOGDIR}/cparcel_error.log
touch ${LOGDIR}/cparcel_output.log
Expand Down Expand Up @@ -176,7 +177,11 @@ function main() {
graceful_exit
fi

# Copy GCS connector jar
cp gcs-connector-hadoop2-latest.jar ${PARCEL_FULLNAME}/lib/hadoop/lib/

# Copy service account key file
cp *.json ${PARCEL_FULLNAME}/lib/hadoop/lib/

# Check the existence of the GCS jar file in lib folder
[[ -f ${PARCEL_FULLNAME}/lib/hadoop/lib/gcs-connector-hadoop2-latest.jar ]] || GCSJAR_FLAG="0"
Expand Down

0 comments on commit dc14b6f

Please sign in to comment.