Skip to content

Cloudera GCS connector parcel script does not copy directory files #209

Closed
@syvlabs

Description

When running the script, files in the same directory (e.g. the JSON key) are not copied over to the parcel folder.

In the following script:
professional-services/tools/cloudera-parcel-gcsconnector/create_parcel.sh

  # Download gcs connector jar and copy all folder content to parcel location
  # Set flag for further parcel file existence validation
  curl -o gcs-connector-hadoop2-latest.jar --fail ${GCSJAR_LINK} || GCSJAR_FLAG="0"

  # Validate if package downloaded properly
  if [[ ${GCSJAR_FLAG} = "0" ]]; then
    echo "Error: hadoop connector failed to download, check network connectivity or file/folder permissions"
    graceful_exit
  fi

  cp gcs-connector-hadoop2-latest.jar ${PARCEL_FULLNAME}/lib/hadoop/lib/

This part of the script only copies the jar file, but not other files in the folder.

I noticed that the correct code was removed in this commit, specifically:

##Download gcs connector jar and copy all folder content to parcel location
wget https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-hadoop2-latest.jar
if [[ $? -ne 0 ]]; then
   echo "Download GCS connector failed!"
   exit 1
fi

cp * ${filen^^}-$version/lib/hadoop/lib/

## Create parcel.json file required for parcel packaging
cat >>${filen^^}-$version/meta/parcel.json<< EOL

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions