Skip to content

Commit

Permalink
Optimize toolchain size by preserving TensorFlow library symlinks. (s…
Browse files Browse the repository at this point in the history
…wiftlang#27032)

Bazel produces multiple TensorFlow library artifacts:
```
$ ls -alh tensorflow/bazel-bin/tensorflow
 18B libtensorflow.so -> libtensorflow.so.1
 23B libtensorflow.so.1 -> libtensorflow.so.1.14.0
277M libtensorflow.so.1.14.0
```

Previously, TensorFlow libraries were copied using `cp -a` via a glob
pattern. For some reason, symlinks were not preserved:
```
$ ls -alh <TOOLCHAIN_BEFORE>.xctoolchain/usr/lib/swift/macosx
 18B libtensorflow.so -> libtensorflow.so.1
277M libtensorflow.so.1
277M libtensorflow.so.1.14.0 # duplicate library
```

Now, rather than copying all libraries via glob,
`copy_file_preserving_symlinks` copies exactly the necessary source
files to the destination while preserving symlinks:
```
$ ls -alh <TOOLCHAIN_AFTER>.xctoolchain/usr/lib/swift/macosx
 18B libtensorflow.so -> libtensorflow.so.1
277M libtensorflow.so.1
```

This combined with removing libtensorflow_framework.so dependency
(swiftlang#27029) led to a macOS toolchain
size reduction from 4.91 GB to 3.45 GB.
  • Loading branch information
dan-zheng authored and pschuh committed Sep 6, 2019
1 parent 0d6cb12 commit 2de0021
Showing 1 changed file with 27 additions and 3 deletions.
30 changes: 27 additions & 3 deletions utils/build-script-impl
Original file line number Diff line number Diff line change
Expand Up @@ -393,6 +393,27 @@ function is_swift_lto_enabled() {
fi
}

# SWIFT_ENABLE_TENSORFLOW
# Copy a file to a destination directory.
# If the file is a symbolic link, then copy the underlying file to the
# destination and create a new symlink. Otherwise, copy the file directly.
function copy_file_preserving_symlinks() {
# $1: source directory.
# $2: source file basename.
# $3: destination directory.

# If source file is a symlink, then copy the underlying file to the
# destination and create a new symlink.
if [[ -L "$1/$2" ]]; then
local target=`readlink "$1/$2"`
copy_file_preserving_symlinks "$1" "$target" "$3"
ln -sv "$3/$target" -f -r "$3/$2"
# Otherwise, directly copy the source file to the destination.
else
cp -v "$1/$2" "$3"
fi
}

# Support for performing isolated actions.
#
# This is part of refactoring more work to be done or controllable via
Expand Down Expand Up @@ -4014,9 +4035,12 @@ for host in "${ALL_HOSTS[@]}"; do
mkdir -p "${TF_DEST_DIR}"
for lib_name in tensorflow; do
lib=".*lib${lib_name}.so[0-9.]*"
dylib=".*lib${lib_name}[0-9.]*.dylib"
find "${TF_LIBDIR}" \( -regex "${lib}" -o -regex "${dylib}" \) -exec echo "{} => ${TF_DEST_DIR}" \;
find "${TF_LIBDIR}" \( -regex "${lib}" -o -regex "${dylib}" \) -exec cp -a {} "${TF_DEST_DIR}" \;
# Wipe existing TensorFlow libraries in the destination
# directory. This ensures that old versions of libraries
# are not copied, taking up unnecessary space.
rm -rf "${TF_DEST_DIR}/${lib}"
# Copy TensorFlow library, preserving symlinks.
copy_file_preserving_symlinks "${TF_LIBDIR}" "lib${lib_name}.so" "${TF_DEST_DIR}"
done
continue
;;
Expand Down

0 comments on commit 2de0021

Please sign in to comment.