Skip to content

Remove dependency on libtensorflow_framework.so. #27029

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

dan-zheng
Copy link
Contributor

Remove libtensorflow_framework.so as a build product.
This should reduce toolchain size.

Follow-up attempt to #20898.

Remove libtensorflow_framework.so as a build product.
This should reduce toolchain size.
@dan-zheng dan-zheng added the tensorflow This is for "tensorflow" branch PRs. label Sep 4, 2019
@dan-zheng dan-zheng requested a review from pschuh September 4, 2019 21:41
@dan-zheng
Copy link
Contributor Author

@swift-ci Please test tensorflow

@dan-zheng
Copy link
Contributor Author

@swift-ci Please clean test tensorflow

@dan-zheng
Copy link
Contributor Author

Please hold off on merging until toolchain building is tested locally.
I'm testing macOS toolchain building locally - can't test Ubuntu toolchain building since I'm currently remote.

@dan-zheng
Copy link
Contributor Author

Toolchain building succeeded on macOS.
Exact toolchain size decrease is not clear because I tested with another WIP patch - will post an update on my next patch.

@dan-zheng dan-zheng merged commit d67c913 into swiftlang:tensorflow Sep 5, 2019
@dan-zheng dan-zheng deleted the remove-tf-framework-library branch September 5, 2019 01:36
dan-zheng added a commit to dan-zheng/swift that referenced this pull request Sep 5, 2019
Bazel produces multiple TensorFlow library artifacts:
```
$ ls -alh tensorflow/bazel-bin/tensorflow
 18B libtensorflow.so -> libtensorflow.so.1
 23B libtensorflow.so.1 -> libtensorflow.so.1.14.0
277M libtensorflow.so.1.14.0
```

Previously, TensorFlow libraries were copied using `cp -a` via a glob
pattern. For some reason, symlinks were not preserved:
```
$ ls -alh <TOOLCHAIN_BEFORE>.xctoolchain/usr/lib/swift/macosx
 18B libtensorflow.so -> libtensorflow.so.1
277M libtensorflow.so.1
277M libtensorflow.so.1.14.0 # duplicate library
```

Now, rather than copying all libraries via glob,
`copy_file_preserving_symlinks` copies exactly the necessary source
files to the destination while preserving symlinks:
```
$ ls -alh <TOOLCHAIN_AFTER>.xctoolchain/usr/lib/swift/macosx
 18B libtensorflow.so -> libtensorflow.so.1
277M libtensorflow.so.1
```

This combined with removing libtensorflow_framework.so dependency
(swiftlang#27029) led to a macOS toolchain
size reduction from 4.91 GB to 3.45 GB.
pschuh pushed a commit that referenced this pull request Sep 6, 2019
…27032)

Bazel produces multiple TensorFlow library artifacts:
```
$ ls -alh tensorflow/bazel-bin/tensorflow
 18B libtensorflow.so -> libtensorflow.so.1
 23B libtensorflow.so.1 -> libtensorflow.so.1.14.0
277M libtensorflow.so.1.14.0
```

Previously, TensorFlow libraries were copied using `cp -a` via a glob
pattern. For some reason, symlinks were not preserved:
```
$ ls -alh <TOOLCHAIN_BEFORE>.xctoolchain/usr/lib/swift/macosx
 18B libtensorflow.so -> libtensorflow.so.1
277M libtensorflow.so.1
277M libtensorflow.so.1.14.0 # duplicate library
```

Now, rather than copying all libraries via glob,
`copy_file_preserving_symlinks` copies exactly the necessary source
files to the destination while preserving symlinks:
```
$ ls -alh <TOOLCHAIN_AFTER>.xctoolchain/usr/lib/swift/macosx
 18B libtensorflow.so -> libtensorflow.so.1
277M libtensorflow.so.1
```

This combined with removing libtensorflow_framework.so dependency
(#27029) led to a macOS toolchain
size reduction from 4.91 GB to 3.45 GB.
dan-zheng added a commit that referenced this pull request Sep 18, 2019
Remove libtensorflow_framework.so as a build product.
This reduces toolchain size.

Follow-up attempt to #20898.
dan-zheng added a commit to dan-zheng/swift that referenced this pull request Sep 18, 2019
Bazel produces multiple TensorFlow library artifacts:
```
$ ls -alh tensorflow/bazel-bin/tensorflow
 18B libtensorflow.so -> libtensorflow.so.1
 23B libtensorflow.so.1 -> libtensorflow.so.1.14.0
277M libtensorflow.so.1.14.0
```

Previously, TensorFlow libraries were copied via `cp -p`.
This did not preserve symlinks, leading to library duplication:
```
$ ls -alh <TOOLCHAIN_BEFORE>.xctoolchain/usr/lib/swift/macosx
 18B libtensorflow.so -> libtensorflow.so.1
277M libtensorflow.so.1
277M libtensorflow.so.1.14.0 # duplicate library
```

Now, `cp -a` is used, which preserves symlinks:
```
$ ls -alh <TOOLCHAIN_AFTER>.xctoolchain/usr/lib/swift/macosx
 18B libtensorflow.so -> libtensorflow.so.1
277M libtensorflow.so.1
```

This combined with removing libtensorflow_framework.so dependency
(swiftlang#27029) led to a macOS toolchain
size reduction from 3.99 GB to 3.62 GB on the tensorflow-0.5 branch.
dan-zheng added a commit that referenced this pull request Sep 18, 2019
)

Bazel produces multiple TensorFlow library artifacts:
```
$ ls -alh tensorflow/bazel-bin/tensorflow
 18B libtensorflow.so -> libtensorflow.so.1
 23B libtensorflow.so.1 -> libtensorflow.so.1.14.0
277M libtensorflow.so.1.14.0
```

Previously, TensorFlow libraries were copied via `cp -p`.
This did not preserve symlinks, leading to library duplication:
```
$ ls -alh <TOOLCHAIN_BEFORE>.xctoolchain/usr/lib/swift/macosx
 18B libtensorflow.so -> libtensorflow.so.1
277M libtensorflow.so.1
277M libtensorflow.so.1.14.0 # duplicate library
```

Now, `cp -a` is used, which preserves symlinks:
```
$ ls -alh <TOOLCHAIN_AFTER>.xctoolchain/usr/lib/swift/macosx
 18B libtensorflow.so -> libtensorflow.so.1
277M libtensorflow.so.1
```

This combined with removing libtensorflow_framework.so dependency
(#27029) led to a macOS toolchain
size reduction from 3.99 GB to 3.62 GB on the tensorflow-0.5 branch.
dan-zheng added a commit that referenced this pull request Sep 18, 2019
)

Bazel produces multiple TensorFlow library artifacts:
```
$ ls -alh tensorflow/bazel-bin/tensorflow
 18B libtensorflow.so -> libtensorflow.so.1
 23B libtensorflow.so.1 -> libtensorflow.so.1.14.0
277M libtensorflow.so.1.14.0
```

Previously, TensorFlow libraries were copied via `cp -p`.
This did not preserve symlinks, leading to library duplication:
```
$ ls -alh <TOOLCHAIN_BEFORE>.xctoolchain/usr/lib/swift/macosx
 18B libtensorflow.so -> libtensorflow.so.1
277M libtensorflow.so.1
277M libtensorflow.so.1.14.0 # duplicate library
```

Now, `cp -a` is used, which preserves symlinks:
```
$ ls -alh <TOOLCHAIN_AFTER>.xctoolchain/usr/lib/swift/macosx
 18B libtensorflow.so -> libtensorflow.so.1
277M libtensorflow.so.1
```

This combined with removing libtensorflow_framework.so dependency
(#27029) led to a macOS toolchain
size reduction from 3.99 GB to 3.62 GB on the tensorflow-0.5 branch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tensorflow This is for "tensorflow" branch PRs.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants