Error Building from Source: ... in proto_lang_toolchain rule @com_google_protobuf//:cc_toolchain: '@com_google_protobuf/

[dbl001]

I am trying to build tensorflow-text in a condo environment with an Apple Machine Learning version of Tensorflow
on Big Sur on an Intel iMac w/AMD Radeon GPU.

I am using bazelisk
E.g.
$ bazelisk build --enable_runfiles oss_scripts/pip_package:build_pip_package

 % ipython
Python 3.8.10 | packaged by conda-forge | (default, May 10 2021, 22:58:09) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.24.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import tensorflow

In [2]: print(tensorflow.__version__)
2.4.0-rc0

ERROR: /private/var/tmp/_bazel_davidlaxer/83b7c011c6dae2910ffe75493cc37f8b/external/com_google_protobuf/BUILD:981:21: in proto_lang_toolchain rule @com_google_protobuf//:cc_toolchain: '@com_google_protobuf/

 ./oss_scripts/run_build.sh
++ uname -s
+ osname=Darwin
+ [[ Darwin == \D\a\r\w\i\n ]]
+ sed -i '' 's/".so"/".dylib"/' tensorflow_text/tftext.bzl
+ sed -i '' 's/*.so/*.dylib/' oss_scripts/pip_package/MANIFEST.in
++ find tensorflow_text/python -type f
+ perl -pi -e 's/(load_library.load_op_library.*)\.so'\''/$1.dylib'\''/' tensorflow_text/python/keras/layers/__init__.py tensorflow_text/python/keras/layers/todense.py tensorflow_text/python/keras/layers/todense_test.py tensorflow_text/python/keras/__init__.py tensorflow_text/python/metrics/text_similarity_metric_ops.py tensorflow_text/python/metrics/__init__.py tensorflow_text/python/metrics/text_similarity_metric_ops_test.py tensorflow_text/python/__init__.py tensorflow_text/python/numpy/viterbi_decode.py tensorflow_text/python/numpy/__init__.py tensorflow_text/python/numpy/viterbi_decode_test.py tensorflow_text/python/benchmarks/ops_benchmarks.py tensorflow_text/python/benchmarks/test_data/uncased_L-12_H-768_A-12/vocab.txt tensorflow_text/python/benchmarks/__init__.py tensorflow_text/python/benchmarks/tokenizers_benchmarks.py tensorflow_text/python/benchmarks/benchmark_utils.py tensorflow_text/python/ops/string_ops.py tensorflow_text/python/ops/viterbi_constrained_sequence_op_test.py tensorflow_text/python/ops/sentencepiece_tokenizer.py tensorflow_text/python/ops/normalize_ops_test.py tensorflow_text/python/ops/trimmer_ops_test.py tensorflow_text/python/ops/masking_ops.py tensorflow_text/python/ops/pointer_ops.py tensorflow_text/python/ops/hub_module_tokenizer_test.py tensorflow_text/python/ops/regex_split_ops_test.py tensorflow_text/python/ops/bert_tokenizer_test.py tensorflow_text/python/ops/pad_model_inputs_ops.py tensorflow_text/python/ops/item_selector_ops.py tensorflow_text/python/ops/state_based_sentence_breaker_op.py tensorflow_text/python/ops/split_merge_from_logits_tokenizer.py tensorflow_text/python/ops/pad_along_dimension_op_test.py tensorflow_text/python/ops/segment_combiner_ops_test.py tensorflow_text/python/ops/unicode_char_tokenizer_test.py tensorflow_text/python/ops/test_data/test_wp_en_vocab.txt tensorflow_text/python/ops/test_data/test_oss_model.model tensorflow_text/python/ops/test_data/segmenter_hub_module/variables/variables.data-00000-of-00001 tensorflow_text/python/ops/test_data/segmenter_hub_module/variables/variables.index tensorflow_text/python/ops/test_data/segmenter_hub_module/saved_model.pb tensorflow_text/python/ops/test_data/segmenter_hub_module/tfhub_module.pb tensorflow_text/python/ops/wordpiece_tokenizer_test.py tensorflow_text/python/ops/ngrams_op_test.py tensorflow_text/python/ops/greedy_constrained_sequence_op_test.py tensorflow_text/python/ops/splitter.py tensorflow_text/python/ops/viterbi_constrained_sequence_op.py tensorflow_text/python/ops/__init__.py tensorflow_text/python/ops/split_merge_from_logits_tokenizer_test.py tensorflow_text/python/ops/tokenization.py tensorflow_text/python/ops/span_alignment_op_test.py tensorflow_text/python/ops/whitespace_tokenizer_test.py tensorflow_text/python/ops/split_merge_tokenizer.py tensorflow_text/python/ops/bert_tokenizer.py tensorflow_text/python/ops/hub_module_splitter_test.py tensorflow_text/python/ops/wordshape_ops_test.py tensorflow_text/python/ops/span_overlaps_op_test.py tensorflow_text/python/ops/sentence_breaking_ops.py tensorflow_text/python/ops/wordpiece_tokenizer.py tensorflow_text/python/ops/sliding_window_op_test.py tensorflow_text/python/ops/trimmer_ops.py tensorflow_text/python/ops/segment_combiner_ops.py tensorflow_text/python/ops/unicode_script_tokenizer.py tensorflow_text/python/ops/normalize_ops.py tensorflow_text/python/ops/hub_module_tokenizer.py tensorflow_text/python/ops/whitespace_tokenizer.py tensorflow_text/python/ops/mst_ops.py tensorflow_text/python/ops/state_based_sentence_breaker_op_test.py tensorflow_text/python/ops/sliding_window_op.py tensorflow_text/python/ops/sentence_breaking_ops_test.py tensorflow_text/python/ops/masking_ops_test.py tensorflow_text/python/ops/greedy_constrained_sequence_op.py tensorflow_text/python/ops/unicode_script_tokenizer_test.py tensorflow_text/python/ops/gather_with_default_op_test.py tensorflow_text/python/ops/split_merge_tokenizer_test.py tensorflow_text/python/ops/create_feature_bitmask_op.py tensorflow_text/python/ops/pad_along_dimension_op.py tensorflow_text/python/ops/create_feature_bitmask_op_test.py tensorflow_text/python/ops/ngrams_op.py tensorflow_text/python/ops/sentencepiece_tokenizer_test.py tensorflow_text/python/ops/item_selector_ops_test.py tensorflow_text/python/ops/regex_split_ops.py tensorflow_text/python/ops/coerce_to_valid_utf8_op_test.py tensorflow_text/python/ops/hub_module_splitter.py tensorflow_text/python/ops/unicode_char_tokenizer.py tensorflow_text/python/ops/wordshape_ops.py tensorflow_text/python/ops/pad_model_inputs_ops_test.py tensorflow_text/python/ops/mst_ops_test.py
+ export CC_OPT_FLAGS=-mavx
+ CC_OPT_FLAGS=-mavx
+ ./oss_scripts/configure.sh
darwin
Using installed tensorflow.
+ bazelisk build --enable_runfiles oss_scripts/pip_package:build_pip_package
DEBUG: /private/var/tmp/_bazel_davidlaxer/83b7c011c6dae2910ffe75493cc37f8b/external/org_tensorflow/third_party/repo.bzl:108:14: 
Warning: skipping import of repository 'icu' because it already exists.
DEBUG: Rule 'io_bazel_rules_docker' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1556410077 -0400"
DEBUG: Repository io_bazel_rules_docker instantiated at:
  /Users/davidlaxer/text/WORKSPACE:78:10: in <toplevel>
  /private/var/tmp/_bazel_davidlaxer/83b7c011c6dae2910ffe75493cc37f8b/external/org_tensorflow/tensorflow/workspace0.bzl:105:34: in workspace
  /private/var/tmp/_bazel_davidlaxer/83b7c011c6dae2910ffe75493cc37f8b/external/bazel_toolchains/repositories/repositories.bzl:37:23: in repositories
Repository rule git_repository defined at:
  /private/var/tmp/_bazel_davidlaxer/83b7c011c6dae2910ffe75493cc37f8b/external/bazel_tools/tools/build_defs/repo/git.bzl:199:33: in <toplevel>
WARNING: /private/var/tmp/_bazel_davidlaxer/83b7c011c6dae2910ffe75493cc37f8b/external/local_config_tf/BUILD:10:11: in linkstatic attribute of cc_library rule @local_config_tf//:libtensorflow_framework: setting 'linkstatic=1' is recommended if there are no object files
ERROR: /private/var/tmp/_bazel_davidlaxer/83b7c011c6dae2910ffe75493cc37f8b/external/com_google_protobuf/BUILD:981:21: in proto_lang_toolchain rule @com_google_protobuf//:cc_toolchain: '@com_google_protobuf//:cc_toolchain' does not have mandatory provider 'ProtoInfo'.
ERROR: Analysis of target '//oss_scripts/pip_package:build_pip_package' failed; build aborted: Analysis of target '@com_google_protobuf//:cc_toolchain' failed
INFO: Elapsed time: 0.250s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (2 packages loaded, 121 targets co\
nfigured)
    Fetching @com_google_sentencepiece; fetching
(tensorflow_mac) davidlaxer@x86_64-apple-darwin13 text % 

[broken]

I think this error appears when using bazel 4.0.0 (bazelbuild/bazel#12887). We currently build with Bazel 3.7.2. Does changing that help?

[dbl001]

With Bazel 3.7.2 and clang:
% clang --version
Apple clang version 12.0.5 (clang-1205.0.22.9)
Target: x86_64-apple-darwin20.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

ERROR: /Users/davidlaxer/text/tensorflow_text/core/kernels/BUILD:355:23: C++ compilation of rule '//tensorflow_text/core/kernels:sentencepiece_kernels' failed (Exit 1): cc_wrapper.sh failed: error executing command external/local_config_cc/cc_wrapper.sh -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -fcolor-diagnostics -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG ... (remaining 53 argument(s) skipped)
tensorflow_text/core/kernels/sentencepiece_kernels.cc:70:65: error: only virtual member functions can be marked 'override'
  Status AsGraphDef(GraphDefBuilder* builder, Node** out) const override {

[dbl001]

In resource_mgr.h Tensorflow 2.4.1 the class ResourceBase : public core::RefCounted
AsGraphDef is not defined as it in Tensorflow 2.5.0 :

virtual Status AsGraphDef(GraphDefBuilder* builder, Node** out) const { return errors::Unimplemented("AsGraphDef not implemented for resource ", DebugString()); }

The Apple Machine Learning version of tensorflow with GPU support for AMD Radeon is based upon tensorflow 2.4.1.
Any ideas?

[dbl001]

I cloned the 2.4 tensorflow_text branch and built with bazel 3.7.2

In [1]: import tensorflow_text
---------------------------------------------------------------------------
NotFoundError                             Traceback (most recent call last)
<ipython-input-1-4dd33f1b1ed3> in <module>
----> 1 import tensorflow_text

~/text/tensorflow_text/__init__.py in <module>
     19 # pylint: disable=wildcard-import
     20 from tensorflow_text.python import keras
---> 21 from tensorflow_text.python import metrics
     22 from tensorflow_text.python.ops import *
     23 

~/text/tensorflow_text/python/metrics/__init__.py in <module>
     18 
     19 # pylint: disable=wildcard-import
---> 20 from tensorflow_text.python.metrics.text_similarity_metric_ops import *
     21 
     22 # Public symbols in the "tensorflow_text.metrics" package.

~/text/tensorflow_text/python/metrics/text_similarity_metric_ops.py in <module>
     26 from tensorflow.python.framework import load_library
     27 from tensorflow.python.platform import resource_loader
---> 28 gen_text_similarity_metric_ops = load_library.load_op_library(resource_loader.get_path_to_datafile('_text_similarity_metric_ops.dylib'))
     29 
     30 

~/anaconda3/envs/tensorflow_mac/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py in load_op_library(library_filename)
     55     RuntimeError: when unable to load the library or get the python wrappers.
     56   """
---> 57   lib_handle = py_tf.TF_LoadLibrary(library_filename)
     58   try:
     59     wrappers = _pywrap_python_op_gen.GetPythonWrappers(

NotFoundError: dlopen(/Users/davidlaxer/text/tensorflow_text/python/metrics/_text_similarity_metric_ops.dylib, 6): Symbol not found: __ZN10tensorflow15TensorShapeBaseINS_11TensorShapeEEC2EN4absl14lts_2020_09_234SpanIKxEE
  Referenced from: /Users/davidlaxer/text/tensorflow_text/python/metrics/_text_similarity_metric_ops.dylib
  Expected in: /Users/davidlaxer/anaconda3/envs/tensorflow_mac/lib/python3.8/site-packages/tensorflow/python/../libtensorflow_framework.2.dylib
 in /Users/davidlaxer/text/tensorflow_text/python/metrics/_text_similarity_metric_ops.dylib

#160

[dbl001]

I copied these shared library files from the Apple Tensorflow 2.4.1 environment libraries:

E.g. /Users/davidlaxer/anaconda3/envs/tensorflow_mac/lib/python3.8/site-packages/tensorflow_text/python/ops/*.dylib

ls -l /Users/davidlaxer/text/tensorflow_text/python/ops/*.dylib
-rwxr-xr-x  1 davidlaxer  staff   137248 Jun 12 14:05 /Users/davidlaxer/text/tensorflow_text/python/ops/_constrained_sequence_op.dylib
-rwxr-xr-x  1 davidlaxer  staff   134760 Jun 12 14:05 /Users/davidlaxer/text/tensorflow_text/python/ops/_mst_ops.dylib
-rwxr-xr-x  1 davidlaxer  staff  2267592 Jun 12 14:05 /Users/davidlaxer/text/tensorflow_text/python/ops/_normalize_ops.dylib
-rwxr-xr-x  1 davidlaxer  staff   563648 Jun 12 14:05 /Users/davidlaxer/text/tensorflow_text/python/ops/_regex_split_ops.dylib
-rwxr-xr-x  1 davidlaxer  staff   969352 Jun 12 14:05 /Users/davidlaxer/text/tensorflow_text/python/ops/_sentence_breaking_ops.dylib
-rwxr-xr-x  1 davidlaxer  staff  2273704 Jun 12 14:05 /Users/davidlaxer/text/tensorflow_text/python/ops/_sentencepiece_tokenizer.dylib
-rwxr-xr-x  1 davidlaxer  staff   310264 Jun 12 14:05 /Users/davidlaxer/text/tensorflow_text/python/ops/_split_merge_from_logits_tokenizer.dylib
-rwxr-xr-x  1 davidlaxer  staff   310840 Jun 12 14:05 /Users/davidlaxer/text/tensorflow_text/python/ops/_split_merge_tokenizer.dylib
-rwxr-xr-x  1 davidlaxer  staff   722216 Jun 12 14:05 /Users/davidlaxer/text/tensorflow_text/python/ops/_state_based_sentence_breaker_op.dylib
-rwxr-xr-x  1 davidlaxer  staff   255656 Jun 12 14:05 /Users/davidlaxer/text/tensorflow_text/python/ops/_unicode_script_tokenizer.dylib
-rwxr-xr-x  1 davidlaxer  staff   235976 Jun 12 14:05 /Users/davidlaxer/text/tensorflow_text/python/ops/_whitespace_tokenizer.dylib
-rwxr-xr-x  1 davidlaxer  staff   129008 Jun 12 14:05 /Users/davidlaxer/text/tensorflow_text/python/ops/_wordpiece_tokenizer.dylib

Tensorflow_text is running.

[broken]

Great! I'm happy to hear you got this running.

We will be looking at trying to add the new mac targets soon.

[broken]

Also, thanks for including your steps and solution for others.

[awav]

@broken,

Hi, I would like to use tf-text with latest tensorflow build (from source) on Linux. However, I'm getting similar error:

(py37) artem@blabla:~/code/text$ ./oss_scripts/run_build.sh
++ uname -s
+ osname=Linux
+ [[ Linux == \D\a\r\w\i\n ]]
+ ./oss_scripts/configure.sh
linux
WARNING: Package(s) not found: tensorflow
Using installed tensorflow.
+ bazel build --enable_runfiles oss_scripts/pip_package:build_pip_package
Starting local Bazel server and connecting to it...
DEBUG: /1TB/artem/.cache/bazel/_bazel_artem/513dad259ce460b27c1f370613b4b9a5/external/org_tensorflow/third_party/repo.bzl:109:14:
Warning: skipping import of repository 'icu' because it already exists.
DEBUG: /1TB/artem/.cache/bazel/_bazel_artem/513dad259ce460b27c1f370613b4b9a5/external/org_tensorflow/third_party/repo.bzl:109:14:
Warning: skipping import of repository 'pybind11' because it already exists.
DEBUG: /1TB/artem/.cache/bazel/_bazel_artem/513dad259ce460b27c1f370613b4b9a5/external/tf_runtime/third_party/cuda/dependencies.bzl:51:10: The following command will download NVIDIA proprietary software. By using the software you agree to comply with the terms of the license agreement that accompanies the software. If you do not agree to the terms of the license agreement, do not use the software.
WARNING: /1TB/artem/.cache/bazel/_bazel_artem/513dad259ce460b27c1f370613b4b9a5/external/local_config_tf/BUILD:10:11: in linkstatic attribute of cc_library rule @local_config_tf//:libtensorflow_framework: setting 'linkstatic=1' is recommended if there are no object files
ERROR: /1TB/artem/.cache/bazel/_bazel_artem/513dad259ce460b27c1f370613b4b9a5/external/com_google_protobuf/BUILD:981:21: in proto_lang_toolchain rule @com_google_protobuf//:cc_toolchain: '@com_google_protobuf//:cc_toolchain' does not have mandatory provider 'ProtoInfo'.
INFO: Repository com_google_sentencepiece instantiated at:
  /home/artem/code/text/WORKSPACE:5:13: in <toplevel>
Repository rule http_archive defined at:
  /1TB/artem/.cache/bazel/_bazel_artem/513dad259ce460b27c1f370613b4b9a5/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
ERROR: Analysis of target '//oss_scripts/pip_package:build_pip_package' failed; build aborted: Analysis of target '@com_google_protobuf//:cc_toolchain' failed
INFO: Elapsed time: 4.845s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (61 packages loaded, 2040 targets configured)

Linux blabla 5.4.0-81-generic #91~18.04.1-Ubuntu SMP Fri Jul 23 13:36:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[awav]

Fixed,

1. created text/.bazelversion with 3.7.2
2. replaced bazel call to bazelisk in run_build.sh
3. had to deal with wrong version of the wheel. Steps from The build system could use some ❤️  #717 (comment) helped

[jhkang-illunex]

on lastest version

prepare_tf_dep.sh had som issue on m1 apple

sed -i "s/project_version = 'REPLACE_ME'/project_version = '${tf_version}'/" oss_scripts/pip_package/setup.nightly.py

short_commit_sha=$($installed_python -c 'import tensorflow as tf; print(tf.git_version)' | tail -1 | grep -oP '(?<=-g)[0-9a-f]*$')

and other line..

that shell run on linux but apple cannot run..

[broken]

You should be able to safely skip running it, and I'll update the build script to skip over running it in the m1 case.

The prepare_tf_dep script's main purpose is to build against the exact version of TF you have installed and set the package version for tf-text. However, tensorflow-macos does not set the git_version used by the script, does not release nightlies, and builds against it aren't for pypi uploaded packages, so it shouldn't be needed for these cases.