[#7092] [#10046] [#10222] [#10224] [#10230] [#10251] [#10295] Enable …

…Clang 12 ASAN build on AlmaLinux 8 and fix relevant bugs Summary: Updating third-party dependencies from 44ba3719652858e1f7816df87b25101e09527c88 to 2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd (updating the LLVM toolchain with a bug fix, and also picking up the program_options Boost library addition by @mikhpolitov, commit yugabyte/yugabyte-db-thirdparty@2d282c3). yugabyte-db-thirdparty diff link: https://github.com/yugabyte/yugabyte-db-thirdparty/compare/44ba3719652858e1f7816df87b25101e09527c88..2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd Enabling the Clang 12 ASAN build on AlmaLinux 8 and fixing these bugs from that build: - #7092 - pick up an updated version of LLVM where libunwind has been patched to work around crashing with an unknown x86_64 register error. - #10046 - suppress harmless undefined behavior in gflags.cc. - #10222 - increase the timeout for waiting for tablet server registration in mini cluster. - #10224 - when looping to wait for leader/follower of a tablet in TestUpdateLagMetrics, reset tablet server pointers to nullptr at every iteration. - #10230, #10251 - use posix_spawn on Linux to create subprocesses to avoid getting stuck in the child process when allocating memory when setting environment variables because some memory allocation lock is already held by the parent process. On macOS, we still use fork+exec because posix_spawnp throws errors related to closing file descriptors. Making sure all files in our code are opened with FD_CLOEXEC is out of scope of this revision, and will be addressed by #10321. - #10295 - fix undefined behavior (adding to a null pointer) in a Postgres sorting function. - Fix HostPort& field type in ysql_upgrade.h (make it HostPort). Otherwise there is an ASAN issue accessing a deallocated stack value. (No separate GitHub issue for this bug.) Also consolidating Status generation from a C standard library error number in errno.h and errno.c, and adding a few utility macros for convenient invocation of C functions that either return an errno as the return value, or return zero vs. non zero depending on whether there is an error, and set errno as a side effect. As part of this diff, we are removing Clang 7 ASAN build on CentOS 7 from Jenkins (we don't need two different ASAN builds). Test Plan: Jenkins Reviewers: bogdan, steve.varnau, sergei Reviewed By: sergei Subscribers: ybase Differential Revision: https://phabricator.dev.yugabyte.com/D13301
yugabyte · Oct 30, 2021 · 3a45d71 · 3a45d71
1 parent 03b4f3e
commit 3a45d71
Show file tree

Hide file tree

Showing 31 changed files with 806 additions and 310 deletions.
diff --git a/build-support/common-build-env.sh b/build-support/common-build-env.sh
@@ -183,6 +183,8 @@ readonly -a VALID_COMPILER_TYPES=(
   gcc8
   gcc9
   gcc10
+  gcc11
+  gcc12
   clang
   clang7
   clang8

diff --git a/build-support/lsan-suppressions.txt b/build-support/lsan-suppressions.txt
@@ -40,3 +40,7 @@ leak:YbgPrepareMemoryContext
 
 # https://gist.githubusercontent.com/mbautin/015cb594c8281e1afc7ee7b3b5230fce/raw
 leak:__strdup
+
+# https://gist.githubusercontent.com/mbautin/864e341c578f30b478da93f59cf098cc/raw
+leak:guc_strdup
+leak:postmaster_strdup
diff --git a/build-support/run_tests_on_spark.py b/build-support/run_tests_on_spark.py
@@ -213,6 +213,16 @@ def log_heading(msg: str) -> None:
     logging.info('\n%s\n%s\n%s' % ('-' * 80, msg, '-' * 80))
 
 
+def find_python_interpreter() -> str:
+    # We are not using "which" here because we don't want to pick up the python3 script inside the
+    # virtualenv's bin directory.
+    candidates = ['/usr/local/bin/python3', '/usr/bin/python3']
+    for python_interpreter_path in candidates:
+        if os.path.isfile(python_interpreter_path):
+            return python_interpreter_path
+    raise ValueError("Could not find Python interpreter at any of the paths: %s" % candidates)
+
+
 # Initializes the spark context. The details list will be incorporated in the Spark application
 # name visible in the Spark web UI.
 def init_spark_context(details: List[str] = []) -> None:
@@ -247,7 +257,9 @@ def init_spark_context(details: List[str] = []) -> None:
     if 'BUILD_URL' in os.environ:
         details.append('URL: {}'.format(os.environ['BUILD_URL']))
 
-    SparkContext.setSystemProperty("spark.pyspark.python", "/usr/local/bin/python3")
+    python_interpreter = find_python_interpreter()
+    logging.info("Using this Python interpreter for Spark: %s", python_interpreter)
+    SparkContext.setSystemProperty("spark.pyspark.python", python_interpreter)
     spark_context = SparkContext(spark_master_url, "YB tests: {}".format(' '.join(details)))
     yb_python_zip_path = yb_dist_tests.get_tmp_filename(
             prefix='yb_python_module_for_spark_workers_', suffix='.zip', auto_remove=True)
@@ -1284,7 +1296,7 @@ def main() -> None:
             failed_test_desc_strs.append(result.test_descriptor.descriptor_str)
         if result.num_errors_copying_artifacts > 0:
             logging.info("Test had errors copying artifacts to build host: %s",
-                         result.test_descriptors)
+                         result.test_descriptor)
         num_tests_by_language[test_language] += 1
 
     if had_errors_copying_artifacts and global_exit_code == 0:

diff --git a/build-support/thirdparty_archives.yml b/build-support/thirdparty_archives.yml
@@ -1,97 +1,92 @@
-sha_for_local_checkout: 44ba3719652858e1f7816df87b25101e09527c88
+sha_for_local_checkout: 2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd
 archives:
-  - os_type: almalinux8
-    architecture: x86_64
-    compiler_type: clang11
-    sha: 44ba3719652858e1f7816df87b25101e09527c88
-    tag: v20210916160342-44ba371965-almalinux8-x86_64-clang11
   - os_type: almalinux8
     architecture: x86_64
     compiler_type: clang12
-    sha: 44ba3719652858e1f7816df87b25101e09527c88
-    tag: v20210916160348-44ba371965-almalinux8-x86_64-clang12
+    sha: 2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd
+    tag: v20211027161918-2d282c38cf-almalinux8-x86_64-clang12
   - os_type: almalinux8
     architecture: x86_64
     compiler_type: gcc8
-    sha: 44ba3719652858e1f7816df87b25101e09527c88
-    tag: v20210916160322-44ba371965-almalinux8-x86_64-gcc8
+    sha: 2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd
+    tag: v20211019082115-2d282c38cf-almalinux8-x86_64-gcc8
   - os_type: almalinux8
     architecture: x86_64
     compiler_type: gcc9
-    sha: 44ba3719652858e1f7816df87b25101e09527c88
-    tag: v20210916160339-44ba371965-almalinux8-x86_64-gcc9
+    sha: 2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd
+    tag: v20211019082136-2d282c38cf-almalinux8-x86_64-gcc9
   - os_type: centos7
     architecture: x86_64
     compiler_type: clang11
-    sha: 44ba3719652858e1f7816df87b25101e09527c88
-    tag: v20210916160331-44ba371965-centos7-x86_64-clang11
+    sha: 2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd
+    tag: v20211019082159-2d282c38cf-centos7-x86_64-clang11
   - os_type: centos7
     architecture: x86_64
     compiler_type: clang12
-    sha: 44ba3719652858e1f7816df87b25101e09527c88
-    tag: v20210916160400-44ba371965-centos7-x86_64-clang12
+    sha: 2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd
+    tag: v20211019082141-2d282c38cf-centos7-x86_64-clang12
   - os_type: centos7
     architecture: x86_64
     compiler_type: clang7
-    sha: 44ba3719652858e1f7816df87b25101e09527c88
-    tag: v20210916160346-44ba371965-centos7-x86_64-clang7
+    sha: 2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd
+    tag: v20211027161952-2d282c38cf-centos7-x86_64-clang7
   - os_type: centos7
     architecture: x86_64
     compiler_type: gcc5
-    sha: 44ba3719652858e1f7816df87b25101e09527c88
-    tag: v20210916160412-44ba371965-centos7-x86_64-linuxbrew-gcc5
+    sha: 2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd
+    tag: v20211027161950-2d282c38cf-centos7-x86_64-linuxbrew-gcc5
   - os_type: centos7
     architecture: x86_64
     compiler_type: gcc8
-    sha: 44ba3719652858e1f7816df87b25101e09527c88
-    tag: v20210916160355-44ba371965-centos7-x86_64-gcc8
+    sha: 2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd
+    tag: v20211019082135-2d282c38cf-centos7-x86_64-gcc8
   - os_type: centos7
     architecture: x86_64
     compiler_type: gcc9
-    sha: 44ba3719652858e1f7816df87b25101e09527c88
-    tag: v20210916160541-44ba371965-centos7-x86_64-gcc9
+    sha: 2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd
+    tag: v20211019082130-2d282c38cf-centos7-x86_64-gcc9
   - os_type: centos8
     architecture: x86_64
     compiler_type: gcc8
-    sha: 44ba3719652858e1f7816df87b25101e09527c88
-    tag: v20210916160428-44ba371965-centos8-x86_64-gcc8
+    sha: 2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd
+    tag: v20211027161938-2d282c38cf-centos8-x86_64-gcc8
   - os_type: centos8
     architecture: x86_64
     compiler_type: gcc9
-    sha: 44ba3719652858e1f7816df87b25101e09527c88
-    tag: v20210916160344-44ba371965-centos8-x86_64-gcc9
+    sha: 2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd
+    tag: v20211027161916-2d282c38cf-centos8-x86_64-gcc9
   - os_type: macos
     architecture: x86_64
     compiler_type: clang
-    sha: 44ba3719652858e1f7816df87b25101e09527c88
-    tag: v20210916160324-44ba371965-macos-x86_64
+    sha: 2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd
+    tag: v20211027161854-2d282c38cf-macos-x86_64
   - os_type: ubuntu18.04
     architecture: x86_64
     compiler_type: clang10
-    sha: 44ba3719652858e1f7816df87b25101e09527c88
-    tag: v20210916160408-44ba371965-ubuntu1804-x86_64-clang10
+    sha: 2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd
+    tag: v20211027162018-2d282c38cf-ubuntu1804-x86_64-clang10
   - os_type: ubuntu18.04
     architecture: x86_64
     compiler_type: clang11
-    sha: 44ba3719652858e1f7816df87b25101e09527c88
-    tag: v20210916160408-44ba371965-ubuntu1804-x86_64-clang11
+    sha: 2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd
+    tag: v20211027162001-2d282c38cf-ubuntu1804-x86_64-clang11
   - os_type: ubuntu18.04
     architecture: x86_64
     compiler_type: gcc7
-    sha: 44ba3719652858e1f7816df87b25101e09527c88
-    tag: v20210916160407-44ba371965-ubuntu1804-x86_64-gcc7
+    sha: 2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd
+    tag: v20211027162012-2d282c38cf-ubuntu1804-x86_64-gcc7
   - os_type: ubuntu18.04
     architecture: x86_64
     compiler_type: gcc8
-    sha: 44ba3719652858e1f7816df87b25101e09527c88
-    tag: v20210916160407-44ba371965-ubuntu1804-x86_64-gcc8
+    sha: 2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd
+    tag: v20211027162015-2d282c38cf-ubuntu1804-x86_64-gcc8
   - os_type: ubuntu20.04
     architecture: x86_64
     compiler_type: clang11
-    sha: 44ba3719652858e1f7816df87b25101e09527c88
-    tag: v20210916160341-44ba371965-ubuntu2004-x86_64-clang11
+    sha: 2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd
+    tag: v20211019082206-2d282c38cf-ubuntu2004-x86_64-clang11
   - os_type: ubuntu20.04
     architecture: x86_64
     compiler_type: gcc9
-    sha: 44ba3719652858e1f7816df87b25101e09527c88
-    tag: v20210916160342-44ba371965-ubuntu2004-x86_64-gcc9
+    sha: 2d282c38cfcbc10af7bdc1c86bf1e8af88f36efd
+    tag: v20211027162001-2d282c38cf-ubuntu2004-x86_64-gcc9
diff --git a/build-support/ubsan-suppressions.txt b/build-support/ubsan-suppressions.txt
@@ -1,14 +1,17 @@
 # https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html#runtime-suppressions
 
-# See ubsan_checks.inc from the UBSAN source for the list of possible UBSAN checks.
-# The source code is at https://github.com/llvm-mirror/compiler-rt
+# See ubsan_checks.inc from the UBSAN source for the list of possible UBSAN checks, e.g.:
+# https://github.com/yugabyte/llvm-project/blob/12.0.1-yb/compiler-rt/lib/ubsan/ubsan_checks.inc
 
 # Here is a copy-and-paste for convenience:
 # https://gist.githubusercontent.com/mbautin/fb08c1864783cfa410c16caf7c1b4833/raw
 
 # It looks like in the suppression files we should use the last argument out of the three arguments
 # to the UBSAN_CHECK macro. Here is a list for convenience.
 #
+# This was produced with the following command, and output slightly edited.
+# curl -qL https://bit.ly/3uE0zDH | grep -Eo '"[^"]+")' | sed 's/[")]//g; s/^/# /g;' | sort | uniq
+#
 # alignment
 # bool
 # bounds
@@ -17,10 +20,17 @@
 # float-cast-overflow
 # float-divide-by-zero
 # function
+# implicit-integer-sign-change
+# implicit-signed-integer-truncation
+# implicit-unsigned-integer-truncation
 # integer-divide-by-zero
 # invalid-builtin-use
+# invalid-objc-cast
 # nonnull-attribute
 # null
+# nullability-arg
+# nullability-assign
+# nullability-return
 # object-size
 # pointer-overflow
 # return
@@ -59,6 +69,7 @@ enum:Aws::Utils::Outcome*operator=
 
 # crcutil works with unaligned data
 alignment:crcutil::Crc32cSSE4::Crc32c
+alignment:crc32c_sse4.cc
 
 # snappy works with unaligned data
 alignment:snappy::
@@ -105,3 +116,13 @@ float-cast-overflow:yb::bfql::SetNumericResult
 # UBSAN issues in ICU 67.
 nonnull-attribute:libicuuc.so
 function:libicuuc.so
+
+# https://github.com/yugabyte/yugabyte-db/issues/10046
+# There is this line in gflags.cc:
+#
+# for (; line_end; flagfile_contents = line_end + 1) {
+#
+# The `flagfile_contents = line_end + 1` part has undefined behavior if line_end is nullptr, but
+# it does not matter because the loop condition (`line_end`) will cause the loop to exit in that
+# case.
+pointer-overflow:CommandLineFlagParser::ProcessOptionsFromStringLocked
diff --git a/ent/src/yb/integration-tests/cdc_service-int-test.cc b/ent/src/yb/integration-tests/cdc_service-int-test.cc
@@ -860,10 +860,12 @@ TEST_P(CDCServiceTestMultipleServersOneTablet, TestUpdateLagMetrics) {
   GetTablet(&tablet_id);
 
   // Get the leader and a follower for the tablet.
-  tserver::MiniTabletServer* leader_mini_tserver;
-  tserver::MiniTabletServer* follower_mini_tserver;
+  tserver::MiniTabletServer* leader_mini_tserver = nullptr;
+  tserver::MiniTabletServer* follower_mini_tserver = nullptr;
 
   ASSERT_OK(WaitFor([&]() -> Result<bool> {
+    leader_mini_tserver = nullptr;
+    follower_mini_tserver = nullptr;
     for (int i = 0; i < cluster_->num_tablet_servers(); i++) {
       std::shared_ptr<tablet::TabletPeer> tablet_peer;
       Status s = cluster_->mini_tablet_server(i)->server()->tablet_manager()->

diff --git a/jenkins_jobs.yml b/jenkins_jobs.yml
@@ -1,8 +1,8 @@
 # This file determines the jobs that we run in this branch on Jenkins.
 # Default architecture is x86_64
 jobs:
-  - os: centos7
-    compiler: clang7
+  - os: alma8
+    compiler: clang12
     build_type: asan
   - os: centos7
     compiler: clang7

diff --git a/python/yb/thirdparty_tool.py b/python/yb/thirdparty_tool.py
@@ -165,22 +165,24 @@ def __init__(self, github_release: GitRelease) -> None:
             branch_name = branch_name.rstrip('-')
         self.branch_name = branch_name
 
-    def validate_url(self) -> None:
+    def validate_url(self) -> bool:
         asset_urls = [asset.browser_download_url for asset in self.github_release.get_assets()]
 
         if len(asset_urls) != 2:
-            raise ValueError(
+            logging.warning(
                 "Expected to find exactly two asset URLs for a release "
                 "(one for the .tar.gz, the other for the checksum), "
                 f"but found {len(asset_urls)}: {asset_urls}")
+            return False
 
         non_checksum_urls = [url for url in asset_urls if not url.endswith('.sha256')]
         assert(len(non_checksum_urls) == 1)
         self.url = non_checksum_urls[0]
         if not self.url.startswith(DOWNLOAD_URL_PREFIX):
-            raise ValueError(
+            logging.warning(
                 f"Expected archive download URL to start with {DOWNLOAD_URL_PREFIX}, found "
                 f"{self.url}")
+            return False
 
         url_suffix = self.url[len(DOWNLOAD_URL_PREFIX):]
         url_suffix_components = url_suffix.split('/')
@@ -189,9 +191,12 @@ def validate_url(self) -> None:
         archive_basename = url_suffix_components[1]
         expected_basename = get_archive_name_from_tag(self.tag)
         if archive_basename != expected_basename:
-            raise ValueError(
+            logging.warning(
                 f"Expected archive name based on tag: {expected_basename}, "
                 f"actual name: {archive_basename}, url: {self.url}")
+            return False
+
+        return True
 
     def as_dict(self) -> Dict[str, str]:
         return {k: getattr(self, k) for k in self.KEY_FIELDS_WITH_TAG}
@@ -389,33 +394,42 @@ def update_archive_metadata_file(self) -> None:
         releases_by_key_without_tag: DefaultDict[Tuple[str, ...], List[YBDependenciesRelease]] = \
             defaultdict(list)
 
+        num_valid_releases = 0
+        num_invalid_releases = 0
         for yb_thirdparty_release in releases_for_one_commit:
-            yb_thirdparty_release.validate_url()
-            releases_by_key_without_tag[
-                yb_thirdparty_release.get_sort_key(include_tag=False)
-            ].append(yb_thirdparty_release)
+            if yb_thirdparty_release.validate_url():
+                num_valid_releases += 1
+                releases_by_key_without_tag[
+                    yb_thirdparty_release.get_sort_key(include_tag=False)
+                ].append(yb_thirdparty_release)
+            else:
+                num_invalid_releases += 1
+        logging.info(
+            f"Valid releases found: {num_valid_releases}, invalid releases: {num_invalid_releases}")
 
-        keys_with_duplicate_releases: List[Tuple[str, ...]] = []
+        filtered_releases_for_one_commit = []
         for key_without_tag, releases_for_key in releases_by_key_without_tag.items():
             if len(releases_for_key) > 1:
+                picked_release = max(releases_for_key, key=lambda r: r.tag)
                 logging.info(
                     "Multiple releases found for the same key (excluding the tag). "
-                    "Key: %s, releases: %s" % (
+                    "Using the latest one: %s\n"
+                    "Key: %s.\nReleases:\n  %s" % (
+                        picked_release,
                         key_without_tag,
-                        releases_for_key))
-                keys_with_duplicate_releases.append(key_without_tag)
-        if keys_with_duplicate_releases:
-            raise ValueError(
-                "Multiple releases found for these keys: %s" % keys_with_duplicate_releases)
+                        '\n  '.join([str(r) for r in releases_for_key])))
+                filtered_releases_for_one_commit.append(picked_release)
+            else:
+                filtered_releases_for_one_commit.append(releases_for_key[0])
 
-        releases_for_one_commit.sort(key=YBDependenciesRelease.get_sort_key)
+        filtered_releases_for_one_commit.sort(key=YBDependenciesRelease.get_sort_key)
 
-        for yb_thirdparty_release in releases_for_one_commit:
+        for yb_thirdparty_release in filtered_releases_for_one_commit:
             new_metadata['archives'].append(yb_thirdparty_release.as_dict())
 
         write_yaml_file(new_metadata, archive_metadata_path)
         logging.info(
-            f"Wrote information for {len(releases_for_one_commit)} pre-built "
+            f"Wrote information for {len(filtered_releases_for_one_commit)} pre-built "
             f"yugabyte-db-thirdparty archives to {archive_metadata_path}.")
 
 

diff --git a/requirements.txt b/requirements.txt
@@ -15,4 +15,5 @@ semantic-version
 six
 sys_detection
 wheel
-yugabyte_pycommon
+yugabyte_pycommon
+compiler-identification
-Original file line number
+Diff line change
@@ Expand Up / @@ -183,6 +183,8 @@ readonly -a VALID_COMPILER_TYPES=( @@
       gcc8
       gcc9
       gcc10
+      gcc11
+      gcc12
       clang
       clang7
       clang8
@@ Expand Down @@