[SR-100] Red-hat - Support - lldb build

[swift-ci]

Previous ID	SR-100
Radar	None
Original Reporter	thawkins (JIRA User)
Type	New Feature

Attachment: Download

Additional Detail from JIRA

Votes	2
Component/s	Compiler, LLDB for Swift
Labels	New Feature
Assignee	None
Priority	Medium

md5: 8e846ffa24d7a9a16bf9b2eda3207878

Issue Description:

1'm positing a few bugs for redhat support issues, whilst it is not an official supported distro, there are only a couple of small blockers for easy integration

when building swift-lldb on redhat os's, the Python support code is placed in ./lib64 and not /lib

all other libs are placed in /lib

This appears to be an upstream lldb issue.

The only place which impacts swift is the building of the install package, which expects everything to be in ./lib, the symlinks inside /lib64/Python2.7 even reffer to the ./lib directory

There are several possible fixes for this

1. Get upstream to fix the lldb bug
2. Patch the lldb build script to copy or move the Python2.7 directory from ./lib64 to ./lib post build.
3. Patch the package install file builder to look in both locations ie ./lib64 after ./lib

[trfiala]

Can you post more info related to this in the upstream llvm.org bugzilla tracker? I've done some work in the past to get llvm.org LLDB working on RHEL 7 and had solved some of the lib64/lib issues for Python on x86_64. It would be good to look at what is different with the configuration where you're seeing the issue.

Thanks!

[swift-ci]

Comment by Timothy S Hawkins (JIRA)

Noted, i will try and setup a vanila build of lldb on f23 and check if it fails in the same way, and document.

Do you know if the llvm/lldb project has a dockerfile for ubuntu i can hack to create a clean test setup for f23.

[swift-ci]

Comment by Timothy S Hawkins (JIRA)

I have built a script to automate the f23 build, see below.

https://github.com/thawkins/fedora-swift

[swift-ci]

Comment by Timothy S Hawkins (JIRA)

This bug has already been reported on lldb upstream (including somebody else hitting it on the swift build)

https://llvm.org/bugs/show_bug.cgi?id=18957

[swift-ci]

Comment by Jeremy Fergason (JIRA)

thawkins (JIRA User) were you able to determine if the base lldb built successfully on fc23? I've put together a specfile and just currently patch lldb. There's a bugzilla ticket for inclusion of the swift RPM's in Fedora:

https://bugzilla.redhat.com/show_bug.cgi?id=1295115

[alblue]

The ability to specify a suffix for lldb was added here:

apple/swift-lldb@35bb23f

Specifically, it added an LLVM_LIBDIR_SUFFIX flag that could be used to have /lib/ or /lib64/ as appropriate.

• set(LLVM_LIBRARY_OUTPUT_INTDIR ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/lib)
  + set(LLVM_LIBRARY_OUTPUT_INTDIR ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/lib${LLVM_LIBDIR_SUFFIX})

If you set LLVM_LIBDIR_SUFFIX to empty, it should be installed into /lib/ as expected. Does this not work for you?

[swift-ci]

Comment by Timothy S Hawkins (JIRA)

@alex, this problem is still happening

Im not sure how to apply your fix,

Im setting up a new project on github to build swift on fedora25, it will support using a docker container.

But this is the only test that fails, and with a few workarounds by running the build-script with tests disabled, and then hacking the build result a bit, then running it again with tests enabled I get a fully functional version of 3.1, works inside my CLion IDE including interactive debugging, so i assume everything is ok. It just fails tests because of this issue this issue.

I will have the scripts and instructions up in the next 24 hours, the config should run on ubuntu or macosx, under docker.

https://github.com/FedoraSwift/fedora-swift2

[swift-ci]

Comment by Timothy S Hawkins (JIRA)

{{this is the one test that fails

******************* TEST 'Swift(linux-x86_64) :: Runtime/linux-fatal-backtrace.swift' FAILED ********************
Script:
--
rm -rf /root/swift-source/build/buildbot_linux_fc25/swift-linux-x86_64/test-linux-x86_64/Runtime/Output/linux-fatal-backtrace.swift.tmp
mkdir -p /root/swift-source/build/buildbot_linux_fc25/swift-linux-x86_64/test-linux-x86_64/Runtime/Output/linux-fatal-backtrace.swift.tmp !Screenshot from 2017-02-04 22-12-33.png|thumbnail! 
/root/swift-source/build/buildbot_linux_fc25/swift-linux-x86_64/bin/swiftc -target x86_64-unknown-linux-gnu  -module-cache-path '/tmp/swift-testsuite-clang-module-cacheXzZsiC' -swift-version 3  /root/swift-source/swift/test/Runtime/linux-fatal-backtrace.swift -o /root/swift-source/build/buildbot_linux_fc25/swift-linux-x86_64/test-linux-x86_64/Runtime/Output/linux-fatal-backtrace.swift.tmp/a.out
not --crash /root/swift-source/build/buildbot_linux_fc25/swift-linux-x86_64/test-linux-x86_64/Runtime/Output/linux-fatal-backtrace.swift.tmp/a.out 2>&1 | PYTHONPATH=/root/swift-source/build/buildbot_linux_fc25/lldb-linux-x86_64/lib/python2.7/site-packages /root/swift-source/swift/utils/symbolicate-linux-fatal /root/swift-source/build/buildbot_linux_fc25/swift-linux-x86_64/test-linux-x86_64/Runtime/Output/linux-fatal-backtrace.swift.tmp/a.out - | /root/swift-source/swift/utils/backtrace-check -u
--
Exit Code: 1

Command Output (stderr):
--
Traceback (most recent call last):
File "/root/swift-source/swift/utils/symbolicate-linux-fatal", line 30, in <module>
 import lldb
File "/root/swift-source/build/buildbot_linux_fc25/lldb-linux-x86_64/lib/python2.7/site-packages/lldb/__init__.py", line 43, in <module>
 _lldb = swig_import_helper()
File "/root/swift-source/build/buildbot_linux_fc25/lldb-linux-x86_64/lib/python2.7/site-packages/lldb/__init__.py", line 42, in swig_import_helper
 return importlib.import_module('_lldb')
File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
 __import__(name)
ImportError: No module named _lldb
Traceback (most recent call last):
File "/root/swift-source/swift/utils/backtrace-check", line 81, in <module>
 main()
File "/root/swift-source/swift/utils/backtrace-check", line 78, in main
 assert(found_stack_trace_entry)
AssertionError

--

********************
}}

[alblue]

Will investigate further regarding this test. If you have an easy way of reproducing it with a docker container, that would be very helpful.

Fortunately it's testing a standalone tool that helps symbolicate crash references, rather than being part of the core Swift runtime, so if this is the only test that fails then you should be good to go.

[swift-ci]

Comment by Gregor Milos (Grzegorz Miłoś) (JIRA)

I tried to reproduce the Fedora 24 compilation & symbolicate-linux-fatal failure using https://github.com/FedoraSwift/fedora-swift and fedora:24 docker image. I hit number of missing packages not provisioned by swiftbuild.sh setup. So far I added the following:

dnf install autoconf sudo git automake libtool make
# The below are for Fedora 22, not sure if they are suitable for 24?)
https://copr-be.cloud.fedoraproject.org/results/lebauce/Darling/fedora-22-x86_64/libkqueue-2.0.1-1/libkqueue-devel-2.0.1-1.x86_64.rpm
https://copr-be.cloud.fedoraproject.org/results/lebauce/Darling/fedora-22-x86_64/libkqueue-2.0.1-1/libkqueue-2.0.1-1.x86_64.rpm

That's still not enough, current failure is:

[89/320] CompileC: closure/data.c
FAILED: ../build/buildbot_linux/foundation-linux-x86_64/Foundation/closure/data.c.o
mkdir -p `dirname ../build/buildbot_linux/foundation-linux-x86_64/Foundation/closure/data.c.o`; /root/tmp/swiftbuild/build/buildbot_linux/llvm-linux-x86_64/bin/clang -fcolor-diagnostics -fdollars-in-identifiers -fblocks -fobjc-runtime=macosx-10.11 -fintegrated-as -fPIC --target=x86_64-linux-gnu -O2   -Ibootstrap/common/usr/include -Ibootstrap/common/usr/local/include   -Ibootstrap/x86_64-linux-gnu/usr/include -Ibootstrap/x86_64-linux-gnu/usr/local/include  -I../build/buildbot_linux/foundation-linux-x86_64/Foundation -I../build/buildbot_linux/foundation-linux-x86_64 -I../build/buildbot_linux/foundation-linux-x86_64/Foundation/usr//lib/swift -I../build/buildbot_linux/foundation-linux-x86_64/Foundation/usr//lib/swift/CoreFoundation -I../build/buildbot_linux/foundation-linux-x86_64/Foundation/usr//lib/swift/CoreFoundation -I../build/buildbot_linux/foundation-linux-x86_64/Foundation/usr//lib/swift/CoreFoundation -DDEPLOYMENT_TARGET_LINUX -D_GNU_SOURCE -DCF_CHARACTERSET_DATA_DIR="CoreFoundation/CharacterSets"-DU_SHOW_DRAFT_API -DCF_BUILDING_CF -DDEPLOYMENT_RUNTIME_SWIFT -fconstant-cfstrings -fexceptions -Wno-shorten-64-to-32 -Wno-deprecated-declarations -Wno-unreachable-code -Wno-conditional-uninitialized -Wno-unused-variable -Wno-int-conversion -Wno-unused-function -I/usr/include/libxml2 -I/usr/include/curl -I./ -DDEPLOYMENT_ENABLE_LIBDISPATCH -I/root/tmp/swiftbuild/swift-corelibs-libdispatch -I/root/tmp/swiftbuild/build/buildbot_linux/libdispatch-linux-x86_64/tests -include CoreFoundation/Base.subproj/CoreFoundation_Prefix.h  -c closure/data.c -o ../build/buildbot_linux/foundation-linux-x86_64/Foundation/closure/data.c.o
In file included from <built-in>:1:
In file included from ./CoreFoundation/Base.subproj/CoreFoundation_Prefix.h:30:
../build/buildbot_linux/foundation-linux-x86_64/Foundation/usr//lib/swift/CoreFoundation/CFBase.h:72:10: fatal error: 'Block.h' file not found
#include <Block.h>
         ^~~~~~~~~
1 error generated.

I'll try https://github.com/FedoraSwift/fedora-swift2 with f24, or https://github.com/FedoraSwift/fedora-swift with centos7.2. I don't fully understand the reasons & differences between the two repos.

[swift-ci]

Comment by Gregor Milos (Grzegorz Miłoś) (JIRA)

Tried building fedora-swift2 with the following buildswift config:

TAG=""
BRANCH="master"
FEDORA_VERSION=25

However, this fails rather quickly with:

+ swiftutils/utils/update-checkout --scheme master --clone --tag --reset-to-remote
usage: update-checkout [-h] [--clone] [--clone-with-ssh] [--skip-history]
                       [--skip-repository DIRECTORY] [--scheme BRANCH-SCHEME]
                       [--reset-to-remote] [--clean] [--config CONFIG]
                       [--github-comment GITHUB-COMMENT] [--dump-hashes]
                       [--dump-hashes-config BRANCH-SCHEME-NAME]
                       [--tag TAG-NAME] [-j N_PROCESSES]
update-checkout: error: argument --tag: expected one argument

Seems like the following comment is invalid:
https://github.com/FedoraSwift/fedora-swift2/blame/1154b5e67fedc5db67cffb4d6ee54213c0680277/README.md#L76

I'll do a bit more work to progress, but would appreciate better build script to repro the problem.

[swift-ci]

Comment by Timothy S Hawkins (JIRA)

hmmm, i though i fixed this, i will test timorrow, i just build a cooy on fedora 25 of swift-3.1-branch which worked fine.

[swift-ci]

Comment by Timothy S Hawkins (JIRA)

Note, even getting past this problem, there are a bunch of issues in the module-import side of the compiler, where it makes a bunch of assumptiins about versiins and layout of gcc that are only true on ubuntu, fedora has gcc6 ubuntu is still running gcc5 . But at least we can get this one sorted.

[swift-ci]

Comment by Gregor Milos (Grzegorz Miłoś) (JIRA)

I tried fedora-swift2 with the following bulidswift:

TAG="swift-3.0.2-RELEASE"
BRANCH="master"
FEDORA_VERSION=25

That progressed much further, but still failed. This time with:

[4/548] Building CXX object lib/Basic/CMakeFiles/swiftBasic.dir/SourceLoc.cpp.o
FAILED: lib/Basic/CMakeFiles/swiftBasic.dir/SourceLoc.cpp.o
/usr/bin/clang++   -DGTEST_HAS_RTTI=0 -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Ilib/Basic -I/root/swift-source/swift/lib/Basic -I/root/swift-source/swift/include -Iinclude -I/root/swift-source/llvm/include -I/root/swift-source/build/buildbot_linux_swift-3.0.2-RELEASE_fc25/llvm-linux-x86_64/include -I/root/swift-source/build/buildbot_linux_swift-3.0.2-RELEASE_fc25/llvm-linux-x86_64/tools/clang/include -I/root/swift-source/llvm/tools/clang/include -I/root/swift-source/cmark/src -I/root/swift-source/build/buildbot_linux_swift-3.0.2-RELEASE_fc25/cmark-linux-x86_64/src -fno-stack-protector -fPIC -fvisibility-inlines-hidden -Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Werror=date-time -std=c++11 -fcolor-diagnostics -ffunction-sections -fdata-sections -Wdocumentation -Wimplicit-fallthrough -Wunreachable-code -Woverloaded-virtual -O2    -UNDEBUG  -fno-exceptions -fno-rtti -I/usr/include -target x86_64-unknown-linux-gnu -O2 -momit-leaf-frame-pointer -g0 -UNDEBUG -MD -MT lib/Basic/CMakeFiles/swiftBasic.dir/SourceLoc.cpp.o -MF lib/Basic/CMakeFiles/swiftBasic.dir/SourceLoc.cpp.o.d -o lib/Basic/CMakeFiles/swiftBasic.dir/SourceLoc.cpp.o -c /root/swift-source/swift/lib/Basic/SourceLoc.cpp
/root/swift-source/swift/lib/Basic/SourceLoc.cpp:85:15: error: use of overloaded operator '=' is ambiguous (with operand types 'std::pair<const char *, const VirtualFile *>' and 'void')
  CachedVFile = {};
  ~~~~~~~~~~~ ^ ~~
/usr/bin/../lib/gcc/x86_64-redhat-linux/6.3.1/../../../../include/c++/6.3.1/bits/stl_pair.h:376:7: note: candidate function
      operator=(typename conditional<
      ^
/usr/bin/../lib/gcc/x86_64-redhat-linux/6.3.1/../../../../include/c++/6.3.1/bits/stl_pair.h:359:7: note: candidate function
      operator=(typename conditional<
      ^
/usr/bin/../lib/gcc/x86_64-redhat-linux/6.3.1/../../../../include/c++/6.3.1/bits/stl_pair.h:370:7: note: candidate function has been explicitly deleted
      operator=(typename conditional<
      ^
/root/swift-source/swift/lib/Basic/SourceLoc.cpp:102:15: error: use of overloaded operator '=' is ambiguous (with operand types 'std::pair<const char *, const VirtualFile *>' and 'void')
  CachedVFile = {};
  ~~~~~~~~~~~ ^ ~~
/usr/bin/../lib/gcc/x86_64-redhat-linux/6.3.1/../../../../include/c++/6.3.1/bits/stl_pair.h:376:7: note: candidate function
      operator=(typename conditional<
      ^
/usr/bin/../lib/gcc/x86_64-redhat-linux/6.3.1/../../../../include/c++/6.3.1/bits/stl_pair.h:359:7: note: candidate function
      operator=(typename conditional<
      ^
/usr/bin/../lib/gcc/x86_64-redhat-linux/6.3.1/../../../../include/c++/6.3.1/bits/stl_pair.h:370:7: note: candidate function has been explicitly deleted
      operator=(typename conditional<
      ^
2 errors generated.

Like before, thawkins (JIRA User), if you can provide me with a reliable repro of the symbolicate-linux-fatal, I'll try fixing that issue.

[swift-ci]

Comment by Timothy S Hawkins (JIRA)

use:

on Fc25

BRANCH="swift-3.1-branch"
TAG=""
That runs to completion. Refresh your scripts, as i just found an issue with detecting empty tags.

Im not sure what happened with

BRANCH="master"
TAG="swift-3.0.2-RELEASE"

It was working a few days ago,

I will investigate further