SEGFAULT with native images fully static performing DNS lookup

[pmlopes]

Hi,

I was trying to build a image that already works when using shared objects with the extra flag: --static.

The application crashes as soon as a DNS resolution is requested:

[ [ SubstrateSegfaultHandler caught signal 11 ] ]

General Purpose Register Set Values:

  RAX 0000000003d4e000
  RBX 000000000148ddd0
  RCX 0000000000ddcc6c
  RDX 0000000000020770
  RBP 0000000001c1b0e0
  RSI 0000000000000080
  RDI 0000000001c1b0e0
  RSP 00007f0fb35fd350
  R8  0000000000000025
  R9  0000000000000001
  R10 0000000000e35c50
  R11 0000000000000202
  R12 0000000000a43e90
  R13 0000000001ed1c60
  R14 00000000c0100702
  R15 00007f0fa00008c0
  EFL 0000000000010287
  RIP 0000000000e0aa7a


JavaFrameAnchor dump:

  Anchor 00007f0fb35fd528 LastJavaSP 00007f0fb35fd450

DeoptStubPointer address: 0000000000000000

TopFrame info:

  Lookup TotalFrameSize in CodeInfoTable:
  SourceTotalFrameSize -1
  Does not look like a Java Frame. Use JavaFrameAnchors to find LastJavaSP:
  Found matching Anchor:00007f0fb35fd528
  LastJavaSP 00007f0fb35fd450
  LastJavaIP 0000000000a8d585

VMThreads info:

  VMThread 00007f0f980008c0  STATUS_IN_NATIVE
  VMThread 00007f0fa40008c0  STATUS_IN_NATIVE
  VMThread 00007f0fa00008c0  STATUS_IN_JAVA (safepoints disabled)
  VMThread 00007f0fa80008c0  STATUS_IN_NATIVE
  VMThread 00007f0fac0008c0  STATUS_IN_NATIVE
  VMThread 00007f0fb40008c0  STATUS_IN_NATIVE
  VMThread 0000000003b83880  STATUS_IN_NATIVE

VM Thread State for current thread 00007f0fa00008c0:

  0 (8 bytes): com.oracle.svm.core.genscavenge.PinnedAllocatorImpl.openPinnedAllocator = null
  8 (8 bytes): com.oracle.svm.core.heap.NoAllocationVerifier.openVerifiers = null
  16 (8 bytes): com.oracle.svm.core.jdk.IdentityHashCodeSupport.hashCodeGeneratorTL = java.util.SplittableRandom  00007f0fb2a02e20
  24 (8 bytes): com.oracle.svm.core.snippets.SnippetRuntime.currentException = null
  32 (8 bytes): com.oracle.svm.core.thread.JavaThreads.currentThread = io.vertx.core.impl.VertxThread  00007f0fb8ab1b68
  40 (8 bytes): com.oracle.svm.core.thread.ThreadingSupportImpl.activeTimer = null
  48 (32 bytes): com.oracle.svm.core.genscavenge.ThreadLocalAllocation.pinnedTLAB =
    00007f0fa00008f0: 0000000000000000 0000000000000000
    00007f0fa0000900: 0000000000000000 0000000000000000


  80 (32 bytes): com.oracle.svm.core.genscavenge.ThreadLocalAllocation.regularTLAB =
    00007f0fa0000910: 00007f0fb2a00000 00007f0fb2b00000
    00007f0fa0000920: 00007f0fb2a06eb8 0000000000000000


  112 (8 bytes): com.oracle.svm.core.genscavenge.ThreadLocalAllocation.freeList = 0  0000000000000000
  120 (8 bytes): com.oracle.svm.core.posix.thread.PosixVMThreads.IsolateTL = 94598687557484880  0150150150150150
  128 (8 bytes): com.oracle.svm.core.stack.JavaFrameAnchors.lastAnchor = 139705410639144  00007f0fb35fd528
  136 (8 bytes): com.oracle.svm.core.thread.VMThreads.nextTL = 139705219811520  00007f0fa80008c0
  144 (4 bytes): com.oracle.svm.core.thread.Safepoint.safepointRequested = -11746  ffffd21e
  148 (4 bytes): com.oracle.svm.core.thread.Safepoint.safepointRequestedValueBeforeSafepoint = 0  00000000
  152 (4 bytes): com.oracle.svm.core.thread.VMOperationControl.isLockOwner = 0  00000000
  156 (4 bytes): com.oracle.svm.core.thread.VMThreads$StatusSupport.safepointsDisabledTL = 1  00000001
  160 (4 bytes): com.oracle.svm.core.thread.VMThreads$StatusSupport.statusTL = 1  00000001

VMOperation dump:

  No VMOperation in progress


RuntimeCodeCache dump:

  == [Recent RuntimeCodeCache operations: ]

  == [RuntimeCodeCache: 0 methods]

Deoptimizer dump:

  == [Recent Deoptimizer Events:
  ]

Dump Counters:


Raw Stacktrace:

  00007f0fb35fd350: 00000000b2a06c98 0000000000642b2f
  00007f0fb35fd360: 0000002501480790 000000000061f1a1
  00007f0fb35fd370: 000000000021af32 0000000100000001
  00007f0fb35fd380: 00007f0fb2a06d58 000000000062166e
  00007f0fb35fd390: 00007f0fb2a04228 00007f0fb2a06d80
  00007f0fb35fd3a0: 00007f0fb2a04240 00007f0fb2a04140
  00007f0fb35fd3b0: 00007f0fb2a041c8 00007f0fb2a06c20
  00007f0fb35fd3c0: 0000000001481d60 00000000c0100702
  00007f0fb35fd3d0: 00007f0fa00008c0 0000000000e0ef17
  00007f0fb35fd3e0: 000000000147fe78 0000000000000012
  00007f0fb35fd3f0: 00007f0fb2a041a0 0000000000000090
  00007f0fb35fd400: 0000000000000000 00007f0fb2a04140
  00007f0fb35fd410: 00007f0fb2a041c8 00007f0fb2a06c20
  00007f0fb35fd420: 0000000001481d60 0000000000e35c6d
  00007f0fb35fd430: 0000000000a8d21b 0000000000e0eed0
  00007f0fb35fd440: 0000000000000000 0000000000a8d585
  00007f0fb35fd450: 0000000000000000 0000000000000000
  00007f0fb35fd460: 0000000001430438 f560319600000000
  00007f0fb35fd470: 0000000000000000 00000000fffffffe
  00007f0fb35fd480: 0000000000000000 0000000000000000
  00007f0fb35fd490: 00000000015908c8 0000000000c2b92d
  00007f0fb35fd4a0: 0000000101c11cd0 0000000100000000
  00007f0fb35fd4b0: 00007f0fb2a041e8 0000000000c798c5
  00007f0fb35fd4c0: 0000000001573898 fffffffe00000000
  00007f0fb35fd4d0: fffffffe00000000 0000000000000000
  00007f0fb35fd4e0: 00007f0fb2a041e8 00007f0fb2a041b0
  00007f0fb35fd4f0: 00007f0fb2a04160 0000000000e35c50
  00007f0fb35fd500: 0000000000e0eed0 0000000001b311d8
  00007f0fb35fd510: 00007f0fb35fd540 00007f0fb35fd548
  00007f0fb35fd520: 0000000001430438 00007f0fb35fd450
  00007f0fb35fd530: 0000000000000000 00007f0fb2a041c8
  00007f0fb35fd540: 00007f0fb2a04140 0000000200000002
  00007f0fb35fd550: 0000000000000000 0000000000000000
  00007f0fb35fd560: 0000000000000000 0000000000000000
  00007f0fb35fd570: 0000000000000000 0000000000a91cff
  00007f0fb35fd580: 00007f0fb2a03d50 0000000000c798c5
  00007f0fb35fd590: 0000000001ba10c0 0000000000bf3988
  00007f0fb35fd5a0: 00000000015c09a0 0000000101ba1120
  00007f0fb35fd5b0: 00007f0fb2a03d50 0000000000c7bd43
  00007f0fb35fd5c0: 0000000101ba10c0 0000000001b31100
  00007f0fb35fd5d0: 00007f0fb2a03d38 0000000000617a25
  00007f0fb35fd5e0: 0000000000000000 0000000ab964baa0
  00007f0fb35fd5f0: 0000000001ba10c0 0000000001430438
  00007f0fb35fd600: 0000000001b311d8 0000000000f3ede8
  00007f0fb35fd610: 00007f0fb2a01ba8 0000000000702f3d
  00007f0fb35fd620: 00007f0fb2a04140 0000000000000000
  00007f0fb35fd630: 0000000000000000 0000000001430438
  00007f0fb35fd640: 0000000001430438 0000000000a94295
  00007f0fb35fd650: 00000020014bc268 0000007700000010
  00007f0fb35fd660: 0000000000eb79b0 0000000000a16414
  00007f0fb35fd670: 00000001b2a01ae8 0000000000000000
  00007f0fb35fd680: 0000000001430438 0000000000a92fab
  00007f0fb35fd690: 00007f0fb2a03a30 000000000071217a
  00007f0fb35fd6a0: 00007f0fb2a03b50 00007f0fb964baa0
  00007f0fb35fd6b0: 00007f0fb2a01ba8 00007f0fb2a03178
  00007f0fb35fd6c0: 00007f0fb2a03ac8 0000000000000077
  00007f0fb35fd6d0: 00007f0fb2a02fe8 0000000000702f3d
  00007f0fb35fd6e0: 0000000000000000 0000000001430438
  00007f0fb35fd6f0: 00007f0fb2a03a30 0000000000a945ab
  00007f0fb35fd700: 000000000000009c 0000000001486b60
  00007f0fb35fd710: 00000000015404a0 0000000000bedaaf
  00007f0fb35fd720: 0000000001430438 000000000085f6db
  00007f0fb35fd730: 00007f0fb8ab1b68 000000020089bc05
  00007f0fb35fd740: 0000000001b205e8 000000400000001a

Stacktrace Stage0:

  RSP 00007f0fb35fd450 RIP 0000000000a8d585 FrameSize 304
  RSP 00007f0fb35fd580 RIP 0000000000a91cff FrameSize 208
  RSP 00007f0fb35fd650 RIP 0000000000a94295 FrameSize 64
  RSP 00007f0fb35fd690 RIP 0000000000a92fab FrameSize 112
  RSP 00007f0fb35fd700 RIP 0000000000a945ab FrameSize 48
  RSP 00007f0fb35fd730 RIP 000000000085f6db FrameSize 64
  RSP 00007f0fb35fd770 RIP 000000000086050a FrameSize 64
  RSP 00007f0fb35fd7b0 RIP 000000000086046f FrameSize 48
  RSP 00007f0fb35fd7e0 RIP 000000000085fd59 FrameSize 64
  RSP 00007f0fb35fd820 RIP 000000000085ff34 FrameSize 48
  RSP 00007f0fb35fd850 RIP 000000000085e236 FrameSize 80
  RSP 00007f0fb35fd8a0 RIP 00000000006805ea FrameSize 112
  RSP 00007f0fb35fd910 RIP 00000000006800fe FrameSize 80
  RSP 00007f0fb35fd960 RIP 000000000067fac0 FrameSize 48
  RSP 00007f0fb35fd990 RIP 000000000099a402 FrameSize 96
  RSP 00007f0fb35fd9f0 RIP 000000000092a289 FrameSize 128
  RSP 00007f0fb35fda70 RIP 0000000000929c4a FrameSize 80
  RSP 00007f0fb35fdac0 RIP 000000000095d9e4 FrameSize 64
  RSP 00007f0fb35fdb00 RIP 000000000095be1f FrameSize 48
  RSP 00007f0fb35fdb30 RIP 000000000087f458 FrameSize 32
  RSP 00007f0fb35fdb50 RIP 000000000089558f FrameSize 112
  RSP 00007f0fb35fdbc0 RIP 000000000073e473 FrameSize 80
  RSP 00007f0fb35fdc10 RIP 000000000088f024 FrameSize 160
  RSP 00007f0fb35fdcb0 RIP 00000000008896cf FrameSize 32
  RSP 00007f0fb35fdcd0 RIP 0000000000a42f1f FrameSize 32
  RSP 00007f0fb35fdcf0 RIP 0000000000660c24 FrameSize 96
  RSP 00007f0fb35fdd50 RIP 000000000061cf24 FrameSize 144

Stacktrace Stage1:

  RSP 00007f0fb35fd450 RIP 0000000000a8d585  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fd580 RIP 0000000000a91cff  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fd650 RIP 0000000000a94295  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fd690 RIP 0000000000a92fab  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fd700 RIP 0000000000a945ab  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fd730 RIP 000000000085f6db  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fd770 RIP 000000000086050a  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fd7b0 RIP 000000000086046f  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fd7e0 RIP 000000000085fd59  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fd820 RIP 000000000085ff34  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fd850 RIP 000000000085e236  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fd8a0 RIP 00000000006805ea  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fd910 RIP 00000000006800fe  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fd960 RIP 000000000067fac0  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fd990 RIP 000000000099a402  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fd9f0 RIP 000000000092a289  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fda70 RIP 0000000000929c4a  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fdac0 RIP 000000000095d9e4  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fdb00 RIP 000000000095be1f  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fdb30 RIP 000000000087f458  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fdb50 RIP 000000000089558f  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fdbc0 RIP 000000000073e473  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fdc10 RIP 000000000088f024  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fdcb0 RIP 00000000008896cf  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fdcd0 RIP 0000000000a42f1f  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fdcf0 RIP 0000000000660c24  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code
  RSP 00007f0fb35fdd50 RIP 000000000061cf24  com.oracle.svm.core.code.ImageCodeInfo@0x1480a88 name = image code

Full Stacktrace:

  RSP 00007f0fb35fd450 RIP 0000000000a8d585  [image code] com.oracle.svm.core.posix.headers.Netdb.freeaddrinfo(Netdb.java)
  RSP 00007f0fb35fd450 RIP 0000000000a8d585  [image code] com.oracle.svm.core.posix.Target_java_net_Inet4AddressImpl.lookupAllHostAddr(PosixJavaNetSubstitutions.java:695)
  RSP 00007f0fb35fd580 RIP 0000000000a91cff  [image code] java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
  RSP 00007f0fb35fd580 RIP 0000000000a91cff  [image code] java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
  RSP 00007f0fb35fd650 RIP 0000000000a94295  [image code] java.net.InetAddress.getAllByName0(InetAddress.java:1276)
  RSP 00007f0fb35fd690 RIP 0000000000a92fab  [image code] java.net.InetAddress.getAllByName(InetAddress.java:1192)
  RSP 00007f0fb35fd700 RIP 0000000000a945ab  [image code] java.net.InetAddress.getAllByName(InetAddress.java:1126)
  RSP 00007f0fb35fd700 RIP 0000000000a945ab  [image code] java.net.InetAddress.getByName(InetAddress.java:1076)
  RSP 00007f0fb35fd730 RIP 000000000085f6db  [image code] io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:146)
  RSP 00007f0fb35fd730 RIP 000000000085f6db  [image code] io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:143)
  RSP 00007f0fb35fd730 RIP 000000000085f6db  [image code] com.oracle.svm.core.jdk.Target_java_security_AccessController.doPrivileged(SecuritySubstitutions.java:90)
  RSP 00007f0fb35fd730 RIP 000000000085f6db  [image code] io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:143)
  RSP 00007f0fb35fd730 RIP 000000000085f6db  [image code] io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:43)
  RSP 00007f0fb35fd770 RIP 000000000086050a  [image code] io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:63)
  RSP 00007f0fb35fd7b0 RIP 000000000086046f  [image code] io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:55)
  RSP 00007f0fb35fd7e0 RIP 000000000085fd59  [image code] io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:57)
  RSP 00007f0fb35fd820 RIP 000000000085ff34  [image code] io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:32)
  RSP 00007f0fb35fd850 RIP 000000000085e236  [image code] io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:108)
  RSP 00007f0fb35fd8a0 RIP 00000000006805ea  [image code] io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:208)
  RSP 00007f0fb35fd910 RIP 00000000006800fe  [image code] io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:170)
  RSP 00007f0fb35fd960 RIP 000000000067fac0  [image code] io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:145)
  RSP 00007f0fb35fd990 RIP 000000000099a402  [image code] io.vertx.core.net.impl.ChannelProvider.connect(ChannelProvider.java:49)
  RSP 00007f0fb35fd9f0 RIP 000000000092a289  [image code] io.vertx.core.http.impl.HttpChannelConnector.doConnect(HttpChannelConnector.java:207)
  RSP 00007f0fb35fda70 RIP 0000000000929c4a  [image code] io.vertx.core.http.impl.HttpChannelConnector.connect(HttpChannelConnector.java:103)
  RSP 00007f0fb35fdac0 RIP 000000000095d9e4  [image code] io.vertx.core.http.impl.pool.Pool.createConnection(Pool.java:291)
  RSP 00007f0fb35fdb00 RIP 000000000095be1f  [image code] io.vertx.core.http.impl.pool.Pool.lambda$acquireConnection$3(Pool.java:201)
  RSP 00007f0fb35fdb00 RIP 000000000095be1f  [image code] io.vertx.core.http.impl.pool.Pool$$Lambda$534/1732033473.run(Unknown Source)
  RSP 00007f0fb35fdb30 RIP 000000000087f458  [image code] io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
  RSP 00007f0fb35fdb50 RIP 000000000089558f  [image code] io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
  RSP 00007f0fb35fdbc0 RIP 000000000073e473  [image code] io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465)
  RSP 00007f0fb35fdc10 RIP 000000000088f024  [image code] io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
  RSP 00007f0fb35fdcb0 RIP 00000000008896cf  [image code] io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
  RSP 00007f0fb35fdcd0 RIP 0000000000a42f1f  [image code] java.lang.Thread.run(Thread.java:748)
  RSP 00007f0fb35fdcf0 RIP 0000000000660c24  [image code] com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:238)
  RSP 00007f0fb35fdd50 RIP 000000000061cf24  [image code] com.oracle.svm.core.code.CEntryPointCallStubs.com_002eoracle_002esvm_002ecore_002eposix_002ethread_002ePosixJavaThreads_002epthreadStartRoutine_0028com_002eoracle_002esvm_002ecore_002eposix_002ethread_002ePosixJavaThreads_0024ThreadStartData_0029(generated:0)

Use runtime option -R:-InstallSegfaultHandler if you don't want to use SubstrateSegfaultHandler.

Bye bye ...

[olpaw]

@pmlopes please provide me a minimal sample along with build and run instructions so that I can investigate further. Thanks!

[pmlopes]

Here a reproducer: dns-issue.zip

I've reduced the code to:

import java.net.InetAddress;

class Main {
  public static void main(String[] args) throws Exception {
    InetAddress.getByName("www.google.com");
  }
}

And to be sure that it's not an issue of building on my machine I've included a Dockerfile that builds the app:

# GraalVM docker image used for AoT compilation
FROM panga/graalvm-ce:latest AS build-aot
# Add maven wrapper
ADD mvnw app/
ADD mvnw.cmd app/
ADD .mvn app/.mvn/
RUN cd /app && \
    ./mvnw -version
# Add pom and download dependencies
ADD pom.xml app/
RUN cd /app && \
    ./mvnw dependency:copy-dependencies
# Add sources
ADD src app/src/
# Set working dir
WORKDIR /app
# Build (java side)
RUN ./mvnw package
# Build image
RUN native-image \
    --no-server \
    --static \
    -Djava.net.preferIPv4Stack=true \
    -H:+ReportUnsupportedElementsAtRuntime \
    -jar "target/dns.jar"
# Create new image from scratch
FROM alpine
# Copy generated native executable from build-aot
COPY --from=build-aot /app/dns /dns
# Set the entrypoint
ENTRYPOINT [ "/dns" ]

And the execution ends with the same result as seen on the initial report.

[olpaw]

thanks! i will have a look.

[olpaw]

The problem is that dns lookup on modern systems always requires dynamically linked libc. Despite having a static binary it will do the following:

sudo docker run --cap-add=SYS_PTRACE d44052a1b485
execve("/dns", ["/dns"], 0x7ffc5d453830 /* 3 vars */) = 0
uname({sysname="Linux", nodename="12cf783382ce", ...}) = 0
brk(NULL)                               = 0x124e000
brk(0x124f1c0)                          = 0x124f1c0
arch_prctl(ARCH_SET_FS, 0x124e880)      = 0
set_tid_address(0x124eb50)              = 9
set_robust_list(0x124eb60, 24)          = 0
rt_sigaction(SIGRTMIN, {sa_handler=0x611270, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x610850}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {sa_handler=0x611300, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x610850}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
readlink("/proc/self/exe", "/dns", 4096) = 4
brk(0x12701c0)                          = 0x12701c0
brk(0x1271000)                          = 0x1271000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
clock_gettime(CLOCK_MONOTONIC, {tv_sec=8377, tv_nsec=144423243}) = 0
mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc62e8eb000
munmap(0x7fc62e8eb000, 86016)           = 0
munmap(0x7fc62ea00000, 962560)          = 0
rt_sigaction(SIGSEGV, {sa_handler=0x407810, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x610850}, NULL, 8) = 0
gettimeofday({tv_sec=1533031235, tv_usec=833139}, NULL) = 0
gettimeofday({tv_sec=1533031235, tv_usec=833175}, NULL) = 0
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(3)                                = 0
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(3)                                = 0
open("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
brk(0x1270000)                          = 0x1270000
open("/etc/host.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
futex(0xbd40b4, FUTEX_WAKE_PRIVATE, 2147483647) = 0
open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=74, ...}) = 0
read(3, "# Generated by NetworkManager\nse"..., 4096) = 74
read(3, "", 4096)                       = 0
close(3)                                = 0
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/tls//libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib/x86_64-linux-gnu/tls/", 0x7ffc618e8bb0) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/tls/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib/x86_64-linux-gnu/tls", 0x7ffc618e8bb0) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu//libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib/x86_64-linux-gnu/", 0x7ffc618e8bb0) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib/x86_64-linux-gnu", 0x7ffc618e8bb0) = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/tls//libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64-linux-gnu/tls/", 0x7ffc618e8bb0) = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/tls/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64-linux-gnu/tls", 0x7ffc618e8bb0) = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu//libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64-linux-gnu/", 0x7ffc618e8bb0) = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64-linux-gnu", 0x7ffc618e8bb0) = -1 ENOENT (No such file or directory)
open("/lib/tls//libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib/tls/", 0x7ffc618e8bb0)       = -1 ENOENT (No such file or directory)
open("/lib/tls/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib/tls", 0x7ffc618e8bb0)        = -1 ENOENT (No such file or directory)
open("/lib//libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/lib/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/usr/lib/tls//libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/tls/", 0x7ffc618e8bb0)   = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/tls", 0x7ffc618e8bb0)    = -1 ENOENT (No such file or directory)
open("/usr/lib//libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/usr/lib/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/lib//libnss_files.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib//libnss_files.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/dev/tty", O_RDWR|O_NOCTTY|O_NONBLOCK) = -1 ENXIO (No such device or address)
writev(2, [{iov_base="*** Error in `", iov_len=14}, {iov_base="/dns", iov_len=4}, {iov_base="': ", iov_len=3}, {iov_base="free(): invalid pointer", iov_len=23}, {iov_base=": 0x", iov_len=4}, {iov_base="00007fc62e901490", iov_len=16}, {iov_base=" ***\n", iov_len=5}], 7*** Error in `/dns': free(): invalid pointer: 0x00007fc62e901490 ***
) = 69
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc62eaea000
mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7fc626900000
munmap(0x7fc626900000, 24117248)        = 0
munmap(0x7fc62c000000, 42991616)        = 0
mprotect(0x7fc628000000, 143360, PROT_READ|PROT_WRITE) = 0
futex(0xbd29b0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
write(2, "======= Backtrace: =========\n", 29======= Backtrace: =========
) = 29
writev(2, [{iov_base="[0x", iov_len=3}, {iov_base="6282e1", iov_len=6}, {iov_base="]\n", iov_len=2}], 3[0x6282e1]
) = 11
writev(2, [{iov_base="[0x", iov_len=3}, {iov_base="630826", iov_len=6}, {iov_base="]\n", iov_len=2}], 3[0x630826]
) = 11
writev(2, [{iov_base="[0x", iov_len=3}, {iov_base="634857", iov_len=6}, {iov_base="]\n", iov_len=2}], 3[0x634857]
) = 11
writev(2, [{iov_base="[0x", iov_len=3}, {iov_base="65b5b8", iov_len=6}, {iov_base="]\n", iov_len=2}], 3[0x65b5b8]
) = 11
writev(2, [{iov_base="[0x", iov_len=3}, {iov_base="44014b", iov_len=6}, {iov_base="]\n", iov_len=2}], 3[0x44014b]
) = 11
write(2, "======= Memory map: ========\n", 29======= Memory map: ========
) = 29
open("/proc/self/maps", O_RDONLY)       = 3
read(3, "00400000-00929000 r-xp 00000000 "..., 1024) = 798
write(2, "00400000-00929000 r-xp 00000000 "..., 79800400000-00929000 r-xp 00000000 fd:01 2380146                            /dns
00b28000-00bcf000 rwxp 00528000 fd:01 2380146                            /dns
00bcf000-00bd6000 rwxp 00000000 00:00 0 
0124e000-01270000 rwxp 00000000 00:00 0                                  [heap]
7fc628000000-7fc628023000 rwxp 00000000 00:00 0 
7fc628023000-7fc62c000000 ---p 00000000 00:00 0 
7fc62e900000-7fc62ea00000 rwxp 00000000 00:00 0 
7fc62eaea000-7fc62eaeb000 rwxp 00000000 00:00 0 
7ffc618cb000-7ffc618ec000 rwxp 00000000 00:00 0                          [stack]
7ffc61969000-7ffc6196c000 r--p 00000000 00:00 0                          [vvar]
7ffc6196c000-7ffc6196e000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
) = 798
read(3, "", 1024)                       = 0
close(3)                                = 0
rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
tgkill(9, 9, SIGABRT)                   = 0
--- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=9, si_uid=0} ---
+++ killed by SIGABRT (core dumped) +++

[olpaw]

If I add all the shared objects the dns binary tries to open it succeeds. E.g:

dns-issue> ls lib/
ld-linux-x86-64.so.2  libc.so.6  libnss_dns.so.2  libnss_files.so.2  libresolv.so.2

Then in the Dockerfile I add COPY lib /lib one line before ENTRYPOINT the image runs in the container just fine. But please don't do that. Instead do not use --static for images that depend on services that require dynamic linking to work correctly.

[olpaw]

The above being said there is one possible route you could take if you insist on having a fully static binary even for such usecases. See https://sourceware.org/glibc/wiki/FAQ#Even_statically_linked_programs_need_some_shared_libraries_which_is_not_acceptable_for_me.__What_can_I_do.3F

So you can either build yourself a system (a docker image) that has c-library built with --enable-static-nss or build the image on a system that predates the libc changes responsible for this.

[pmlopes]

@olpaw thanks. The goal here was to make the smallest image, like one would with go. I was just exploring the limits.

[jamesward]

Ran into this issue as well. Here is the terrible workaround:
https://github.com/jamesward/graalvm-dns-issue/blob/master/Dockerfile.scratch-fix#L33-L39

[olpaw]

We have a better solution for this now. Build a static image with muslc. @gradinac knows all about it.