Crash: Unexpectedly found nil on 1.6.0-async-await

[qusc]

Describe the bug

I caught this crash on my server running on 1.6.0-async-await: (stack trace seems to be missing line numbers, probably because it's compiled in release mode (?))

Received signal 4. Backtrace:
0x55b3845db582, Backtrace.(printBacktrace in _B82A8C0ED7C904841114FDF244F9E58E)(signal: Swift.Int32) -> () at /.build/checkouts/swift-backtrace/Sources/Backtrace/Backtrace.swift:66
0x7fc4820e67df
0x55b384e2a195, Swift runtime failure: Unexpectedly found nil while implicitly unwrapping an Optional value at /<compiler-generated>:0
0x55b384e2a195, generic specialization <GRPC.ProtobufSerializer<MyGRPCResponseMessage>, GRPC.ProtobufDeserializer<MyGRPCMessage>> of GRPC.AsyncServerHandler._interceptResponse(_: A.Input, metadata: GRPC.MessageMetadata) -> () at /<compiler-generated>:0
0x55b384e2db79, generic specialization <GRPC.ProtobufSerializer<MyGRPCResponseMessage>, GRPC.ProtobufDeserializer<MyGRPCMessage>> of closure #1 () -> () in GRPC.AsyncServerHandler.interceptResponse(_: A.Input, metadata: GRPC.MessageMetadata) -> () at /<compiler-generated>:0
0x55b384e2db79, generic not re-abstracted specialization <GRPC.ProtobufSerializer<MyGRPCResponseMessage>, GRPC.ProtobufDeserializer<MyGRPCMessage>> of closure #1 () -> () in GRPC.AsyncServerHandler.interceptResponse(_: A.Input, metadata: GRPC.MessageMetadata) -> () at /<compiler-generated>:0
0x55b384e2db79, partial apply forwarder for generic not re-abstracted specialization <GRPC.ProtobufSerializer<MyGRPCResponseMessage>, GRPC.ProtobufDeserializer<MyGRPCMessage>> of closure #1 () -> () in GRPC.AsyncServerHandler.interceptResponse(_: A.Input, metadata: GRPC.MessageMetadata) -> () at /<compiler-generated>:0
0x55b384c512cb, reabstraction thunk helper from @escaping @callee_guaranteed () -> () to @escaping @callee_guaranteed () -> (@out ()) at /<compiler-generated>:0
0x55b384c512cb, partial apply forwarder for reabstraction thunk helper from @escaping @callee_guaranteed () -> () to @escaping @callee_guaranteed () -> (@out ()) at /<compiler-generated>:0
0x55b384c4ac00, reabstraction thunk helper from @escaping @callee_guaranteed () -> (@out ()) to @escaping @callee_guaranteed () -> () at /<compiler-generated>:0
0x55b384c4ac00, closure #4 () -> () in NIOPosix.SelectableEventLoop.run() throws -> () at /.build/checkouts/swift-nio/Sources/NIOPosix/SelectableEventLoop.swift:520
0x55b384c4ac00, reabstraction thunk helper from @callee_guaranteed () -> (@error @owned Swift.Error) to @escaping @callee_guaranteed () -> (@out (), @error @owned Swift.Error) at /<compiler-generated>:0
0x55b384c4ac00, generic specialization <()> of NIOPosix.withAutoReleasePool<A>(() throws -> A) throws -> A at /.build/checkouts/swift-nio/Sources/NIOPosix/SelectableEventLoop.swift:28
0x55b384c4ac00, NIOPosix.SelectableEventLoop.run() throws -> () at /.build/checkouts/swift-nio/Sources/NIOPosix/SelectableEventLoop.swift:519
0x55b384c361a4, static NIOPosix.MultiThreadedEventLoopGroup.(runTheLoop in _C2B1528F4FBA68A3DBFA89DBAEBE9D4D)(thread: NIOPosix.NIOThread, parentGroup: Swift.Optional<NIOPosix.MultiThreadedEventLoopGroup>, canEventLoopBeShutdownIndividually: Swift.Bool, selectorFactory: () throws -> NIOPosix.Selector<NIOPosix.NIORegistration>, initializer: (NIOPosix.NIOThread) -> (), _: (NIOPosix.SelectableEventLoop) -> ()) -> () at /.build/checkouts/swift-nio/Sources/NIOPosix/MultiThreadedEventLoopGroup.swift:89
0x55b384c361a4, closure #1 (NIOPosix.NIOThread) -> () in static NIOPosix.MultiThreadedEventLoopGroup.(setupThreadAndEventLoop in _C2B1528F4FBA68A3DBFA89DBAEBE9D4D)(name: Swift.String, parentGroup: NIOPosix.MultiThreadedEventLoopGroup, selectorFactory: () throws -> NIOPosix.Selector<NIOPosix.NIORegistration>, initializer: (NIOPosix.NIOThread) -> ()) -> NIOPosix.SelectableEventLoop at /.build/checkouts/swift-nio/Sources/NIOPosix/MultiThreadedEventLoopGroup.swift:110
0x55b384c38cee, partial apply forwarder for closure #1 (NIOPosix.NIOThread) -> () in static NIOPosix.MultiThreadedEventLoopGroup.(setupThreadAndEventLoop in _C2B1528F4FBA68A3DBFA89DBAEBE9D4D)(name: Swift.String, parentGroup: NIOPosix.MultiThreadedEventLoopGroup, selectorFactory: () throws -> NIOPosix.Selector<NIOPosix.NIORegistration>, initializer: (NIOPosix.NIOThread) -> ()) -> NIOPosix.SelectableEventLoop at /<compiler-generated>:0
0x55b384c38fe9, reabstraction thunk helper from @escaping @callee_guaranteed (@guaranteed NIOPosix.NIOThread) -> () to @escaping @callee_guaranteed (@in_guaranteed NIOPosix.NIOThread) -> (@out ()) at /<compiler-generated>:0
0x55b384c38d0d, partial apply forwarder for reabstraction thunk helper from @escaping @callee_guaranteed (@guaranteed NIOPosix.NIOThread) -> () to @escaping @callee_guaranteed (@in_guaranteed NIOPosix.NIOThread) -> (@out ()) at /<compiler-generated>:0
0x55b384c655d5, closure #1 (Swift.Optional<Swift.UnsafeMutableRawPointer>) -> Swift.Optional<Swift.UnsafeMutableRawPointer> in static NIOPosix.ThreadOpsPosix.run(handle: inout Swift.Optional<Swift.UInt>, args: NIOPosix.Box<(body: (NIOPosix.NIOThread) -> (), name: Swift.Optional<Swift.String>)>, detachThread: Swift.Bool) -> () at /.build/checkouts/swift-nio/Sources/NIOPosix/ThreadPosix.swift:105
0x7fc4820dc40a
0x7fc47ffa740e
0xffffffffffffffff

To reproduce

Unfortunately I can't tell what the cause of this crash was. The server ran without problems during all previous testing.

I'll be using my server in production in about two weeks. I was hoping that I could use the async/await-based interface of GRPC already. Could you give me some insights on when this might be production-ready / close to production-ready? Should I switch back to the regular interface for the time being and try to manually bridge to my async code? Thanks!

[glbrntt]

Looking at some of the stack trace:

0x55b384e2db79, generic specialization <GRPC.ProtobufSerializer<MyGRPCResponseMessage>, GRPC.ProtobufDeserializer<MyGRPCMessage>> of closure #1 () -> () in GRPC.AsyncServerHandler.interceptResponse(_: A.Input, metadata: GRPC.MessageMetadata) -> () at /<compiler-generated>:0

This is here:

grpc-swift/Sources/GRPC/AsyncAwaitSupport/GRPCAsyncServerHandler.swift

Lines 539 to 541 in 8fcf4f3

	self.context.eventLoop.execute {
	self._interceptResponse(response, metadata: metadata)
	}

which calls this function on the correct event-loop:

grpc-swift/Sources/GRPC/AsyncAwaitSupport/GRPCAsyncServerHandler.swift

Lines 509 to 532 in 8fcf4f3

	internal func _interceptResponse(_ response: Response, metadata: MessageMetadata) {
	self.context.eventLoop.assertInEventLoop()
	switch self.state {
	case .idle:
	// The user handler cannot send responses before it has been invoked.
	preconditionFailure()
	case var .active(activeState):
	if !activeState.haveSentResponseHeaders {
	activeState.haveSentResponseHeaders = true
	self.state = .active(activeState)
	// Send response headers back via the interceptors.
	self.interceptors.send(.metadata(activeState.context.initialResponseMetadata), promise: nil)
	}
	// Send the response back via the interceptors.
	self.interceptors.send(.message(response, metadata), promise: nil)
	case .completed:
	/// If we are in the completed state then the async writer delegate will have been cancelled,
	/// however the cancellation is asynchronous so there's a chance that we receive this callback
	/// after that has happened. We can drop the response.
	()
	}
	}

0x55b384e2a195, generic specialization <GRPC.ProtobufSerializer<MyGRPCResponseMessage>, GRPC.ProtobufDeserializer<MyGRPCMessage>> of GRPC.AsyncServerHandler._interceptResponse(_: A.Input, metadata: GRPC.MessageMetadata) -> () at /<compiler-generated>:0

and then we hit an IUO which happens to be nil:

0x55b384e2a195, Swift runtime failure: Unexpectedly found nil while implicitly unwrapping an Optional value at /<compiler-generated>:0

The only implicitly unwrapped optional in this class is interceptors which we initialize in init here:

grpc-swift/Sources/GRPC/AsyncAwaitSupport/GRPCAsyncServerHandler.swift

Lines 281 to 291 in 8fcf4f3

	self.interceptors = ServerInterceptorPipeline(
	logger: context.logger,
	eventLoop: context.eventLoop,
	path: context.path,
	callType: .bidirectionalStreaming,
	remoteAddress: context.remoteAddress,
	userInfoRef: userInfoRef,
	interceptors: interceptors,
	onRequestPart: self.receiveInterceptedPart(_:),
	onResponsePart: self.sendInterceptedPart(_:promise:)
	)

and only set to nil in finish here:

grpc-swift/Sources/GRPC/AsyncAwaitSupport/GRPCAsyncServerHandler.swift

Lines 323 to 337 in 8fcf4f3

	internal func finish() {
	switch self.state {
	case .idle:
	self.interceptors = nil
	self.state = .completed
	case .active:
	self.state = .completed
	self.interceptors = nil
	self.userHandlerTask?.cancel()
	case .completed:
	self.interceptors = nil
	}
	}

In finish when we nil out interceptors we also set state to .completed. Back in _interceptResponse we only use interceptors when in the .active state so interceptors must not be nil (although it clearly can be!). In some cases we use interceptors twice: first to send headers if we haven't already done so and then again to send a response message.

The only way we might hit this (as far as I can see) is if the state changes after sending headers which is entirely possible. The fix here should be reasonably straightforward.

---

I'll be using my server in production in about two weeks. I was hoping that I could use the async/await-based interface of GRPC already. Could you give me some insights on when this might be production-ready / close to production-ready?

Async/await in grpc is still experimental and support for it is best-effort for the time being. We'd love to get it merged into main but it currently isn't a priority for us.

[glbrntt]

I realise I didn't explicitly say this: I'll put a patch together to fix this particular bug.

[qusc]

Great, thanks for your explanation :)

[qusc]

I guess this issue should stay open until it is fixed