Support asynchronous block callback arguments in methods

[LeoNatan]

I am not sure how in scope this request is, but this is a very common use case, where a remote server is requested for information, which is not immediately available.

NSXPCConnection allows for asynchronous block callback arguments, which do not require an immediate result. (This isn't supported by NSConnection/NSDistantObject). Apple does this with NSInvocation, which is able to target blocks.

With eDistantObject, block arguments seem broken or unsupported in two ways. If a block argument is called synchronously in the remote method, there is a +[NSInvocation _invocationWithMethodSignature:frame:]: method signature argument cannot be nil crash in EDOInvocationMessage:

    frame #0: 0x00007fff503b5af0 libobjc.A.dylib`objc_exception_throw
    frame #1: 0x00007fff23b9bc83 CoreFoundation`+[NSInvocation _invocationWithMethodSignature:frame:] + 355
  * frame #2: 0x0000000103e6efb8 DetoxHelper`__38+[EDOInvocationRequest requestHandler]_block_invoke(.block_descriptor=0x0000600003a7cd20, originalRequest=0x0000600002174540, service=0x0000600001f78540) at EDOInvocationMessage.m:313:34
    frame #3: 0x0000000103e6bb1e DetoxHelper`-[EDOExecutor edo_handleMessage:](self=0x0000600003a7ce10, _cmd="edo_handleMessage:", message=0x0000600003a68060) at EDOExecutor.m:143:16
    frame #4: 0x0000000103e6ac2f DetoxHelper`-[EDOExecutor runWithBlock:](self=0x0000600003a7ce10, _cmd="runWithBlock:", executeBlock=0x0000000103e7c660) at EDOExecutor.m:88:5
    frame #5: 0x0000000103e7bca9 DetoxHelper`+[EDOClientService sendSynchronousRequest:onPort:withExecutor:](self=EDOClientService, _cmd="sendSynchronousRequest:onPort:withExecutor:", request=0x000060000216c400, port=0x0000600003478180, executor=0x0000600003a7ce10) at EDOClientService.m:273:9
    frame #6: 0x0000000103e66bfb DetoxHelper`-[EDOObject(self=0x0000600002178700, _cmd="edo_forwardInvocation:selector:returnByValue:", invocation=0x0000600002172d80, selector="waitForIdleWithCompletionHandler:", returnByValue=NO) edo_forwardInvocation:selector:returnByValue:] at EDOObject+Invocation.m:89:32
    frame #7: 0x0000000103e669f9 DetoxHelper`-[EDOObject(self=0x0000600002178700, _cmd="forwardInvocation:", invocation=0x0000600002172d80) forwardInvocation:] at EDOObject+Invocation.m:66:3
    frame #8: 0x00007fff23b9d566 CoreFoundation`___forwarding___ + 838
    frame #9: 0x00007fff23b9f6c8 CoreFoundation`__forwarding_prep_0___ + 120
    frame #10: 0x0000000103e65300 DetoxHelper`__23-[DetoxManager connect]_block_invoke(.block_descriptor=0x0000000103e9b4e0) at DetoxManager.m:58:3
    frame #11: 0x00007fff511fc7f9 libdispatch.dylib`_dispatch_client_callout + 8
    frame #12: 0x00007fff511fda25 libdispatch.dylib`_dispatch_once_callout + 20
    frame #13: 0x0000000103e65184 DetoxHelper`-[DetoxManager connect] [inlined] _dispatch_once(predicate=0x0000000103ea7830, block=0x0000000103e651b0) at once.h:84:3
    frame #14: 0x0000000103e65169 DetoxHelper`-[DetoxManager connect](self=0x000060000347c780, _cmd="connect") at DetoxManager.m:54
    frame #15: 0x0000000103e65036 DetoxHelper`+[DetoxManager load](self=DetoxManager, _cmd="load") at DetoxManager.m:36:3
    frame #16: 0x00007fff503bcdf0 libobjc.A.dylib`load_images + 1226
    frame #17: 0x0000000103dc2d79 dyld_sim`dyld::notifySingle(dyld_image_states, ImageLoader const*, ImageLoader::InitializerTimingList*) + 418
    frame #18: 0x0000000103dcf970 dyld_sim`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 438
    frame #19: 0x0000000103dce786 dyld_sim`ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 188
    frame #20: 0x0000000103dce826 dyld_sim`ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 82
    frame #21: 0x0000000103dc3046 dyld_sim`dyld::initializeMainExecutable() + 129
    frame #22: 0x0000000103dc70fc dyld_sim`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) + 3831
    frame #23: 0x0000000103dc21cd dyld_sim`start_sim + 122
    frame #24: 0x00000001094eb6b7 dyld`dyld::useSimulatorDyld(int, macho_header const*, char const*, int, char const**, char const**, char const**, unsigned long*, unsigned long*) + 2308
    frame #25: 0x00000001094e9375 dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) + 818
    frame #26: 0x00000001094e4227 dyld`dyldbootstrap::start(dyld3::MachOLoaded const*, int, char const**, dyld3::MachOLoaded const*, unsigned long*) + 453
    frame #27: 0x00000001094e4025 dyld`_dyld_start + 37

On the other hand, if the block argument is called asynchronously after the remote method call has ended, nothing happens.

XPC has two ways of dealing with the future callback call. If a normal proxy is used, the resulting asynchronous call is performed on a background queue, and if a synchronous proxy is called, the the client waits until the callback block is invoked.

[wuhao5]

can you provide how NSXPCConnection handles async callback, API etc?

when you mentioned block, you mean the actual block type, the anonymous function, right?

eDO doesn't have future but can provide API that will integrate with future.

[LeoNatan]

Yes, block = lambda = anon function.

Let's look at a simple example:

I have a shared protocol:

@protocol TestServiceProtocol

- (void)upperCaseString:(NSString *)aString withReply:(void (^)(NSString *))reply;
    
@end

On the server (listener), when a new connection arrives, the protocol is explicitly given to the XPC system:

- (BOOL)listener:(NSXPCListener *)listener shouldAcceptNewConnection:(NSXPCConnection *)newConnection
{
    newConnection.exportedInterface = [NSXPCInterface interfaceWithProtocol:@protocol(TestServiceProtocol)];
    newConnection.exportedObject = self;
    [newConnection resume];
    return YES;
}

Finally on the client, the protocol is again provided explicitly. Without providing this, calling a remote method fails with unknown selector.

NSXPCConnection* c = [x initWithServiceName:@"com.LeoNatan.XPCTester.XPCService"];
c.remoteObjectInterface = [NSXPCInterface interfaceWithProtocol:@protocol(TestServiceProtocol)];
[c resume];

[[c synchronousRemoteObjectProxyWithErrorHandler:^(NSError * _Nonnull error) {
	NSLog(@"%@", error);
}] upperCaseString:@"hello" withReply:^(NSString *aString) {
	NSLog(@"Result string was: %@", aString);
}];

On the server, when the method is invoked by the connection, the supplied block is a block created at runtime during decoding of the XPC connection:

(lldb) po [reply debugDescription]
<__NSStackBlock__: 0x7000031e1378>
 signature: "v16@?0@8"
 invoke   : 0x7fff309dd3a2 (/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation`__58-[NSXPCConnection _decodeAndInvokeMessageWithEvent:flags:]_block_invoke.93)

Interestingly, if I use protocol_copyMethodDescriptionList to obtain the protocol methods, the returned argument types are the usual ObjC types:

2019-09-23 12:38:16.612025+0300 ExampleApp[7233:315365] getSpringboardRegionOverride:reply:: v32@0:8@16@?24
2019-09-23 12:38:18.252195+0300 ExampleApp[7233:315365] getServerAnswerForQuestion:reply:: v32@0:8@16@?24
2019-09-23 12:38:19.229449+0300 ExampleApp[7233:315365] getAppleTVMode:: v24@0:8@?16
2019-09-23 12:38:22.786022+0300 ExampleApp[7233:315365] rebuildCache:: v24@0:8@?16

But if I use _protocol_getMethodTypeEncoding, a more interesting types are available:

2019-09-23 13:17:06.789312+0300 ExampleApp[7418:337846] getSpringboardRegionOverride:reply:: v32@0:8@"NSString"16@?<v@?@"NSNumber">24
2019-09-23 13:17:06.789477+0300 ExampleApp[7418:337846] getServerAnswerForQuestion:reply:: v32@0:8@"NSString"16@?<v@?@"NSDictionary">24
2019-09-23 13:17:06.789604+0300 ExampleApp[7418:337846] getAppleTVMode:: v24@0:8@?<v@?@"NSDictionary">16
2019-09-23 13:17:06.789714+0300 ExampleApp[7418:337846] rebuildCache:: v24@0:8@?<v@?B>16

I'm guessing this is the reason they require a protocol, because this type of type metadata is only output for protocols if I recall correctly.

Once the reply block is called on the server, it is serialized and the originally provided client block is then called using NSInvocation.

[LeoNatan]

There also appears to exist _Block_signature which returns the extended signature of an arbitrary block object.

[wuhao5]

- (void)upperCaseString:(NSString *)aString withReply:(void (^)(NSString *))reply;

are you saying that you declare your own method signature where you have a callback block? then in your impl in the server side, the callback is, for the purpose of this discussion, invoked after the method returns. and NSXPCConnection is able to invoke a remote block as is?

I'm guessing this is the reason they require a protocol, because this type of type metadata is only output for protocols if I recall correctly.

I also found in typedefs, more detailed encoding info is provided compared to the encoding directly retrieved from the method or class itself. I would guess this is just how the runtime is programmed right now, protocol or typedef is more typed driven so it would contain more type info? but in either case, the signatures look good to me, and it actually doesn't matter how detail the block signature is, eDO also forwards block invocation but doesn't require the detailed info from encoding, there is a runtime structure that eDO to use to construct the NSInvocation, or to be more precisely, to let objc runtime to construct NSInvocation. And this might be the reason why it didn't work for you as I see your callstack was coming from a dylib.

[LeoNatan]

I only declare a protocol. The system deduces the type of the block and creates a proxy. NSXPC provides two types of proxy objects for clients, an async one and a sync one. The sync one blocks the client call until the reply block is called on the server; the reply block is called on the same thread as method call in client process. The async one does not block, and the reply block is called on a background queue.

Regarding +load, +initialize and attr(constructor), I've hit similar issues of order.
I've resorted to hacks:
https://github.com/wix/DetoxSync/blob/ff1e278cac69fcac2e88195f59ceeef3beeaa71c/DetoxSync/DetoxSync/DetoxSync.pch#L24
https://github.com/wix/DetoxSync/blob/ff1e278cac69fcac2e88195f59ceeef3beeaa71c/DetoxSync/DetoxSync/SyncManager/DTXSyncManager.m#L124

I'm not sure this hack specifically will fit the usecase here, as eDO is using a category. But the current behavior is a bug.

[LeoNatan]

I will open a new issue regarding the load order.

[wuhao5]

I only declare a protocol.

Did you declare the last reply: in your protocol as well?

Regarding +load, +initialize and attr(constructor), I've hit similar issues of order.

Thanks for the sharing, it's a smart workaround. I don't know if there is an order issue in eDO, because when it forwards a block, it should add those two forwarding methods to NSBlock, unless it doesn't because you load it from dylib? because it's a dummy empty block placeholder, it doesn't do anything and just pass through quietly.

[LeoNatan]

The issue here is that +load is called earlier in the start process, before attr(constructor). So the functionality there hasn't had a chance to run yet.

[wuhao5]

but that only matters after the block is invoked remotely? as long as by the time you invoke a remote block, the attr(constructor) is set, you should be good, right?

[LeoNatan]

It's not the behavior I observed. In my case, the remote call was blocking (so blocking in a +load), and so the start process was blocked.

[wuhao5]

Oh I see. Let's continue the discussion on a different thread, and focus on the async call for this one. Both look like legitimate issues to resolve to me.

[LeoNatan]

Regarding the reply:, yes, it is part of the method. I'm not sure how Apple deals with methods that have two blocks, for example, but it's an edge case that is less interesting. The normal use-case is to have a reply block (with or without arguments), to be called asynchronously when the information is available or task is completed remotely.

[wuhao5]

oh so you are saying, Apple forces the method to be a pattern that has a reply: in the end with the return value as its argument?

[LeoNatan]

Right, forgot to mention. In NSXPX, method return value can only be void or NSProgress. Reply block must be used to return a value, but the block doesn’t necessarily have to include arguments.