Skip to content

Support marshalling of null SafeHandle. #47944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Feb 10, 2021

Conversation

AaronRobinsonMSFT
Copy link
Member

@AaronRobinsonMSFT AaronRobinsonMSFT commented Feb 6, 2021

@ghost ghost added the area-Interop-coreclr label Feb 6, 2021
@AaronRobinsonMSFT AaronRobinsonMSFT added this to the 6.0.0 milestone Feb 6, 2021
@AaronRobinsonMSFT
Copy link
Member Author

AaronRobinsonMSFT commented Feb 6, 2021

@jkotas or @MichalStrehovsky I imagine CrossGen2 needs to be updated. But it isn't obvious to me where. I assume around the following. I don't see the expected helpers so I think I need to check for null manually, correct?

class SafeHandleMarshaller : Marshaller
{
private void AllocSafeHandle(ILCodeStream codeStream)
{
var ctor = ManagedType.GetParameterlessConstructor();
if (ctor == null || ((MetadataType)ManagedType).IsAbstract)
{
ThrowHelper.ThrowMissingMethodException(ManagedType, ".ctor",
new MethodSignature(MethodSignatureFlags.None, genericParameterCount: 0,
ManagedType.Context.GetWellKnownType(WellKnownType.Void), TypeDesc.EmptyTypes));
}
codeStream.Emit(ILOpcode.newobj, _ilCodeStreams.Emitter.NewToken(ctor));
}
protected override void EmitMarshalReturnValueManagedToNative()
{
ILEmitter emitter = _ilCodeStreams.Emitter;
ILCodeStream marshallingCodeStream = _ilCodeStreams.MarshallingCodeStream;
ILCodeStream returnValueMarshallingCodeStream = _ilCodeStreams.ReturnValueMarshallingCodeStream;
SetupArgumentsForReturnValueMarshalling();
AllocSafeHandle(marshallingCodeStream);
StoreManagedValue(marshallingCodeStream);
StoreNativeValue(returnValueMarshallingCodeStream);
LoadManagedValue(returnValueMarshallingCodeStream);
LoadNativeValue(returnValueMarshallingCodeStream);
returnValueMarshallingCodeStream.Emit(ILOpcode.call, emitter.NewToken(
InteropTypes.GetSafeHandle(Context).GetKnownMethod("SetHandle", null)));
}
protected override void EmitMarshalArgumentManagedToNative()
{
ILEmitter emitter = _ilCodeStreams.Emitter;
ILCodeStream marshallingCodeStream = _ilCodeStreams.MarshallingCodeStream;
ILCodeStream callsiteCodeStream = _ilCodeStreams.CallsiteSetupCodeStream;
ILCodeStream unmarshallingCodeStream = _ilCodeStreams.UnmarshallingCodestream;
ILCodeStream cleanupCodeStream = _ilCodeStreams.CleanupCodeStream;
SetupArguments();
if (IsManagedByRef && In)
{
PropagateFromByRefArg(marshallingCodeStream, _managedHome);
}
var safeHandleType = InteropTypes.GetSafeHandle(Context);
if (In)
{
if (IsManagedByRef)
PropagateFromByRefArg(marshallingCodeStream, _managedHome);
var vAddRefed = emitter.NewLocal(Context.GetWellKnownType(WellKnownType.Boolean));
LoadManagedValue(marshallingCodeStream);
marshallingCodeStream.EmitLdLoca(vAddRefed);
marshallingCodeStream.Emit(ILOpcode.call, emitter.NewToken(
safeHandleType.GetKnownMethod("DangerousAddRef",
new MethodSignature(0, 0, Context.GetWellKnownType(WellKnownType.Void),
new TypeDesc[] { Context.GetWellKnownType(WellKnownType.Boolean).MakeByRefType() }))));
LoadManagedValue(marshallingCodeStream);
marshallingCodeStream.Emit(ILOpcode.call, emitter.NewToken(
safeHandleType.GetKnownMethod("DangerousGetHandle",
new MethodSignature(0, 0, Context.GetWellKnownType(WellKnownType.IntPtr), TypeDesc.EmptyTypes))));
StoreNativeValue(marshallingCodeStream);
ILCodeLabel lNotAddrefed = emitter.NewCodeLabel();
cleanupCodeStream.EmitLdLoc(vAddRefed);
cleanupCodeStream.Emit(ILOpcode.brfalse, lNotAddrefed);
LoadManagedValue(cleanupCodeStream);
cleanupCodeStream.Emit(ILOpcode.call, emitter.NewToken(
safeHandleType.GetKnownMethod("DangerousRelease",
new MethodSignature(0, 0, Context.GetWellKnownType(WellKnownType.Void), TypeDesc.EmptyTypes))));
cleanupCodeStream.EmitLabel(lNotAddrefed);
}
if (Out && IsManagedByRef)
{
// 1) If this is an output parameter we need to preallocate a SafeHandle to wrap the new native handle value. We
// must allocate this before the native call to avoid a failure point when we already have a native resource
// allocated. We must allocate a new SafeHandle even if we have one on input since both input and output native
// handles need to be tracked and released by a SafeHandle.
// 2) Initialize a local IntPtr that will be passed to the native call.
// 3) After the native call, the new handle value is written into the output SafeHandle and that SafeHandle
// is propagated back to the caller.
var vSafeHandle = emitter.NewLocal(ManagedType);
AllocSafeHandle(marshallingCodeStream);
marshallingCodeStream.EmitStLoc(vSafeHandle);
var lSkipPropagation = emitter.NewCodeLabel();
if (In)
{
// Propagate the value only if it has changed
ILLocalVariable vOriginalValue = emitter.NewLocal(NativeType);
LoadNativeValue(marshallingCodeStream);
marshallingCodeStream.EmitStLoc(vOriginalValue);
cleanupCodeStream.EmitLdLoc(vOriginalValue);
LoadNativeValue(cleanupCodeStream);
cleanupCodeStream.Emit(ILOpcode.beq, lSkipPropagation);
}
cleanupCodeStream.EmitLdLoc(vSafeHandle);
LoadNativeValue(cleanupCodeStream);
cleanupCodeStream.Emit(ILOpcode.call, emitter.NewToken(
safeHandleType.GetKnownMethod("SetHandle",
new MethodSignature(0, 0, Context.GetWellKnownType(WellKnownType.Void),
new TypeDesc[] { Context.GetWellKnownType(WellKnownType.IntPtr) }))));
cleanupCodeStream.EmitLdArg(Index - 1);
cleanupCodeStream.EmitLdLoc(vSafeHandle);
cleanupCodeStream.EmitStInd(ManagedType);
cleanupCodeStream.EmitLabel(lSkipPropagation);
}
LoadNativeArg(callsiteCodeStream);
}
protected override void EmitMarshalArgumentNativeToManaged()
{
throw new NotSupportedException();
}
protected override void EmitMarshalElementNativeToManaged()
{
throw new NotSupportedException();
}
protected override void EmitMarshalElementManagedToNative()
{
throw new NotSupportedException();
}
protected override void EmitMarshalFieldManagedToNative()
{
throw new NotSupportedException();
}
protected override void EmitMarshalFieldNativeToManaged()
{
throw new NotSupportedException();
}
}

@MichalStrehovsky
Copy link
Member

I don't see the expected helpers so I think I need to check for null manually, correct?

Yeah. The background of why this needs to be manual is that the helpers are not public API - and everything crossgen2-generated code calls into needs to be a public API or it becomes a R2R-breaking versioning risk that we're not willing to take.

@stephentoub
Copy link
Member

If memory serves, we have a few places where manual marshaling with DangerousAddRef/Release is being used to work around this. It's hard to search for, but I know of at least:

bool releaseRef = false;
IntPtr fileHandlePtr = IntPtr.Zero;
try
{
if (fileHandle != null)
{
fileHandle.DangerousAddRef(ref releaseRef);
fileHandlePtr = fileHandle.DangerousGetHandle();
}
return Interop.Mswsock.TransmitFile(
socket, fileHandlePtr, 0, 0, overlapped,
needTransmitFileBuffers ? &transmitFileBuffers : null, flags);
}
finally
{
if (releaseRef)
{
fileHandle!.DangerousRelease();
}
}
}

Could we fix that up as well?

@AaronRobinsonMSFT
Copy link
Member Author

@stephentoub I can do that. I was planning on doing that in a follow up PR, but if we are okay with support and usage updates I can accommodate.

@stephentoub
Copy link
Member

I was planning on doing that in a follow up PR

Follow-up is fine. Thanks.

@AaronRobinsonMSFT
Copy link
Member Author

@stephentoub I can update the one that originated this issue. There are many places (approx 250) where these APIs are used directly in the product. Most seem to be focusing on blittability and not the nullable aspect of the SafeHandle (i.e. the SafeHandle itself is assumed non-null).

I think this change is going to require inspection of each use and then an update on an ad-hoc basis. If you know of another like the above I can change it but I don't really know the best way to see all the cases where SafeHandle being null was rewrote and fixed up stack vs where the desire for a blittable P/Invoke was the focus of the change.

@AaronRobinsonMSFT
Copy link
Member Author

Could we fix that up as well?

Yes! Missed that specific request.

@jkotas
Copy link
Member

jkotas commented Feb 8, 2021

Do we need to fix Mono too?

private static readonly SafeCreateHandle s_nullExportString = new SafeCreateHandle();
is another place where this can be used.

@AaronRobinsonMSFT
Copy link
Member Author

Do we need to fix Mono too?

@jkotas Good question. I finally get it drilled into my head to always consider CrossGen2, but now I need to also consider Mono. I will take a look.

/cc @lambdageek

@AaronRobinsonMSFT
Copy link
Member Author

AaronRobinsonMSFT commented Feb 8, 2021

@lambdageek or @CoffeeFlux I assume the correct location for addressing this is at:

mono_mb_emit_ldarg (mb, argnum);
pos = mono_mb_emit_branch (mb, CEE_BRTRUE);
mono_mb_emit_exception (mb, "ArgumentNullException", NULL);

and

} else {
/* safehandle.DangerousAddRef (ref release) */
mono_mb_emit_ldarg (mb, argnum);
mono_mb_emit_ldloc_addr (mb, dar_release_slot);
mono_mb_emit_managed_call (mb, sh_dangerous_add_ref, NULL);
/* Pull the handle field from SafeHandle */
mono_mb_emit_ldarg (mb, argnum);
mono_mb_emit_ldflda (mb, MONO_STRUCT_OFFSET (MonoSafeHandle, handle));
mono_mb_emit_byte (mb, CEE_LDIND_I);
mono_mb_emit_stloc (mb, conv_arg);
}

@vargaz
Copy link
Contributor

vargaz commented Feb 9, 2021

The mono changes look ok to me.

@AaronRobinsonMSFT
Copy link
Member Author

Android arm64 timeout is unrelated.

@stephentoub or @jkotas any other concerns with this change? The Mono team appears to be satisfied with the changes on their side.

Copy link
Member

@jkotas jkotas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thank you!

@AaronRobinsonMSFT
Copy link
Member Author

/azp list

@AaronRobinsonMSFT
Copy link
Member Author

/azp run runtime-coreclr outerloop

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@wfurt
Copy link
Member

wfurt commented Feb 11, 2021

we see new crashes that may be related to this - #48106 and #48124. @ManickaP is leading the investigation but it looks like null pointer from safe handle passed to OpenSSL.

@AaronRobinsonMSFT
Copy link
Member Author

@wfurt Okay. Seems like one of the P/Invokes I updated wasn't supposed to support a null SafeHandle. Do we know which one?

@jkotas
Copy link
Member

jkotas commented Feb 11, 2021

I think it is more likely that one of the PInvokes that you have not updated is affected by this. Before this change, it was throwing nice ArgumentNullException that was caught and handled gracefully. After this change, it is passing NULL into the unmanaged code that results into a fatal crash. If it is the case, it suggests that this change has breaking potential and it may be better to leave it for the source-generated interop plan.

@AaronRobinsonMSFT
Copy link
Member Author

I think it is more likely that one of the PInvokes that you have not updated is affected by this. Before this change, it was throwing nice ArgumentNullException that was caught and handled gracefully.

That would make sense. So users are relying on the IL Stub to check - boo.

If it is the case, it suggests that this change has breaking potential and it may be better to leave it for the source-generated interop plan.

Agree. @stephentoub Any thoughts on reverting this change?

@stephentoub
Copy link
Member

I think it is more likely that one of the PInvokes that you have not updated is affected by this. Before this change, it was throwing nice ArgumentNullException that was caught and handled gracefully.

Any thoughts on reverting this change?

Before we back anything out, can we confirm this? Generally we don't consider removal of ANEs to be a breaking change, even though obviously it's possible someone is depending on it. If we're relying on catching an ANE for our own code to be correct, that's something I'd like to fix, anyway.

@AaronRobinsonMSFT
Copy link
Member Author

Generally we don't consider removal of ANEs to be a breaking change

Ah. Didn't realize that. Okay. @ManickaP Do you happen to know which test this is failing on? It isn't very clear from the logs.

@ManickaP
Copy link
Member

ManickaP commented Feb 11, 2021

There are no logs, the test process crashed with segfault.
From the dump I see few tests that have been running but I cannot say for sure which triggered the crash since it happened on a different thread. If you need this info, I'll dig deeper into it and try to figure it out. Do you want me to?

Anyway, I'm no expert on this, but I assume we can catch and handle managed exception in managed code, but we cannot catch SIGSEGV in catch (Exception ex). Am I right?

@ManickaP
Copy link
Member

BTW, it would be handled here in ProcessOutgoingFramesAsync:

catch (Exception e)
{
if (NetEventSource.Log.IsEnabled()) Trace($"Unexpected exception in {nameof(ProcessOutgoingFramesAsync)}: {e}");
Debug.Fail($"Unexpected exception in {nameof(ProcessOutgoingFramesAsync)}: {e}");
}

@wfurt
Copy link
Member

wfurt commented Feb 11, 2021

I don't think it is a particular test @AaronRobinsonMSFT. The general sequence is like this:
SslStrem is disposed and that calls Dispose on SafeDeleteSslContext()

internal void Close()
{
if (!_remoteCertificateExposed)
{
_remoteCertificate?.Dispose();
_remoteCertificate = null;
}
_securityContext?.Dispose();
_credentialsHandle?.Dispose();
_ssl = null;
GC.SuppressFinalize(this);

That sets the pointer to null:

protected override void Dispose(bool disposing)
{
if (disposing)
{
if (null != _sslContext)
{
_sslContext.Dispose();
_sslContext = null!;
}
}
base.Dispose(disposing);
}

There is pending encrypt/decrypt operation on another thread. In the past I think the p/invoke would fail and I think we would eventually wrap that in ObjectDisposed exception (or pass the raw error about closed safe handle. The safe handle was sufficient to guard agains passing garbage/null to OS call.

@wfurt
Copy link
Member

wfurt commented Feb 11, 2021

BTW while @ManickaP beat me with response. I think the issue is between SslStream and SafeHandles. While HTTP probably should not try to write to disposed stream, I feel it should not crash the process. We could perhaps find different mechanism instead of depending on null. But I'm also not sure if SslStream is only one case.

@AaronRobinsonMSFT
Copy link
Member Author

I think the issue is between SslStream and SafeHandles.

@wfurt Thanks. I will take a look at this.

But I'm also not sure if SslStream is only one case.

Agree. Let me take a look around.

but we cannot catch SIGSEGV in catch (Exception ex). Am I right?

@ManickaP Correct - we will not catch a SIGSEGV.

@wfurt
Copy link
Member

wfurt commented Feb 11, 2021

Could the handle.IsInvalid possibly drive the behavior? e.g. pass the null if the handle instance feels it is valid? While maybe the SafeHandle main purpose is to prevent disappearance while in p/invoke and freeing resource, it was convenient to use it for synchronization as closing the handle on one thread would give predictable failure when used concurrently elsewhere.

As far as the scope, this goes beyond runtime, right @AaronRobinsonMSFT? If I understand it right, this impacts anybody who inherits and extend from SafeHandle, right?

@AaronRobinsonMSFT
Copy link
Member Author

Could the handle.IsInvalid possibly drive the behavior? e.g. pass the null if the handle instance feels it is valid?

@wfurt I don't fully understand this statement.

If I understand it right, this impacts anybody who inherits and extend from SafeHandle, right?

Yes, I suppose. The change was only for P/Invokes marshalling a type derived from SafeHandle. Previously if null was passed for the SafeHandle the generated IL Stub would throw an ArgumentNullException. This made scenarios where a native API could handle a null handle cumbersome because SafeHandle could no longer be used in the signature and thus passing an IntPtr was required.

This PR simply says that a P/Invoke signature with a SafeHandle? is fine and behaves as one would expect - pass on IntPtr.Zero.

@jkotas
Copy link
Member

jkotas commented Feb 11, 2021

behaves as one would expect - pass on IntPtr.Zero.

It is not necessarily what one would expect for SafeHandles where 0 is valid value. For example, SafeHandleMinusOneIsInvalid

@stephentoub
Copy link
Member

Seems like there's enough of a possible issue here that we should both revert the SafeHandle change and also continue to explore why exactly this is causing SslStream problems, as it suggests there's a deeper issue there.

@wfurt
Copy link
Member

wfurt commented Feb 11, 2021

SslStream has dangling #46770 to investigate. That one seems to fail only on Windows and may be different. And pre-dates this PR -> something fishy with SslStream.

@AaronRobinsonMSFT
Copy link
Member Author

It is not necessarily what one would expect for SafeHandles where 0 is valid value. For example, SafeHandleMinusOneIsInvalid

Oh dear. That is a great point @jkotas. I truthfully hadn't considered those scenarios. This makes null ambiguous in the best of circumstances. I will back this change out.

AaronRobinsonMSFT added a commit that referenced this pull request Feb 11, 2021
AaronRobinsonMSFT added a commit that referenced this pull request Feb 12, 2021
@ghost ghost locked as resolved and limited conversation to collaborators Mar 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

SafeHandle marshaler should allow null handles
10 participants