Safer failure with COM/C++/vtable interop and RTTI

[DaZombieKiller]

Interop with COM/vtables is currently performed by allocating an array of function pointers. However, this may be dangerous if some C++ code attempts to use dynamic_cast, typeid, or some other feature that requires the existence of RTTI. This is because the RTTI pointer is located just before the first entry of the vtable, which is out of bounds of the memory that was allocated.

To my knowledge, both the Microsoft and Itanium ABIs will treat a nullptr value for the RTTI pointer to mean "no RTTI is present for this type" (in the case of MSVC, __RTDynamicCast will catch an AV and treat that as "no RTTI"). I am not sure how the older GCC ABI deals with this.

Would it be desirable for the COM and vtable source generators (and the general guidance on vtable interop in .NET) to be changed from this:

void** vtable = new void*[methodCount];
// fill in vtable

to this?

void** vtable = new void*[methodCount + 1];
vtable[0] = nullptr;
vtable++;
// fill in vtable

It would ensure that the RTTI pointer is null and the proper failure messages can be displayed instead of potentially reading an invalid (but readable) pointer.

I think supporting RTTI is out of scope for .NET interop, but adding a dummy null pointer before the vtable seems like a good "best effort" to ensure that the failure is still deterministic if some code attempts to use RTTI with an interop vtable.

[AaronRobinsonMSFT]

@DaZombieKiller This is an interesting idea. I don't think we're opposed to it, but it is a new feature that is a very niche scenario. This isn't something the runtime team is likely to put effort into, but we can see if there is a community member who wants to take a crack at it.

If this were to go forward we would need positive and negative testing for this on all platforms.

[jkoritzinsky]

For reference. the location of the std::type_info* for a type with-respect-to a vtable is specified in the Itanium ABI at item 2.9.4.3. Whoever takes up this work should also make sure that we don't need to handle 2.9.4.2 or add handling (and testing) for it.

There's no good spec for the MSVC data structures, so that will be harder to validate.

[DaZombieKiller]

There's no good spec for the MSVC data structures, so that will be harder to validate.

The closest thing would be the include\rttidata.h, include\ehdata_forceinclude.h and crt\src\vcruntime\rtti.cpp files under VC\Tools\MSVC\<version>\ in Visual Studio.

[AaronRobinsonMSFT]

I'd also like to say that even with an implementation PR, there is no guarantee it will be merged. Setting expectations not trying to say no one should look into this effort.

The biggest pushback for this is the sentiment that we don't have official support for any C++ construct in the interop space. We support C constructs and adding an explicit change for a C++ feature, like the above, forces us to add support that likely has narrow utility and could potentially be limiting for future use cases.

Having concrete user scenarios where this is helpful changes that calculus a lot.

[MichalStrehovsky]

Just wondering - how does this work across C++ ABI breaks? Like if I have a COM object written in VC++ 6.0 and try to use it with RTTI today. Or a clang-generated COM object in VC++?

[AaronRobinsonMSFT]

Just wondering - how does this work across C++ ABI breaks? Like if I have a COM object written in VC++ 6.0 and try to use it with RTTI today. Or a clang-generated COM object in VC++?

It will likely fail in some inscrutable way, unless effort was taken to align the cross ABI. It is related to my pushback and why adding "C++ support" outside of a C++ compiler is generally folly. The Itanium ABI adoption is great for the community and something that has improved a lot of things, but .NET trying to adhere to or even follow it seems premature until the C++ standards committee actually takes an official stance.

[DaZombieKiller]

For VC++6.0 specifically, as far as I know the structure of RTTI data is identical to the current one under x86. The format has only been revised once: pointers were changed to RVAs on x64 to avoid increasing the size of the data. Clang has full support for MSVC RTTI when targeting the MSVC ABI too.

Just to clarify, this proposal isn't about supporting RTTI, it's just about decreasing the chances of a dynamic_cast, typeid, etc. operation resulting in subtle memory corruption instead of a crash. dynamic_cast and the related operations shouldn't be expected to work as a result of this change.

You could compare this to Roslyn detecting Git conflict markers in source files, it's a best-effort attempt to improve the developer experience if something is wrong, but there would be no qualms about removing it if the feature proved to be a blocker for something else.