Produce idata content for dll/dylib import on windows

[SlugFiller]

From what I can tell the kind="dylib" link hint is mostly ignored on windows, as rust expects a matching import library to be found. I suggest it should, instead, produce an appropriate idata section in the LLVM code.

For example, for the following:

#[cfg(windows)]
#[link(name = "kernel32", kind="dylib")]
#[allow(non_snake_case)]
extern "system" {
    fn GetStdHandle(nStdHandle: u32) -> *mut u8;
    fn WriteConsoleA(hConsoleOutput: *mut u8, lpBuffer: *const u8, nNumberOfCharsToWrite: u32, lpNumberOfCharsWritten: *mut u32, lpReserved: *const u8) -> i32;
}

Rust should produce (Assuming 32-bit. For 64-bit target, a bunch of i32s should replaced with i64s below):

@__ImageBase = external global i8

define internal i8* @GetStdHandle(i32 %nStdHandle) unnamed_addr alwaysinline nounwind {
    %lpaddr = bitcast i8** getelementptr ([3 x i8*]* @.import.kernel32.ptr, i32 0, i32 0) to i8*(i32 )**
    %addr = load i8*(i32)** %lpaddr
    %ret = call cc 64 i8* %addr(i32 %nStdHandle)
    ret i8* %ret
}

define internal i32 @WriteConsoleA(i8* %hConsoleOutput, i8* %lpBuffer, i32 %nNumberOfCharsToWrite, i32* %lpNumberOfCharsWritten, i8* %lpReserved) unnamed_addr alwaysinline nounwind {
    %lpaddr = bitcast i8** getelementptr ([3 x i8*]* @.import.kernel32.ptr, i32 0, i32 1) to i32(i8*, i8*, i32, i32*, i8*)**
    %addr = load i32(i8*, i8*, i32, i32*, i8*)** %lpaddr
    %ret = call cc 64 i32 %addr(i8* %hConsoleOutput, i8* %lpBuffer, i32 %nNumberOfCharsToWrite, i32* %lpNumberOfCharsWritten, i8* %lpReserved)
    ret i32 %ret
}

%.win32.image_import_descriptor = type {
    i8*, ; Characteristics
    i32, ; TimeDateStamp
    i32, ; ForwarderChain
    i8*, ; DLL Name
    i8* ; FirstThunk
}

@.import.kernel32.dllname = private constant [16 x i8] c"KERNEL32.DLL\00\00\00\00", section ".idata$7"

@.import.kernel32.func.GetStdHandle = private constant [18 x i8] c"\00\00GetStdHandle\00\00\00\00", section ".idata$6"
@.import.kernel32.func.WriteConsoleA = private constant [19 x i8] c"\00\00WriteConsoleA\00\00\00\00", section ".idata$6"

@.import.kernel32.desc = private global [3 x i8*] [
    i8* inttoptr (i32 sub(i32 ptrtoint([18 x i8]* @.import.kernel32.func.GetStdHandle to i32), i32 ptrtoint(i8* @__ImageBase to i32)) to i8*),
    i8* inttoptr (i32 sub(i32 ptrtoint([19 x i8]* @.import.kernel32.func.WriteConsoleA to i32), i32 ptrtoint(i8* @__ImageBase to i32)) to i8*),
    i8* null
], section ".idata$4"
@.import.kernel32.ptr = private global [3 x i8*] [
    i8* inttoptr (i32 sub(i32 ptrtoint([18 x i8]* @.import.kernel32.func.GetStdHandle to i32), i32 ptrtoint(i8* @__ImageBase to i32)) to i8*),
    i8* inttoptr (i32 sub(i32 ptrtoint([19 x i8]* @.import.kernel32.func.WriteConsoleA to i32), i32 ptrtoint(i8* @__ImageBase to i32)) to i8*),
    i8* null
], section ".idata$3"

@.dllimport = appending constant [2 x %.win32.image_import_descriptor] [
    %.win32.image_import_descriptor {
        i8* inttoptr (i32 sub(i32 ptrtoint([3 x i8*]* @.import.kernel32.desc to i32), i32 ptrtoint(i8* @__ImageBase to i32)) to i8*),
        i32 0,
        i32 0,
        i8* inttoptr (i32 sub(i32 ptrtoint([16 x i8]* @.import.kernel32.dllname to i32), i32 ptrtoint(i8* @__ImageBase to i32)) to i8*),
        i8* inttoptr (i32 sub(i32 ptrtoint([3 x i8*]* @.import.kernel32.ptr to i32), i32 ptrtoint(i8* @__ImageBase to i32)) to i8*)
    },
    %.win32.image_import_descriptor {
        i8* null,
        i32 0,
        i32 0,
        i8* null,
        i8* null
    }
], section ".idata$2"

[nagisa]

So… you basically want to have import .lib generated on the fly by rustc?

This, at very least, would be problematic with extern blocks that have overlapping definitions.

[SlugFiller]

Using the right markers for the globals, it's possible to make LLVM handle that gracefully, by concatenating the necessary elements. However, a better solution would be for rust to do overlap checking prior to generating the LLVM code, and generate a single import block for the entire program, thus avoiding any duplicate imports.

At any rate, avoiding duplicate creation of imports can't be much more challenging than avoiding duplicate reification of generics.

[alexcrichton]

Could you elaborate on the rationale for this? Do other compilers do this? e.g. MSVC or clang?

[SlugFiller]

MSVC comes with a (proprietary) tool for producing .lib from .def, and .def from .dll, if you're missing the prior. Most other compilers targeting Windows actually lack this facility, requiring reliance on MSVC's toolchain.

The rationale is twofold:

• First, removing a dependency on a specific toolchain. For instance, a file with the appropriate idata section could be compiled with LLVM and linked with LLD without needing an additional runtime or libraries.
• Second, without this, the kind="dylib" hint is superfluous, since the behavior is identical to kind="static", at least when targeting Windows (I haven't tested how the behavior is when targeting Mac or Linux yet)

[retep998]

The behavior of kind="dylib" and kind="static" are not the same. dylib causes the library to be passed to the linker and static causes rustc to bundle the library into the .rlib. There's also a proposal to make the application of dllimport based on whether the library the definitions are from is dylib or static.

[alexcrichton]

@SlugFiller could you elaborate on the LLD aspect of this? From what I understand import libraries are basically a "linker thing" in the sense that the MSVC linker is the only one that consumes them. The GNU linker in the MinGW toolchain, for example, I believe doesn't use the same strategy? (or at least encodes it differently)

Along those lines I would expect that LLD would basically take care of itself and not require a lot of manual interaction with these sorts of imports and whatnot. I'm not entirely sure about the story here, though, as I've played around very little with LLD.

[SlugFiller]

@alexcrichton MinGW is first and foremost a set of header files and libraries mimicking those available in MSVC, starting with those import libraries. The rest of the toolchain is simply a port of the GCC toolchain. So, I can guarantee you that MinGW consumes import libraries, and is bundled with "off-brand" replicas of import libraries for common Microsoft DLLs. As far as I know, they should be cross-compatible/interchangeable with the Microsoft ones, although there's always a chance that tiny differences would cause incompatibilities if trying to mix-and-match. Avoiding that minefield is reason enough to avoid dependency on them entirely.

LLD, at it's core, is just a linker, but it does successfully consume import libraries, as the changelog for it maintains up to date compatibility reports with import libraries. A recent patch (dated Jun 17), mentions a dependency on an MSVC tool (lib.exe) to produce import libraries. To my knowledge, LLD is not bundled with any import libraries, but can consume both those bundled with MSVC and MinGW. This does mean it still has an external dependency on one of those two.

As I've demonstrated above, import libraries can be created manually through LLVM, thereby eliminating dependency on any external toolchains.

Note that this is not a "linker thing", but a "compiler thing": Most compilers output DLL imports simply as static imports, indistinguishable from dependencies on static libraries. True DLL imports require a matching idata section. The linker can link such an idata section into the program, without needing any special awareness of its contents. In particular, without the idata section, the compiler "erases" the data of which DLL any given import must come from. This information is stored in the idata section.

[retep998]

If this can be done with no runtime overhead over the linker resolving imports from static libraries, and if it can be done without needing the compiler to look at the DLLs itself, and if it doesn't further overload the meaning of kind=dylib but instead uses its own opt-in way of specifying it, then I'd be in favor of something like this.

[SlugFiller]

@retep998 I don't understand what you mean by "runtime overhead". It doesn't use LoadLibrary, if that's what you mean. The idata section is how all Windows programs must define their import tables. It's part of the PE format. Import libraries must use the same method (Although import libraries skip any inlining optimizations LLVM can offer for the import stubs)

The compiler doesn't need to look in the DLLs. The information provided in the link attribute and extern section is sufficient, as I've demonstrated above. Adding an extra attribute for the ordinal hint could also help, but is not necessary.

For not overloading the meaning, a possibility is to use a different type of kind, like kind=dll.

[retep998]

Well if it ends up with the same PE imports as using import libraries, and all it needs is the information in the extern section with that link attribute, then by all means go for it. This might need an RFC though since it does add a new kind (or some other syntax to specify the usage of this feature).

[ahicks92]

if I understand what this does correctly--removes the need to have the import library on Windows--I really want this.
Getting users to have the import libraries in the right places is painful in basically any language. Not to mention getting the users to have the right import libraries for the toolchain they're using, etc. You can probably build a lot of stuff from source, but at least one of my projects requires CMake and 6 or so Python packages. On top of that, it uses Libsndfile (can only be built under MinGW) but must itself be built under VS (because it's audio and it needs the Wasapi interfaces). This setup is perhaps atypical, but it's all for very good reasons. The net result is that getting it working with a build script is tricky, as is providing an import library to everyone who needs one.
Or, if we had this, I could just not bother and keep giving out binary dlls all day long and everyone would be happy. This would simplify my life greatly.
It might also help the Sdl2 bindings, which currently instruct you to copy a library to a MinGW directory and then to put a DLL in your cargo project. Either those instructions or some other instructions elsewhere further specify that you have to be careful to not accidentally use the Visual Studio Sdl or it will all explode.

[retep998]

@SlugFiller I really wish you'd create an RFC for this. It's such a genuinely useful thing to have.

[retep998]

Just as a note, whatever the proposed syntax is, it would need to have a way to specify an ordinal instead of a symbol name, as some DLLs have symbols that are only identifiable by their ordinal rather than their name.