libloading support

[Wenzel]

Hi,

I would like to rewrite a virtual machine introspection library in Rust:
https://github.com/libvmi/libvmi

This library is building a unified API accross hypervisor's VMI APIs.

I considered bindgen for the task, which would be perfect to generate a kvm-sys or xen-sys crate, but I have a requirement that it should be able to load these libraries dynamically, at runtime, by locating a lib.so file, loading it with libloading. like a plugin.
https://docs.rs/libloading/0.5.0/libloading/

I haven't found anything in the docs regarding dynamic loading.

Is it a use case that have already think about ?
How difficult would it be to implement ?

Thanks !

[emilio]

So, basically what you want, I assume, is something like bindgen, but generating something like:

struct Lib {
    foo: Option<extern "C" fn(i32) -> i32>,
    bar: Option<extern "C" fn(*mut c_void) -> ()>,
}

And such, instead of:

extern "C" {
    fn foo(arg: i32) -> i32;
    fn bar(arg: *mut c_void) -> ();
}

(and co, which is basically what bindgen does)

Is that right? If so, I don't know what the best way to approach this would be.

Tweaking bindgen to generate something like this wouldn't be hard, but it's not clear it'd be flexible enough for the general case...

It should be possible to use syn and such to parse bindgen's output and generate an arbitrary library as well... Depends on the requirements really, and on how stable the API is.

If the API is stable (which it should if it's a dylib), then writing a rust program that processes bindgen's output and outputs the struct you want with some helpers and such should be pretty easy. We could even put it in the repo if it's useful for more people.

I'd be happy to help if you want to give that a shot and get stuck or what not.

[emilio]

I could also be convinced to add a special mode to bindgen that generates what you want, but I think adding it in a separate program would be easier.

[hug-dev]

Hi!
This functionality would also be very helpful for us and I think for every crate loading dynamically a C library with a very big API.
As far as I know, people today seem to manually write the API when using libloading (example with PKCS 11).

Having bindgen (or another program) do it would be fantastic!
I would be happy to have a look on how to do it as a separate program: do you have any leads on where would one start 😃 ?

[emilio]

I think this generally needs a concrete proposal of what would be needed and what kind of inputs would bindgen get. I don't know the use case well enough. I'm not sure if my comments above reflect the use case given I got no reply from the reporter.

In particular, how is bindgen supposed to know which functions are part of the dylib vs. part of some other imported header?

[astraw]

I am also interested in this.

At least in linux, I think the output of "nm -D" could be used by bindgen to know which functions are in the .so file.

[Wenzel]

@emilio i have to apologize here, I read your reply in public transport In March 2018, but couldn't add a reply.
Then, I just forgot.

So, basically what you want, I assume, is something like bindgen, but generating something like:

struct Lib {
    foo: Option<extern "C" fn(i32) -> i32>,
    bar: Option<extern "C" fn(*mut c_void) -> ()>,
}

Indeed, that's what I'm looking for.
For example, there would be an option in bindgen to optionally generate dynamic bindings:

    let bindings = bindgen::Builder::default()
        // The input header we would like to generate
        // bindings for.
        .header(XEN_HEADERS_WRAPPER)
        // Run rustfmt on the bindings
        .rustfmt_bindings(true)
        // allow dynamic linking
        .allow_dynamic_link(true)
        // Finish the builder and generate the bindings.
        .generate()

And this would generate a struct Lib containiing function pointer, that would be filled when you would run Lib::init().

I could also be convinced to add a special mode to bindgen that generates what you want, but I think adding it in a separate program would be easier.

I think it could be integrated in bindgen, as it's one more option available for users ?

But I'm still interested, more now than before !

Thanks @emilio

[Wenzel]

In particular, how is bindgen supposed to know which functions are part of the dylib vs. part of some other imported header?

When you are loading a library at runtime, your program has to know in advance which symbols it wants to load from this library.
dlopen or libloading doesn't provide a way to enumerate the available symbols.

Also, you have to explicitely define the functions parameters.
Example with xenctrl in libvmi:
https://github.com/libvmi/libvmi/blob/master/libvmi/driver/xen/libxc_wrapper.h#L33

typedef struct {
    void *handle;

    /* Xen 4.1+ */
    xc_interface* (*xc_interface_open)
    (xentoollog_logger *logger, xentoollog_logger *dombuild_logger, unsigned open_flags);

    int (*xc_interface_close)
    (xc_interface *xch);

    int (*xc_version)
    (xc_interface *xch, int cmd, void *arg);

    void* (*xc_map_foreign_range)
    (xc_interface *xch, uint32_t domid, int size, int prot, unsigned long mfn );

So what we could have, is a build.rs where the user specifies which symbols he is interested in, and bindgen can generate the function signature automatically, as well as a struct Lib, where the init() method is calling libloading and filling the function pointers.

what do you think @hug-dev, @emilio ?

[emilio]

At least in linux, I think the output of "nm -D" could be used by bindgen to know which functions are in the .so file.

Right, but this grows quite a dependency on bindgen, plus part of the point of using a dylib is that there can be functions that are not present in different versions of the library. Bindgen itself uses this quite a lot. For example, if you have libclang 9, you'll be able to use some functions that are not present in libclang 5.

Just magically running nm or any other tool like that would mean that generating bindings for a library would depend on the library the user had installed at the point of generating the bindings which is not great.

Listing all the symbols looks fair enough. I think that should be reasonably straight-forward to implement.

Also, btw, this wouldn't be that bad to implement as an input step to bindgen if you're allowed to use some C++. bindgen generates the right thing for this:

extern "C" void my_function();

struct my_type_t {
  int b;
};

extern "C" int my_other_function(struct my_type_t*);

// This could be generated from build.rs pretty easily and used as an input to
// bindgen.
struct Library {
  decltype(my_function)* my_function;
  decltype(my_other_function)* my_other_function;
};

[hug-dev]

In particular, how is bindgen supposed to know which functions are part of the dylib vs. part of some other imported header?

Can it only care about the functions defined in the main header file as opposed to the ones that get imported from it?
I would expect a dynamic library to implement all of the APIs defined in its header file? Maybe I am wrong because I do not see the whole picture or there are other uses cases.
That would make things easier if the developper does not have to write by hand all of the function symbols they want to load.

Concerning integration with libloading, as the get method returns a Symbol<T> where T is the function type maybe it would be convenient for the fields of the Library structure to also be of type Symbol<T>?

So what we could have, is a build.rs where the user specifies which symbols he is interested in, and bindgen can generate the function signature automatically, as well as a struct Lib, where the init() method is calling libloading and filling the function pointer

I agree! A pub fn init<P: AsRef<OsStr>>(filename: P) -> Result<bindgen::Library> wound indeed be excellent if it could:

• call libloading::Library::new on the filename given to create a new libloading Library instance
• instantiate a new bindgen::Library instance by assigning each field function_name to lib.get(b"function_name\0").unwrap()

[Wenzel]

hey @emilio , could you give us an update of the status of this issue so far ?

are we still brainstorming on the interface, should we write some specifications, or maybe are you allocating this project for this year's GSoC ?

Thanks.