using nbind / node-gyp with lot's of extra libraries - how to tell node-gyp where all sources are?

[subwaystation]

Take a C++ framework, e.g. https://github.com/vgteam/vg/tree/master/src having lots of dependencies when building.
Now I would like to e.g. nbind all classes in file https://github.com/vgteam/vg/tree/master/src/vg.cpp. The problem is that when executing
"npm run -- node-gyp configure build" the following happens:

../src/vg.hpp:11:10: fatal error: 'omp.h' file not found
#include <omp.h>
^
1 error generated.

This error does not occur if I build the project as specified in the wiki. Following this issue http://stackoverflow.com/questions/25990296/how-to-include-omp-h-in-os-x I have to state that gcc 4.9.3 via homebrew is installed and globally available. Maybe node-gyp does not use gcc / g++?
Is there anyway to pass all the source files necessary for the build to node-gyp or how do I proceed here? (I have a CMakeLists.txt file containing over 3000 lines of sources, can I pass this to node-gyp? Or would node-gyp be able to do this automatically?)

All the best.

[jjrv]

You should probably compile vg separately into a library first, and then link your Node.js addon with it. This will be some work but you can look at libui-node for reference. If you git clone and npm install it, you'll see it loads precompiled binary libraries and then links the nbind-based addon with them. The binding.gyp file there shows how to do it on Windows, Linux and OS X.

Any kind of notes or a tutorial would be very welcome if you get it working.

[jjrv]

If you simply install vg globally, it should be quite easy to configure compiler and linker flags in binding.gyp to use it. libui-node has some extra complexity because it installs the external binaries together with the npm package, which is easier for others using the package.

[subwaystation]

Yeah I already compiled it, as was instructed by the vg wiki.
How would I install it globally?
Could you please give me some hints about the compiler configuration and linker flags?
Basically I have to transfer what it says in the vg makefile intobinding.gyp, right?

I will sort out a detailed description, as soon as it works ;)

[subwaystation]

I looked into the binding.gyp of libui-node and I am not quite sure how to interpret it.
Where does the variable module_root_dir point to (Especially on Mac OS)? Does this differ from OS to OS?
Is -lui the library on Mac OS corresponding to libui.lib on WIN and libui.so on Linux?
If I have compiled whole vg, what files do I have to add here? The object files ending with .o?

[subwaystation]

So out of every *.o file I have created after compiling vg I create a library for each object file as given here: http://www.adp-gmbh.ch/cpp/gcc/create_lib.html. I would like to create a shared library, right? Can I put several object files into one lib?
Then I will have to specify all libs in the binding.gyp file, right? Can I put them to e.g. /usr/lib to make them available globally?

So far so good the theory, but I am not quite getting how I can then (n)bind to these libraries.
Here is what I am thinking of, but it would be some work to do:
Every class and corresponding functions or global functions I want to use via nbind I will have to 'reimplement' again.
Small example, lets stick to your given Greeter example:

1. construction of library file out of hello.cc --> libhello.a
2. create new file hello.cc (is the same name possible? - else hello1.cc) as follows:

#include

void sayHey(std::string name) {
sayHello(name);
}

#include "nbind/nbind.h"

NBIND_GLOBAL() {
function(sayHello);
}

That should work, right?

[jjrv]

Looking at vg's Makefile it seems it should produce a file libvg.a. That file should contain all the code you need (it's an archive of all of vg's relevant .o files in a form the C++ compiler understands).

It seems like you'll have quite a few classes and methods to bind. Looking at the class VG there's several methods like normalize and has_node that should be possible to bind without additional code (for example break_cycles will however require binding also the Edge class).

In theory you should be able to create a new .cpp file like:

#include "vg.hpp"
#include "nbind/api.h"

namespace vg {
  // This inherits methods from VG for directly binding them when possible.
  class JS_VG : public VG {
  public:
    // Anything taking a callback will have to be wrapped!
    for_each_node(nbind::cbFunction lambda) { /* your code here */ }

    // Alternatively you can give your new implementation a new name.
    js_for_each_node(nbind::cbFunction lambda) { /* your code here */ }
  };
} // namespace

#include "nbind/nbind.h"

// You can rename your JS_VG reimplementation back to VG on the JavaScript side.
NBIND_CLASS(vg :: JS_VG, VG) {
  // You don't have to reimplement this, just tell nbind that it exists.
  method(normalize);

  // This C++ method is overloaded so we need to tell nbind which arguments it takes
  // (otherwise the C++ compiler cannot distinguish between overloads).
  multimethod(has_node, args(Node *));

  // If we want to bind several overloads, they need to be renamed uniquely
  // because nbind doesn't yet support overloading by type on the JavaScript side.
  multimethod(has_node, args(id_t), "has_node_id");

  // We've overloaded this with a new implementation.
  multimethod(for_each_node, args(nbind::cbFunction));

  // Alternatively if we renamed it uniquely, we can just rename it back here
  // and have nbind autodetect the arguments. Actually we can use camelCase
  // which is more common in JavaScript.
  method(js_for_each_node, "forEachNode");
}

Note that I didn't really check the syntax of the above very carefully, but hopefully it will get you started...

I think as long as you don't need to call your new C++ methods (reimplemented for nbind purposes) from other classes, you can put your new class declaration in the .cpp file and when required as an argument or return type from other classes, just forward declare it like:

class JS_VG;

[jjrv]

It seems vg doesn't have an install option in its Makefile so maybe you should just copy libvg.a inside your own project and use it from there. You could ask people in the vg project how you're supposed to use it as a library, or look at how its command line tool is linked with the library.

You could look for command line flags in gcc docs if you're targeting Linux or OS X. On OS X the default compiler is probably clang which should be compatible with gcc command line flags.

libui-node is a pretty good reference how to put the compiler flags in your binding.gyp.

I think you should start with a really simple C++ application that will just call a single method from vg and print something. When you get the compilation step working, it should be easier to get your nbind bindings working afterwards. For the basic compilation, the people maintaining vg should also be able to help you. They'll probably be happy that someone is working on Node.js bindings.

[adamnovak]

Hello, I'm from the vg project and vaguely understand the build system. Unfortunately I know nothing about nbind or Gyp, but I'd like to get node bindings working if we can.

We pull in a lot of libraries as Git submodules, because our target environment is academic clusters and other places where users don't have access to a package manager. As part of our Makefile, we manually copy library headers into vg/include and their compiled libraries into vg/lib, and then compile and link against those. All the vg sources and headers are comingled in vg/src, except for some which are generated using Protobuf and live in vg/cpp. Most of the vg code goes into libvg.a which lands in vg/lib next to the dependency libraries. The final binary lands in vg/bin.

To build its dependencies, vg relies on a bunch of recursive Make calls. Some of the dependencies (SDSL-lite) demand to be installed using their own shell scripts.

We've never actually written any code to install vg globally; we've just been adding it to our PATHs manually. Projects that depend on vg generally include it as a Git submodule, build it with recursive Make, and just point to its includes and libraries inside the submodule.

Is it possible to use nbind without getting gyp to be able to build vg and all its dependencies? Is it possible to configure gyp for vg alone, and to call make or other scripts on the bundled dependencies? Or to get this to work, do we have to repackage all the dependencies?

[jjrv]

@adamnovak nbind will work as long as it's possible to link a new project against vg by passing command line flags to the compiler, something like gcc ... -lvg. So ideally if vg can be compiled into a library so that a single test program test.cpp that calls vg can be easily linked with it, the same compiler flags when placed in a binding.gyp file should allow defining Node.js bindings for it.

[jjrv]

@subwaystation It seems you should copy libvg.a and all the .hpp files from vg's src directory to your own project's directory (also create the subdirectories subcommand and unittest and copy the hpp files from those too just in case). Then create a new file test.cpp and try to get it to compile. ultrabubble_eval seems like a good starting point.

Once that works, try:

export CXX=echo
make

Now it should print the necessary gcc command line flags. They need to go in cflags (arguments from the first command that compiles an individual source file) and ldflags (arguments from the last command that links everything together) in binding.gyp.

[subwaystation]

I was able to compile a small example test.cpp making use of the vg library.
My file setup was the following:
../git/vg/
../git/vg_compile_test
../git/nbind-testing

In the attachment one can find the vg_compile_test folder including MAKEFILE and compiler output for MAC OS.
vg_compile_test.zip
Thanks for all the tips!

@jjrv However, when trying to build with node-gyp, I ran into the following problem:

CXX(target) Release/obj.target/nbind/JS_VG.o
In file included from ../JS_VG.cpp:1:
../../vg/src/vg.hpp:11:10: fatal error: 'omp.h' file not found
#include <omp.h>
^
1 error generated.

The omp library is being shipped with g++ 4.9.3 as far as I know.
Do I have to specify which g++ to use in the binding.gyp?
Maybe node-gyp is using clang as default on MAC OS? Still, g++ points to /usr/local/bin/g++ being the correct g++ compiler on my machine (I have used this one to compile vg
in the first place).

Here is the current version of my binding.gyp that I have derived from the compiler output:

{
    "targets": [
        {
            "includes": [
                "auto.gypi"
            ],
            "sources": [
                "JS_VG.cpp"
            ],
            "conditions": [
                ["OS=='mac'", {
                    "xcode_settings": {
                        "OTHER_CFLAGS": [
                            "-c",
                            "-msse4.1",
                            "-fopenmp",
                            "-frtti",
                            "-std=c++11",
                            "-ggdb",
                            "-g",
                            "-I../vg/",
                            "-I../vg/include/"
                        ],
                        "OTHER_LDFLAGS": [
                            "-03",
                            "-msse4.1",
                            "-fopenmp",
                            "-std+c++11",
                            "ggdb",
                            "-g",
                            "-I../vg/",
                            "-I../vg/include",
                            "-lm",
                            "-lpthread",
                            "-ly",
                            "-lbz2",
                            "-lsnappy",
                            "-ldivsufsort",
                            "-ldivsufsort64",
                            "-ljansson",
                            "-L../vg/src/",
                            "-L../vg/lib/"
                        ]
                    },
                }],
            ]
        }
    ],
    "includes": [
        "auto-top.gypi"
    ]
}

[subwaystation]

Or ist node-gyp not able to grasp my -fopenmp CFLAG?

[jjrv]

It should use whatever compiler is defined in the environment variable CXX. You can also set that at the top level of binding.gyp like this:

"make_global_settings": [
  ["CXX",  "/usr/local/bin/g++"],
  ["LINK", "/usr/local/bin/g++"]
]

[subwaystation]

Then I run into the following:

 CXX(target) Release/obj.target/nbind/JS_VG.o
g++: error: unrecognized command line option '-stdlib=libc++'
g++: error: unrecognized command line option '-stdlib=libc++'
make: *** [Release/obj.target/nbind/JS_VG.o] Error 1

Although, I did not even specify this command line option in binding.gyp.

[jjrv]

Try removing "OTHER_LDFLAGS": ["-stdlib=libc++"] from src/nbind.gypi inside nbind. It's needed for clang...

Not sure how gcc will work without it, hopefully it uses an stdlib compatible with C++11 anyway.

Also, I think -03 in LDFLAGS should be -O3 and -std+c++11 should be -std=c++11 or you can just leave them out, they're already in that default nbind.gypi file.

ggdb in OTHER_LDFLAGS should be -ggdb.