How to compile a version static library(.a) of online2-tcp-nnet3-decode-faster?

[ppamorim]

Hi,

Currently I am trying to compile online2-tcp-nnet3-decode-faster.cc and create an static library. I tried following the command ./configure --static inside src folder but it didn't create any static library. I tried to do this by myself based on this script below that I got from the makefile:

(this code below build a executable)

g++   -g   online2-tcp-nnet3-decode-faster.o ../online2/kaldi-online2.a ../ivector/kaldi-ivector.a ../nnet3/kaldi-nnet3.a ../chain/kaldi-chain.a ../nnet2/kaldi-nnet2.a ../cudamatrix/kaldi-cudamatrix.a ../decoder/kaldi-decoder.a ../lat/kaldi-lat.a ../fstext/kaldi-fstext.a ../hmm/kaldi-hmm.a ../feat/kaldi-feat.a ../transform/kaldi-transform.a ../gmm/kaldi-gmm.a ../tree/kaldi-tree.a ../util/kaldi-util.a ../matrix/kaldi-matrix.a ../base/kaldi-base.a ../kaldi/tools/openfst-1.6.7/lib/libfst.a -framework Accelerate -lm -lpthread -ldl  -o online2-tcp-nnet3-decode-faster

I got all the static libraries and tried to create a new big object file, it wasn't sucessful and very boring to do.

What you guys recommend? I need something that let me only import a header and link the library easily.

Thank you.

[danpovey]

We can't diagnose from saying it "wasn't successful". More details are needed. You might want to look at the Makefile in one of the non-"bin" directories and see how libraries are created there.. just copying/modifying the relevant Makefile lines from there would probably do it. You should look at one of the link lines used when creating some Kaldi binary (as output by `make`), to get an idea of what might be necessary when linking your final application.

…

On Sun, Sep 15, 2019 at 6:37 AM Pedro Paulo Amorim ***@***.***> wrote: Hi, Currently I am trying to compile online2-tcp-nnet3-decode-faster.cc and create an static library. I tried following the command ./configure --static inside src folder but it didn't create any static library. I tried to do this by myself based on this script below that I got from the makefile: g++ -g online2-tcp-nnet3-decode-faster.o ../online2/kaldi-online2.a ../ivector/kaldi-ivector.a ../nnet3/kaldi-nnet3.a ../chain/kaldi-chain.a ../nnet2/kaldi-nnet2.a ../cudamatrix/kaldi-cudamatrix.a ../decoder/kaldi-decoder.a ../lat/kaldi-lat.a ../fstext/kaldi-fstext.a ../hmm/kaldi-hmm.a ../feat/kaldi-feat.a ../transform/kaldi-transform.a ../gmm/kaldi-gmm.a ../tree/kaldi-tree.a ../util/kaldi-util.a ../matrix/kaldi-matrix.a ../base/kaldi-base.a ../kaldi/tools/openfst-1.6.7/lib/libfst.a -framework Accelerate -lm -lpthread -ldl -o online2-tcp-nnet3-decode-faster I got all the static libraries and tried to create a new big object file, it wasn't sucessful and very boring to do. What you guys recommend? I need something that let me only import a header and link the library easily. Thank you. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3589?email_source=notifications&email_token=AAZFLO5E42E24ZFRH2OV6Q3QJVYUHA5CNFSM4IWYR2R2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HLM5KCQ>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAZFLO4NPV6QPSFHF4IUUOLQJVYUHANCNFSM4IWYR2RQ> .

[ppamorim]

@danpovey Thank you. I will try to follow the path that you guys used to create the kaldi-online2.a archive.

[ppamorim]

@danpovey Do you know where I can find any example of how to link kaldi-online2.a with another C++ source code? I am trying here but I getting Undefined symbols for architecture x86_64. Thank you anyway.

[danpovey]

You wouldn't just link that library because it depends on other libraries. You should look at the commands that Kaldi uses to compile binaries in the bin directories, those commands will be printed out when it is compiled.

…

On Mon, Sep 16, 2019 at 12:09 PM Pedro Paulo Amorim < ***@***.***> wrote: @danpovey <https://github.com/danpovey> Do you know where I can find any example of how to link kaldi-online2.a with another C++ source code? I am trying here but I getting Undefined symbols for architecture x86_64. Thank you anyway. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3589?email_source=notifications&email_token=AAZFLO6TJDASBVAAI7XYD73QJ5LPLA5CNFSM4IWYR2R2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6YVOVA#issuecomment-531715924>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAZFLO3WFJCUEU6KFP2QRWDQJ5LPLANCNFSM4IWYR2RQ> .

[nshmyrev]

@ppamorim something like https://github.com/alphacep/kaldi-websocket-python/blob/master/Makefile should be able to help you.

[ppamorim]

@danpovey @nshmyrev I managed to compile it and list it against all kaldi static libraries sucessfully, thank you for this.
Now I am getting random crashes of Incorrect checksum for freed object and SIGSEGV when I call a new stack instance of DecodableNnetSimpleLoopedInfo, sometimes the script works but most of the time it doesn't and crash.
But this only happens when I playing with the static library added to an external project, with the same script being called from the runnable, it does work.

Basically what I did was to get that script and move all allocation to the stac, it strange to see these errors happening since the only difference is the static artifact.

[nshmyrev]

That's probably not Kaldi issue but the issue with your code. You can close this one, share your complete code on GitHub, I will take a look.

[danpovey]

Check that the compilation flags are the same, e.g. there might be some flags which lead to a difference in the size of certain objects. Look at what's on the line where it does `g++ .... -o something.o ...`, there will be various options and -DXXXX-type flags. Also you could try using valgrind to debug any memory issues.

…

On Tue, Sep 17, 2019 at 6:06 PM Nickolay V. Shmyrev ***@***.***> wrote: That's probably not Kaldi issue but the issue with your code. You can close this one, share your complete code on GitHub, I will take a look. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ppamorim]

@danpovey

What I was able to detect are these flags.

To compile the object:

g++ -std=c++11 -I.. -I/Users/user/Repository/kaldi/tools/openfst-1.6.7/include -O1 \
 -Wall -Wno-sign-compare -Wno-unused-local-typedefs -Wno-deprecated-declarations \
 -Winit-self -DKALDI_DOUBLEPRECISION=0 -DHAVE_EXECINFO_H=1 -DHAVE_CXXABI_H \
 -DHAVE_CLAPACK -msse -msse2 -pthread -g  -Wno-mismatched-tags   -c \
 -o online2-tcp-nnet3-decode-faster.o online2-tcp-nnet3-decode-faster.cc

To build the bin:

g++   -g   online2-tcp-nnet3-decode-faster.o ../online2/kaldi-online2.a \ 
../ivector/kaldi-ivector.a ../nnet3/kaldi-nnet3.a ../chain/kaldi-chain.a \ 
../nnet2/kaldi-nnet2.a ../cudamatrix/kaldi-cudamatrix.a ../decoder/kaldi-decoder.a  \
../lat/kaldi-lat.a ../fstext/kaldi-fstext.a ../hmm/kaldi-hmm.a ../feat/kaldi-feat.a \ 
../transform/kaldi-transform.a ../gmm/kaldi-gmm.a ../tree/kaldi-tree.a \ 
../util/kaldi-util.a ../matrix/kaldi-matrix.a ../base/kaldi-base.a  \ 
/Users/user/Repository/kaldi/tools/openfst-1.6.7/lib/libfst.a -framework Accelerate \
 -lm -lpthread -ldl  -o online2-tcp-nnet3-decode-faster

I am still investigating what is missing, I am using the makefile from Kaldi to compile the library too.

Edit 1:

To compile the lib, the makefile is doing:

g++ -std=c++11 -I.. -I/Users/user/Repository/kaldi/tools/openfst-1.6.7/include -O1 \
 -Wall -Wno-sign-compare -Wno-unused-local-typedefs -Wno-deprecated-declarations \
 -Winit-self -DKALDI_DOUBLEPRECISION=0 -DHAVE_EXECINFO_H=1 -DHAVE_CXXABI_H \
 -DHAVE_CLAPACK -msse -msse2 -pthread -g  -Wno-mismatched-tags   -c \
 -o online2-tcp2-nnet3-decode-faster.o online2-tcp2-nnet3-decode-faster.cc

ar -cr kaldi-tcp2.a online2-tcp2-nnet3-decode-faster.o
ranlib kaldi-tcp2.a

To compile the library that uses the static library created above, I run:

g++ -g -std=c++11 \
 main.cpp \
-L./lib/ \
-lkaldi \
-lkaldi-online2 \
-lkaldi-ivector \
-lkaldi-nnet3 \
-lkaldi-chain \
-lkaldi-nnet2 \
-lkaldi-cudamatrix \
-lkaldi-decoder \
-lkaldi-lat \
-lkaldi-fstext \
-lkaldi-hmm \
-lkaldi-feat \
-lkaldi-transform \
-lkaldi-gmm \
-lkaldi-tree \
-lkaldi-util \
-lkaldi-matrix \
-lkaldi-base \
-lfst \
-framework Accelerate \
-lm -lpthread -ldl \
-DKALDI_DOUBLEPRECISION=0 -DHAVE_EXECINFO_H=1 -DHAVE_CXXABI_H -DHAVE_CLAPACK -msse -msse2 -pthread \
-o build/run

[ppamorim]

The script starts crashing here:

header:

class Foo {
  private:
    //...
    TransitionModel* trans_model_;
    nnet3::AmNnetSimple* am_nnet_;
    nnet3::DecodableNnetSimpleLoopedInfo* decodable_info_;
    //...
};

cc:

this->trans_model_ = new TransitionModel();
this->am_nnet_ = new nnet3::AmNnetSimple();
{
    bool binary;
    Input ki(nnet3_rxfilename, &binary);
    trans_model_->Read(ki.Stream(), binary);
    am_nnet_->Read(ki.Stream(), binary);
    SetBatchnormTestMode(true, &(am_nnet_->GetNnet()));
    SetDropoutTestMode(true, &(am_nnet_->GetNnet()));
    nnet3::CollapseModel(nnet3::CollapseModelConfig(), &(am_nnet_->GetNnet()));
}

// this object contains precomputed stuff that is used by all decodable
// objects.  It takes a pointer to am_nnet because if it has iVectors it has
// to modify the nnet to accept iVectors at intervals.
this->decodable_info_ = new nnet3::DecodableNnetSimpleLoopedInfo(
    *decodable_opts_,
    am_nnet_); //Crash here, sometimes it works...

Full stacktrace:

Click to expand

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
* frame #0: 0x00007fff6695d2c6 libsystem_kernel.dylib`__pthread_kill + 10
  frame #1: 0x00007fff66a18bf1 libsystem_pthread.dylib`pthread_kill + 284
  frame #2: 0x00007fff668c76a6 libsystem_c.dylib`abort + 127
  frame #3: 0x00007fff669d6077 libsystem_malloc.dylib`malloc_vreport + 545
  frame #4: 0x00007fff669eee0d libsystem_malloc.dylib`malloc_zone_error + 183
  frame #5: 0x00007fff669cef17 libsystem_malloc.dylib`tiny_malloc_from_free_list + 1310
  frame #6: 0x00007fff669ce43b libsystem_malloc.dylib`tiny_malloc_should_clear + 314
  frame #7: 0x00007fff669ce20f libsystem_malloc.dylib`szone_malloc_should_clear + 66
  frame #8: 0x00007fff669ccc99 libsystem_malloc.dylib`malloc_zone_malloc + 103
  frame #9: 0x00007fff669ccc15 libsystem_malloc.dylib`malloc + 24
  frame #10: 0x00007fff63aa1378 libc++abi.dylib`operator new(unsigned long) + 40
  frame #11: 0x0000000100008509 run`std::__1::__libcpp_allocate(__size=<unavailable>, __align=0) at new:252:10 [opt]
  frame #12: 0x000000010000aff6 run`std::__1::allocator<int>::allocate(this=<unavailable>, __n=<unavailable>, (null)=<unavailable>) at memory:1813:37 [opt]
  frame #13: 0x000000010000af59 run`std::__1::allocator_traits<std::__1::allocator<int> >::allocate(__a=<unavailable>, __n=<unavailable>) at memory:1546:21 [opt]
  frame #14: 0x000000010000af0c run`std::__1::__split_buffer<int, std::__1::allocator<int>&>::__split_buffer(this=0x00007ffeefbfe620, __cap=1, __start=0, __a=<unavailable>) at __split_buffer:311:29 [opt]
  frame #15: 0x000000010000ae10 run`std::__1::__split_buffer<int, std::__1::allocator<int>&>::__split_buffer(this=<unavailable>, __cap=<unavailable>, __start=<unavailable>, __a=<unavailable>) [opt] [artificial]
  frame #16: 0x000000010000acc3 run`void std::__1::vector<int, std::__1::allocator<int> >::__push_back_slow_path<int const&>(this=0x0000000102bf4350, __x=0x00007ffeefbfe7fc) at vector:1611:49 [opt]
  frame #17: 0x000000010000aa70 run`std::__1::vector<int, std::__1::allocator<int> >::push_back(this=<unavailable>, __x=<unavailable>) [opt] [artificial]
  frame #18: 0x00000001000cf6a0 run`kaldi::nnet3::ComputationGraphBuilder::AddDependencies(this=0x00007ffeefbfecb0, cindex_id=<unavailable>) at nnet-computation-graph.cc:713:36 [opt]
  frame #19: 0x00000001000ce405 run`kaldi::nnet3::ComputationGraphBuilder::BuildGraphOneIter(this=0x00007ffeefbfecb0) at nnet-computation-graph.cc:906:7 [opt]
  frame #20: 0x00000001000ce1b2 run`kaldi::nnet3::ComputationGraphBuilder::Compute(this=0x00007ffeefbfecb0, request=<unavailable>) at nnet-computation-graph.cc:479:5 [opt]
  frame #21: 0x00000001001013cb run`kaldi::nnet3::EvaluateComputationRequest(nnet=0x0000000103b000d0, request=0x00007ffeefbfee90, is_computable=0x00007ffeefbfeef8) at nnet-utils.cc:77:11 [opt]
  frame #22: 0x0000000100101ab0 run`kaldi::nnet3::ComputeSimpleNnetContextForShift(nnet=0x0000000103b000d0, input_start=<unavailable>, window_size=<unavailable>, left_context=0x0000000101f088a0, right_context=0x0000000101f09090) at nnet-utils.cc:130:3 [opt]
  frame #23: 0x00000001001016ce run`kaldi::nnet3::ComputeSimpleNnetContext(nnet=0x0000000103b000d0, left_context=0x0000000103b00140, right_context=0x0000000103b00144) at nnet-utils.cc:172:12 [opt]
  frame #24: 0x00000001000e50a2 run`kaldi::nnet3::AmNnetSimple::Read(this=0x0000000103b000d0, is=0x0000000103b00168, binary=true) at am-nnet-simple.cc:53:3 [opt]
  frame #25: 0x0000000100002d7e run`kaldi::Foo::Foo(this=0x0000000101f00300, argc=14, argv=0x00007ffeefbff530) at online2-tcp2-nnet3-decode-faster.cc:176:17 [opt]
  frame #26: 0x0000000100000b3b run`main(argc=15, argv=0x00007ffeefbff530) at main.cpp:34:39 <-- Entry point of my program
  frame #27: 0x00007fff668223d5 libdyld.dylib`start + 

[danpovey]

An error in malloc would normally point to an earlier error. You should run in valgrind.

…

On Wed, Sep 18, 2019 at 1:15 PM Pedro Paulo Amorim ***@***.***> wrote: Full stacktrace: Click to expand (lldb) bt * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT * frame #0: 0x00007fff6695d2c6 libsystem_kernel.dylib`__pthread_kill + 10 frame #1: 0x00007fff66a18bf1 libsystem_pthread.dylib`pthread_kill + 284 frame #2: 0x00007fff668c76a6 libsystem_c.dylib`abort + 127 frame #3: 0x00007fff669d6077 libsystem_malloc.dylib`malloc_vreport + 545 frame #4: 0x00007fff669eee0d libsystem_malloc.dylib`malloc_zone_error + 183 frame #5: 0x00007fff669cef17 libsystem_malloc.dylib`tiny_malloc_from_free_list + 1310 frame #6: 0x00007fff669ce43b libsystem_malloc.dylib`tiny_malloc_should_clear + 314 frame #7: 0x00007fff669ce20f libsystem_malloc.dylib`szone_malloc_should_clear + 66 frame #8: 0x00007fff669ccc99 libsystem_malloc.dylib`malloc_zone_malloc + 103 frame #9: 0x00007fff669ccc15 libsystem_malloc.dylib`malloc + 24 frame #10: 0x00007fff63aa1378 libc++abi.dylib`operator new(unsigned long) + 40 frame #11: 0x0000000100008509 run`std::__1::__libcpp_allocate(__size=<unavailable>, __align=0) at new:252:10 [opt] frame #12: 0x000000010000aff6 run`std::__1::allocator<int>::allocate(this=<unavailable>, __n=<unavailable>, (null)=<unavailable>) at memory:1813:37 [opt] frame #13: 0x000000010000af59 run`std::__1::allocator_traits<std::__1::allocator<int> >::allocate(__a=<unavailable>, __n=<unavailable>) at memory:1546:21 [opt] frame #14: 0x000000010000af0c run`std::__1::__split_buffer<int, std::__1::allocator<int>&>::__split_buffer(this=0x00007ffeefbfe620, __cap=1, __start=0, __a=<unavailable>) at __split_buffer:311:29 [opt] frame #15: 0x000000010000ae10 run`std::__1::__split_buffer<int, std::__1::allocator<int>&>::__split_buffer(this=<unavailable>, __cap=<unavailable>, __start=<unavailable>, __a=<unavailable>) [opt] [artificial] frame #16: 0x000000010000acc3 run`void std::__1::vector<int, std::__1::allocator<int> >::__push_back_slow_path<int const&>(this=0x0000000102bf4350, __x=0x00007ffeefbfe7fc) at vector:1611:49 [opt] frame #17: 0x000000010000aa70 run`std::__1::vector<int, std::__1::allocator<int> >::push_back(this=<unavailable>, __x=<unavailable>) [opt] [artificial] frame #18: 0x00000001000cf6a0 run`kaldi::nnet3::ComputationGraphBuilder::AddDependencies(this=0x00007ffeefbfecb0, cindex_id=<unavailable>) at nnet-computation-graph.cc:713:36 [opt] frame #19: 0x00000001000ce405 run`kaldi::nnet3::ComputationGraphBuilder::BuildGraphOneIter(this=0x00007ffeefbfecb0) at nnet-computation-graph.cc:906:7 [opt] frame #20: 0x00000001000ce1b2 run`kaldi::nnet3::ComputationGraphBuilder::Compute(this=0x00007ffeefbfecb0, request=<unavailable>) at nnet-computation-graph.cc:479:5 [opt] frame #21: 0x00000001001013cb run`kaldi::nnet3::EvaluateComputationRequest(nnet=0x0000000103b000d0, request=0x00007ffeefbfee90, is_computable=0x00007ffeefbfeef8) at nnet-utils.cc:77:11 [opt] frame #22: 0x0000000100101ab0 run`kaldi::nnet3::ComputeSimpleNnetContextForShift(nnet=0x0000000103b000d0, input_start=<unavailable>, window_size=<unavailable>, left_context=0x0000000101f088a0, right_context=0x0000000101f09090) at nnet-utils.cc:130:3 [opt] frame #23: 0x00000001001016ce run`kaldi::nnet3::ComputeSimpleNnetContext(nnet=0x0000000103b000d0, left_context=0x0000000103b00140, right_context=0x0000000103b00144) at nnet-utils.cc:172:12 [opt] frame #24: 0x00000001000e50a2 run`kaldi::nnet3::AmNnetSimple::Read(this=0x0000000103b000d0, is=0x0000000103b00168, binary=true) at am-nnet-simple.cc:53:3 [opt] frame #25: 0x0000000100002d7e run`kaldi::Foo::Foo(this=0x0000000101f00300, argc=14, argv=0x00007ffeefbff530) at online2-tcp2-nnet3-decode-faster.cc:176:17 [opt] frame #26: 0x0000000100000b3b run`main(argc=15, argv=0x00007ffeefbff530) at main.cpp:34:39 <-- Entry point of my program frame #27: 0x00007fff668223d5 libdyld.dylib`start + — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3589?email_source=notifications&email_token=AAZFLO5WISLHGEK5JMFJN3DQKIEWHA5CNFSM4IWYR2R2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD67WYDI#issuecomment-532638733>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAZFLO4LRPMPUI4HAAKVJCTQKIEWHANCNFSM4IWYR2RQ> .

[ppamorim]

@danpovey Thank you, here is the full valgrind result: https://gist.github.com/ppamorim/86cfb75f8586e10ef6e9abfbeab07354

[ppamorim]

I will share a minimum project with this error with you guys.

[danpovey]

Sorry, I'm not going to help with this! This is a very simple bug in your code. Not a Kaldi issue. Closing the issue.