Steps to get bap source tree available for navigation in ocaml lsp

[sidkshatriya]

I'd like to be able to navigate the full bap source tree using ocaml lsp. This is to help me understand bap source code...

What steps do I need to follow to achieve that?

I've tried:

1. Creating a fresh switch with OCaml 4.12
2. opam install oasis
3. ./configure --enable-everything
4. Unfortunately configure complains that a lot of dependencies are missing and I'm unable to make. Should I install those dependencies myself?

If ocaml lsp is not possible, I guess I could try to use merlin directly. However, I still am not able to build "manually" everything from source. I also keep getting mysterious ppx_bap < 0.14.0 requirements (which cannot be fulfilled).

I've tried a lot of approaches involving opam install bap --deps-only, pinning etc. but I'm not able to get things working for development. (A simple opam install bap in a fresh switch will work for running bap not for development).

Sorry I've missing something basic/trivial. I'm not very familiar with the oasis workflow...

Would appreciate if you can point me in the correct direction...

[ivg]

Here are instructions that will build bap from the source code,

opam switch create 4.09.0 # creates a new switch
eval $(opam env) # activates the opam switch
git clone git@github.com:BinaryAnalysisPlatform/bap.git
cd bap
# the next three commands will install bap depedencies from opam
opam pin add . -n
opam depext bap # installs the system dependencies
opam install bap --deps-only # installs OCaml dependencies
opam pin remove bap
# enables every bap component
./configure --enable-everything --prefix=`opam config var prefix`
make && make reinstall

You may also consider trying our alternative build system that uses omake, it will result in a much faster builds and nearly instant rebuilds, but a little bit harder to setup. Recommended if you want to iteratively rebuild and run bap many times a day.

[sidkshatriya]

@ivg Thanks for your prompt reply. I will try this out and report my results. I notice that you've chosen ocaml 4.09. Does ocaml 4.12 work or should I use 4.09 for bap development?

Also will ocaml lsp work after the alternative build system link or the above steps?

[sidkshatriya]

Another question: I am interested in exploring primus. However, I notice that when you do opam pin add . -n only bap will get pinned. Its seems that the plugins would still get installed from opam but shouldn't I aim for a workflow in which the bap source code that is compiled should be all local?

Hopefully ocaml lsp will work on the primus plugin also...

[sidkshatriya]

(Sorry for inundating you with updates)
I tried following your posted instructions. I get the following failure at the depext stage:

$ opam depext bap                     
# Detecting depexts using vars: arch=x86_64, os=linux, os-distribution=ubuntu, os-family=debian
[ERROR] No solution for bap:   * Missing dependency:                     
            - bap -> ppx_bap < v0.14.0                                             
            no matching version                                                    
                                                                                   
                                                                                   
Command failed: opam list --readonly --external  '--resolve=bap' returned 20

The same issue is encountered in ocaml 4.09.1 and 4.12 switches

[ivg]

Does ocaml 4.12 work or should I use 4.09 for bap development?

yes, 4.12 will work, even 4.13 should work

Also will ocaml lsp work after the alternative build system link or the above steps?

I am using merlin and it works :) Not sure if lsp needs extra setup but don't hesitate to ask for help if it won't work.

Another question: I am interested in exploring primus. However, I notice that when you do opam pin add . -n only bap will get pinned. Its seems that the plugins would still get installed from opam but shouldn't I aim for a workflow in which the bap source code that is compiled should be all local?

BAP is a monorepo will all plugins and libraries in the same git repository. The pin is only necessary to install bap dependencies and system dependencies, not to install bap itself. So everything will be build (with make) and installed (with make reinstall) from the repository.

I tried following your posted instructions. I get the following failure at the depext stage:

Ah, that's annoying. I will fix it (#1323), but you can do it locally in your cloned version. Just remove {= "master"} from "ppx_bap" line in opam/opam. Another fix, is to add bap snapshot opam repository (which is expected to be set up as this opam file is used mostly for CI), e.g.,

opam repo add bap-testing git://github.com/BinaryAnalysisPlatform/opam-repository#testing

[sidkshatriya]

Thank you for your patient and useful explanation. I have been able to make more progress.

$ ./configure --enable-everything --prefix=/home/sidk/bap-explore/_opam --disable-radare2

Configuration:                                                                                                                                                        
ocamlfind: ................................................... /home/sidk/bap-explore/_opam/bin/ocamlfind                                                             
ocamlc: ...................................................... /home/sidk/bap-explore/_opam/bin/ocamlc.opt                                                            
ocamlopt: .................................................... /home/sidk/bap-explore/_opam/bin/ocamlopt.opt                                                          
ocamlbuild: .................................................. /home/sidk/bap-explore/_opam/bin/ocamlbuild                                                            
Package name: ................................................ bap                                                                                                    
...
...

$ make
./setup.exe -quiet -build 
Finished, 0 targets (0 cached) in 00:00:05.
Fatal error: exception Failure("No variable llvm_cxxflags defined when trying to expand \"${llvm_cxxflags}\".")
Raised at Stdlib.failwith in file "stdlib.ml", line 29, characters 17-33
Called from Stdlib__buffer.add_substitute.subst in file "buffer.ml", line 332, characters 22-31
Called from Myocamlbuild.BaseEnvLight.var_expand in file "myocamlbuild.ml", line 496, characters 4-313
Called from Myocamlbuild.expand in file "myocamlbuild.ml", line 1706, characters 15-50
make: *** [Makefile:8: build] Error 1

A github search shows #1115 I've not been able to glean anything useful for my case in it yet. (I'm using ubuntu 20.04). How can I resolve this llvm_cxxflags issue?

Also, interestingly very little related to primus is enabled:

Build Primus Systems Plugin: ................................. false
build Primus round robin scheduler: .......................... false
build Primus promiscuous plugin: ............................. false
Build primus-propagate-taint plugin: ......................... false
Build Primus Machine monad: .................................. false
build Primus loader: ......................................... false
Build Primus Lisp Library: ................................... false
Build Primus Dictionary Tool: ................................ false
build Primus x86 support package: ............................ false
Build Primus Symbolic Executor: .............................. false
Build Primus Lisp Program Testing Framework: ................. false
build Primus print plugin: ................................... false
Build primus-taint plugin: ................................... false
build Primus exploring scheduler: ............................ false
Build Primus Region Tool: .................................... false
build Primus track visited library: .......................... false
build Primus wandering scheduler: ............................ false
build Primus powerpc support package: ........................ false
build Primus mark visited plugin: ............................ false
build Primus limit plugin: ................................... false
Build Primus randomization components: ....................... false
build Primus greedy scheduler: ............................... false

Is this because I've disabled radare2 ?

[sidkshatriya]

This may also have something to do with

opam install bap --deps-only

This installs and builds a lot of stuff. But since I didn't want to install radare2 on my system, I let it fail for bap-radare2. Its possible that some of this is causing the bap build to fail as mentioned above...

<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>                                                                           [1581/2710]
+- The following actions failed                                                                                                                                       
| - build conf-radare2 0.1                                                                                                                                            
+-                                                                                                                                                                    
+- The following changes have been performed                                                                                                                          
| - install angstrom                0.15.0                                                                                                                            
| - install bap-signatures          2.2.0                                                                                                                             
| - install base                    v0.14.1                                                                                                                           
| - install base-bytes              base                                                                                                                              
| - install base64                  3.5.0                                                                                                                             
| - install base_bigstring          v0.14.0                                                                                                                           
| - install base_quickcheck         v0.14.0                                                                                                                           
| - install bigarray-compat         1.0.0                                                                                                                             
| - install bigstringaf             0.7.0                                                                                                                             
| - install bin_prot                v0.14.0                                                                                                                           
| - install biniou                  1.2.1                                                                                                                             
| - install bitstring               3.1.1                                                                                                                             
| - install camlzip                 1.10                                                                                                                              
| - install camomile                1.0.2                                                                                                                             
| - install charInfo_width          1.1.0                                                                                                                             
| - install cmdliner                1.0.4                                                                                                                             
| - install conf-bap-llvm           1.6                                                                                                                               
| - install conf-gmp                3                                                                                                                                 
| - install conf-libcurl            1                                                                                                                                 
| - install conf-pkg-config         2                                                                                                                                 
| - install conf-python-2-7         1.1                                                                                                                               
| - install conf-which              1
| - install conf-zlib               1
| - install core_kernel             v0.14.1
| - install cppo                    1.6.7
| - install csexp                   1.5.1
| - install cstruct                 6.0.0
| - install dune                    2.8.5
| - install dune-configurator       2.8.5
| - install easy-format             1.3.2
| - install ezjsonm                 1.2.0
| - install FrontC                  4.1.0
| - install FrontC                  4.1.0                                                                                                                  [1545/2710]
| - install fieldslib               v0.14.0
| - install fileutils               0.6.3
| - install gen                     0.5.3
| - install hex                     1.4.0
| - install jane-street-headers     v0.14.0
| - install jsonm                   1.0.1
| - install jst-config              v0.14.0
| - install lambda-term             3.1.0
| - install linenoise               1.3.1
| - install lwt                     5.4.1
| - install lwt_log                 1.1.1
| - install lwt_react               1.1.4
| - install menhir                  20210419
| - install menhirLib               20210419
| - install menhirSdk               20210419
| - install mew                     0.1.0
| - install mew_vi                  0.5.0
| - install mmap                    1.1.0
| - install num                     1.4
| - install oasis                   0.4.11
| - install ocaml-compiler-libs     v0.12.3
| - install ocaml-migrate-parsetree 1.8.0
| - install ocaml-syntax-shims      1.0.0
| - install ocamlbuild              0.14.0
| - install ocamlfind               1.9.1
| - install ocamlgraph              2.0.0
| - install ocamlify                0.0.1
| - install ocamlmod                0.0.9
| - install ocplib-endian           1.1
| - install octavius                1.2.2
| - install ocurl                   0.9.1
| - install ounit                   2.2.4
| - install ounit2                  2.2.4
| - install parsexp                 v0.14.0
| - install piqi                    0.7.7
| - install piqilib                 0.6.15
| - install ppx_assert              v0.14.0
| - install ppx_bap                 v0.14.0
| - install ppx_base                v0.14.0
| - install ppx_bench               v0.14.1
| - install ppx_bin_prot            v0.14.0
| - install ppx_cold                v0.14.0
| - install ppx_compare             v0.14.0
| - install ppx_custom_printf       v0.14.0
| - install ppx_derivers            1.2.1
| - install ppx_enumerate           v0.14.0
| - install ppx_expect              v0.14.0
| - install ppx_fields_conv         v0.14.1
| - install ppx_fixed_literal       v0.14.0
| - install ppx_hash                v0.14.0
| - install ppx_here                v0.14.0
| - install ppx_inline_test         v0.14.1
| - install ppx_jane                v0.14.0
| - install ppx_js_style            v0.14.0
| - install ppx_let                 v0.14.0
| - install ppx_module_timer        v0.14.0
| - install ppx_optcomp             v0.14.0
| - install ppx_optional            v0.14.0
| - install ppx_pipebang            v0.14.0
| - install ppx_sexp_conv           v0.14.1
| - install ppx_sexp_message        v0.14.0
| - install ppx_sexp_value          v0.14.0
| - install ppx_stable              v0.14.1
| - install ppx_string              v0.14.1
| - install ppx_tools_versioned     5.4.0
| - install ppx_typerep_conv        v0.14.1
| - install ppx_variants_conv       v0.14.1
| - install ppxlib                  0.15.0
| - install re                      1.9.0
| - install react                   1.2.1
| - install result                  1.4
| - install sedlex                  2.2
| - install seq                     base                                                                                                                   [1472/2710]
| - install sexplib                 v0.14.0
| - install sexplib0                v0.14.0
| - install splittable_random       v0.14.0
| - install stdio                   v0.14.0
| - install stdlib-shims            0.3.0
| - install stringext               1.6.0
| - install time_now                v0.14.0
| - install topkg                   1.0.3
| - install trie                    1.0.0
| - install typerep                 v0.14.0
| - install uchar                   0.0.2
| - install uri                     4.2.0
| - install utop                    2.7.0
| - install uuidm                   0.9.7
| - install uutf                    1.0.2
| - install variantslib             v0.14.0
| - install xmlm                    1.3.0
| - install yojson                  1.7.0
| - install z3                      4.8.9-1
| - install zarith                  1.12
| - install zed                     3.1.0
+- 

The former state can be restored with:
    /home/sidk/.local/bin/opam switch import "/home/sidk/bap-explore/_opam/.opam-switch/backup/state-20210607162257.export"

[ivg]

Yep, it is crucial that all steps should succeed¹. So make sure that step N-1 is successful before moving to step N. It looks like that in your case, we have failed on the conf-radare2 package installation, which is a pseudo-package that checks if radare2 is properly installed. Most likely, it failed because the previous step (opam depext bap) failed to install radare2.

BAP itself is modular and you can pick any components, but the procedure that I am showing is a tradeoff between modularity and ease of use. If you don't want or for some reason can't install radare2 then you can opt into removing this component. Just edit the opam/opam file in your cloned repository and remove and remove conf-radare2 from the list of dependencies. And do not forget to pass --no-radare2 to the configure script.

---

¹⁾ to test if the command succeed in bash and other shells, use $? it will show 0 if everything went fine.

[sidkshatriya]

Thanks again. I removed the conf-radare2 line from opam/opam.

$ opam depext bap
# Detecting depexts using vars: arch=x86_64, os=linux, os-distribution=ubuntu, os-family=debian
# The following system packages are needed:
clang
debianutils
dejagnu
libcurl4-gnutls-dev
libgmp-dev
libzip-dev
llvm-10-dev
llvm-9-dev
m4
pkg-config
python2.7
time
zlib1g-dev
# All required OS packages found.

After this, opam install bap --deps-only ran successfully.

$ ./configure --enable-everything --prefix=/home/sidk/bap-explore/_opam --disable-radare2

(Note the use of --disable-radare2. The configure script does not like --no-radare2)

Unfortunately the pesky error still occurs:

$ make
./setup.exe -quiet -build 
Finished, 0 targets (0 cached) in 00:00:04.
Fatal error: exception Failure("No variable llvm_cxxflags defined when trying to expand \"${llvm_cxxflags}\".")
Raised at Stdlib.failwith in file "stdlib.ml", line 29, characters 17-33
Called from Stdlib__buffer.add_substitute.subst in file "buffer.ml", line 332, characters 22-31
Called from Myocamlbuild.BaseEnvLight.var_expand in file "myocamlbuild.ml", line 496, characters 4-313
Called from Myocamlbuild.expand in file "myocamlbuild.ml", line 1706, characters 15-50
make: *** [Makefile:8: build] Error 1

[ivg]

I see, it looks like your llvm installation is not adding llvm-config to the path (propbably you should install the llvm package). But so far, you can select a specific version of llvm by adding the following to the ./configure script invocation,

--with-llvm-version=9   --with-llvm-config=llvm-config-9

(I will update the building instructions in our wiki after you will succeed)

[sidkshatriya]

I tried with:

--with-llvm-version=10   --with-llvm-config=llvm-config-10

This seems to have done the trick. (I also installed the llvm package on ubuntu but that was not sufficient).

Finished, 2438 targets (0 cached) in 00:10:15

Also how would I turn a few of:

Build Primus Systems Plugin: ................................. false
build Primus round robin scheduler: .......................... false
build Primus promiscuous plugin: ............................. false
Build primus-propagate-taint plugin: ......................... false
Build Primus Machine monad: .................................. false
build Primus loader: ......................................... false
Build Primus Lisp Library: ................................... false
Build Primus Dictionary Tool: ................................ false
build Primus x86 support package: ............................ false
Build Primus Symbolic Executor: .............................. false
Build Primus Lisp Program Testing Framework: ................. false
build Primus print plugin: ................................... false
Build primus-taint plugin: ................................... false
build Primus exploring scheduler: ............................ false
Build Primus Region Tool: .................................... false
build Primus track visited library: .......................... false
build Primus wandering scheduler: ............................ false
build Primus powerpc support package: ........................ false
build Primus mark visited plugin: ............................ false
build Primus limit plugin: ................................... false
Build Primus randomization components: ....................... false
build Primus greedy scheduler: ............................... false

the above to true? I just need a hint, I will explore if multiple switches need to be used.

[ivg]

Don't worry, it is ok that they are false (the real constraint is everything || pkg-name so when you do --enable-everything you enable everything.
After make && make reinstall, you may run bap list plugins to make sure that you have the plugins you want.

[sidkshatriya]

And I'm happy to report that ocamllsp is working too -- I have just done a quick check and code navigation/type info etc. is showing.

I'll give a further update if there are any issues I detect. Thanks for all your help!

[ivg]

Awesome, and happy hacking! I will update the wiki. Also, don't hesitate to drop by our gitter channel, it is where most of the discussion is taking place.