Deploying hint cross-platform

[yaxu]

I'd love to deploy haskell binaries making use of hint, to avoid having to ship a haskell interpreter with the whole of haskell.

#3 demonstrates this working under linux, with https://github.com/gelisam/deploy-hint/ from @gelisam as an example, although that isn't up to date.

Are there any other guides / examples towards getting this working? + anything for mac os / windows?

[gelisam]

The only reason I targeted linux in that example is because it's easy to create a linux VM which doesn't have ghc on it and then automate a few steps to automatically test that it works. The same approach should work on other platforms.

[yaxu]

That's good to know @gelisam, the discussion on #3 shows up some issues around making binaries relocatable though, which I think is more of an issue under mac os / windows. I'd love to see some worked examples on that.

[yaxu]

I've been trying to get hint working with a package database for some time..

A hint-based interpreter making use of tidal, which works fine with unsafeRunInterpreterWithArgs:
https://github.com/tidalcycles/tidali/blob/master/src/Main.hs

However with unsafeRunInterpreterWithArgsLibdir,

alex@stanage:~/src/tidali$ ./dist/build/tidali/tidali
tidali: sigaction
tidali: sigaction
"bd sn"
tidali: sigaction
tidali: sigaction
Error: Compile error: <no location info>: error:
    Could not load module ‘Sound.Tidal.Context’
    It is a member of the package ‘tidal-1.0.5-8A9BMedqL2r4J1dJKy2bMe’
    which is unusable due to missing dependencies:
      bifunctors-5.5.3-1BoyHhHPokNHM3BHbuJlQo containers-0.6.0.1 hashable-1.2.7.0-2SI038axTEd7AEZJ275kpi hosc-0.17-I360LVldbm9EFN3gVUhetB mtl-2.2.2 mwc-random-0.14.0.0-IaBBbMxuD2dBfng7mxv9hk network-2.8.0.0-AkCJm1aNSYz7ekXKYyI0pF parsec-3.1.13.0 random-1.1-3ypV4EIycgb35PKjTYYr5q text-1.2.3.1 time-1.8.0.2 vector-0.12.0.2-H1Eu1OCXL0L9y980iV8EwU
    Use -v to see a list of the files searched for.
tidali: <stdin>: hGetLine: end of file

However, ghc-pkg says it's all there:

ghc-pkg-8.6.3 --package-db=haskell-libs/package.conf.d list
    StateVar-1.1.1.1
    base-4.12.0.0
    base-orphans-0.8
    bifunctors-5.5.3
    blaze-builder-0.4.1.0
    cabal-doctest-1.0.6
    colour-2.3.4
    comonad-5.0.4
    contravariant-1.5
    data-binary-ieee754-0.4.4
    data-default-class-0.1.2.0
    distributive-0.6
    exceptions-0.10.0
    ghc-paths-0.1.0.9
    ghc-prim-0.5.3
    hashable-1.2.7.0
    hint-0.9.0
    hosc-0.17
    integer-gmp-1.0.2.0
    math-functions-0.3.1.0
    monad-loops-0.4.3
    mwc-random-0.14.0.0
    network-2.8.0.0
    primitive-0.6.4.0
    random-1.1
    rts-1.0
    safe-0.3.17
    semigroups-0.18.5
    tagged-0.8.6
    temporary-1.3
    th-abstraction-0.2.10.0
    tidal-1.0.5
    transformers-compat-0.6.2
    vector-0.12.0.2
    vector-th-unbox-0.2.1.6

a ghc-pkg check only turns up warnings about missing haddock stuff:

alex@stanage:~/src/tidali$ ghc-pkg-8.6.3 --package-db=haskell-libs/package.conf.d check 2>&1|grep -v haddock 
alex@stanage:~/src/tidali$ 

Here's what I'm doing to build the haskell-libs folder:
https://github.com/tidalcycles/tidali/blob/master/make-haskell-libs.sh

I'm using ${pkgroot} in an effort at making the folder relocatable but trying with an absolute path doesn't make a difference that I can see.

At this point I'm not trying to get fake_gcc to work or collect system libraries, I'm just trying to get a useable haskell-libs folder.

Here's one clue, ghci seems to fail to find a shared object file (even though it does exist):

alex@stanage:~/src/tidali/haskell-libs$ ghci-8.6.3 -package-db package.conf.d
GHCi, version 8.6.3: http://www.haskell.org/ghc/  :? for help
Prelude> import Sound.Tidal.Context 
Prelude Sound.Tidal.Context> (parseBP_E "bd sn") :: Pattern String
can't load .so/.DLL for: ./libHSbase-orphans-0.8-9c3aMxqS7QbGl4Ot5i8lwK-ghc8.6.3.so (libHSbase-orphans-0.8-9c3aMxqS7QbGl4Ot5i8lwK-ghc8.6.3.so: cannot open shared object file: No such file or directory)
Prelude Sound.Tidal.Context> 
Leaving GHCi.
alex@stanage:~/src/tidali/haskell-libs$ ls ./libHSbase-orphans-0.8-9c3aMxqS7QbGl4Ot5i8lwK-ghc8.6.3.so
./libHSbase-orphans-0.8-9c3aMxqS7QbGl4Ot5i8lwK-ghc8.6.3.so

I'm not sure what to try next. Any clues very much appreciated!

[yaxu]

Having deleted my ~/.ghc folder, ghci is happily loading and using the tidal package in haskell-libs (verified with strace).

Still the same error from hint, though.

[yaxu]

Seems I just didn't have all the dependencies in place.. getting there

[yaxu]

Working now, and ${pkgroot} is working fine as a way to give relative paths in .conf files, so no need to run recache any more, @gelisam

[yaxu]

I never got this to work in a satisfactory way. Anyone get hint binary deployment working recently?

[gelisam]

But you said it was working in January! Did I change something which broke your solution, or did you mean that you only got part of the solution working? If so, which part works?

[yaxu]

It was a while ago now.. I managed to get to the point that it seemed to work on my laptop, but not to get it to successfully transfer on to another system. I then had little idea how to replicate this process on mac os or windows. I should try again, but was wondering if anyone else had this working recently, perhaps with an up-to-date example.

[gelisam]

I haven't worked on https://github.com/gelisam/deploy-hint/ since I created the prototype, but since I pinned both hint, ghc, and the linux VM to a specific version, the demo should still work, no? I haven't run it in a while...

It sounds like the work you did so far was to update my demo to make it work with a more recent version of hint and ghc? That does sound like a good first step. Do you have a PR for deploy-hint? It would be a shame for all that work to get lost to time!

Like I said last year, the only reason I targeted linux in that example is because it's easy to create and control a linux VM. Do you know of a good way to create Windows or MacOS VMs? Wait, we do that for a living, I should ask at work 😅

[yaxu]

@gelisam I was trying to get this working with the tidal library, getting inspiration from deploy-hint, rather than using it directly. My workings are here: https://github.com/tidalcycles/tidali , I'm not sure how useful that is, though! I don't have ready access to windows or macos.

[yaxu]

We have tidal-listener in the tidal repo automatically building across linux, mac and windows now :)

[gelisam]

I'm glad you found a solution to your problem! Is it a workaround which somehow makes it possible to use tidal without having to deploy hint on a machine which doesn't have ghc installed, or did you find a way to package up the required library files as part of the tidal installation, like in the deploy-hint prototype? In either case, do you have some advice for other projects who might want to accomplish the same thing?

[yaxu]

@polymorphicengine did the work but yes it follows the good principles of your deploy-hint prototype. I believe there are still some problems, e.g. Mac OS big sur and later refuse to run the fake_gcc due to malware protection. I don't think we found a workaround for this.

Other projects could have a look at the github workflow here: https://github.com/polymorphicengine/tidal-gui/