[proposal] Make executables available for download

[jgbradley1]

Can executable versions of Theia be made available with each new release for easier packaging and distribution in docker images? This would result in smaller image size and not require so many dependencies just to run Theia.

I may be misunderstanding the build process, but is it necessary to keep the node_modules folder around when there are options like nexe and pkg out there?

This solution might even be best to add as a build option to the theia cli so developers could use multi-stage builds to produce a final production-ready binary and not require dependencies like node/yarn to be installed.

yarn theia build --mode binary

[akosyakov]

Can executable versions of Theia be made available with each new release for easier packaging and distribution in docker images?

I think it should be a part of docker build to use nexe or pkg. You don't want to use all extensions from this repo in all docker images, and one can add custom Theia/VS Code extensions.

[jgbradley1]

It looks like pkg is the preferred package to use for the past couple years. I can play around with this issue and see if I get anything to work on Theia.

Instead of adding a separate build mode for this
yarn theia build --mode binary
It would make more sense to just add to the build command when not in development mode.

Is anyone aware of issues with software licensing between these two packages that would dictate which one should be used?

[akosyakov]

Can Theia be agnostic to them, that an end user can pick one of them or any new packaging lib in the future? I would imaging that an example of docker image doing packaging would be enough. Or there are any issues that it has to be backed in Theia?

[jgbradley1]

An example docker image would probably be the easiest solution for now. An end user must know quite a bit about Node package development in order to generate a lean binary though. For example, removing the source files via yarn clean works but is not something I would expect to be immediately known by someone learning just enough NodeJS to build their own IDE.

Wrapping that logic up in some way other than just sample code could standardize the production binary builds.

[akosyakov]

An example docker image would probably be the easiest solution for now.

Could we start with it, if it becomes a standard way we can include it in theia-cli? I don't want to rush and enforce one or another way of packaging on everybody. At the end Theia is a framework. Agree though that it should be easier to consume for open-source users interested in self-hosting.

or example, removing the source files via yarn clean works but is not something I would expect to be immediately known by someone learning just enough NodeJS to build their own IDE.

for yarn clean it makes sense to include in theia cli: #3320

[beeman]

I've started a proof of concept with the goal of getting this to work. There's a post on Spectrum where I will share my progress.

[kittaakos]

Here is a POC without the debugging support: https://github.com/kittaakos/theia-nexe
I have briefly verified it with the TS-LS support on macOS and in Gitpod.

[jgbradley1]

Just wanted to point that it looks like codercom encountered similar issues trying to use nexe or pkg. They ended up developing their own nodejs binary compiler, nbin, that looks like it is able to package everything up (including native modules).

[kittaakos]

Just wanted to point that it looks like codercom encountered similar issues trying to use nexe or pkg

Thanks for the pointer! For the record, this does not work out of the box either:

{ Error: ENOENT: no such file or directory, open '/Users/akos.kitta/Desktop/bin-tmp/package.json'
    at Object.openSync (fs.js:439:3)
    at Object.readFileSync (fs.js:344:35)
    at Object.readJsonFile (/Users/akos.kitta/Desktop/theia-nexe/node_modules/@theia/application-package/lib/json-file.js:23:26)
    at ApplicationPackage.get [as pck] (/Users/akos.kitta/Desktop/theia-nexe/node_modules/@theia/application-package/lib/application-package.js:162:44)
    at ApplicationPackage.get [as props] (/Users/akos.kitta/Desktop/theia-nexe/node_modules/@theia/application-package/lib/application-package.js:148:30)
    at ApplicationPackage.get [as target] (/Users/akos.kitta/Desktop/theia-nexe/node_modules/@theia/application-package/lib/application-package.js:138:25)
    at ApplicationPackage.isElectron (/Users/akos.kitta/Desktop/theia-nexe/node_modules/@theia/application-package/lib/application-package.js:336:21)
    at new ApplicationPackage (/Users/akos.kitta/Desktop/theia-nexe/node_modules/@theia/application-package/lib/application-package.js:114:18)
    at Binding.dynamicValue (/Users/akos.kitta/Desktop/theia-nexe/node_modules/@theia/core/lib/node/backend-application-module.js:72:16)
    at /Users/akos.kitta/Desktop/theia-nexe/node_modules/inversify/lib/resolution/resolver.js:63:118
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: '/Users/akos.kitta/Desktop/bin-tmp/package.json' }
root INFO Theia app listening on http://localhost:3000.

After moving the package.json next to the binary, I have the following errors:

Failed to resolve module: @theia/callhierarchy
Failed to resolve module: @theia/core
Failed to resolve module: @theia/editor
Failed to resolve module: @theia/file-search
Failed to resolve module: @theia/filesystem
Failed to resolve module: @theia/git
Failed to resolve module: @theia/json
Failed to resolve module: @theia/keymaps
Failed to resolve module: @theia/languages
Failed to resolve module: @theia/markers
Failed to resolve module: @theia/merge-conflicts
Failed to resolve module: @theia/messages
Failed to resolve module: @theia/mini-browser
Failed to resolve module: @theia/monaco
Failed to resolve module: @theia/navigator
Failed to resolve module: @theia/outline-view
Failed to resolve module: @theia/output
Failed to resolve module: @theia/preferences
Failed to resolve module: @theia/preview
Failed to resolve module: @theia/process
Failed to resolve module: @theia/search-in-workspace
Failed to resolve module: @theia/task
Failed to resolve module: @theia/terminal
Failed to resolve module: @theia/textmate-grammars
Failed to resolve module: @theia/typescript
Failed to resolve module: @theia/userstorage
Failed to resolve module: @theia/workspace
Failed to resolve module: typescript
root INFO Theia app listening on http://localhost:3000.

However, it works when I start the binary from the build location. I hope it is not required to externalize all the modules. Have you tried nbin, @jgbradley1?

[jgbradley1]

Quick update: I gave up on attempting nexe/pkg as a solution. Instead, starting with this guide, I found a simple solution that has allowed me to package theia as a rpm. I can successfully yum install theia and make it executable from the command line like any other yum installed program (i.e. python).

Check out my proof of concept repo for a full end-to-end solution.

After some code review, I think an example of building an rpm should be added to the theia-apps repo for end users to consult. Also we can probably work up a similar example for debian packaging.

[jgbradley1]

Here is a proof of concept that demonstrates end-to-end how to create a Theia debian package.

[jgbradley1]

The theia-apps repo now has a debian packaging example and an rpm packaging example. Is there still a need for an nexe/pkg solution now?

In my opinion, this issue can be closed unless others are still interested.

[dono1986]

I'm still interested in a pkg / nexe solution for windows. The idea is to be able to package theia to a distributable .exe file without having node.js to be installed. That way a domain specific IDE could be bundled with other software

[a1994846931931]

I am trying to package theia as binary with nbin. And some weird errors occured running the binary:

root INFO options.projectPath is/opt
root INFO this.projectPath is/opt/workspace/github/theia-as-binary-nexe-production-2/theia
root INFO Configuring to accept webviews on '.+.webview..+' hostname.
root INFO resolutionPaths_1 is /opt/workspace/github/theia-as-binary-nexe-production-2/theia/package.json
root INFO Deploy plugins list took: 1.7 ms
root INFO Theia app listening on http://0.0.0.0:8888.
root INFO Using Git [2.21.0] from the PATH. (/usr/local/bin/git)
root INFO Checking whether '--no-optional-locks' can be used with the current Git executable. Minimum required version is '2.15.0'.
root INFO '--no-optional-locks' is a valid Git option for the current Git version: '2.21.0'.
root INFO [41ce728d-0449-4f9c-b848-c16ab349d1c2] Sync of 0 plugins took: 33.1 ms
root INFO [41ce728d-0449-4f9c-b848-c16ab349d1c2] Load contributions of 0 plugins took: 0.0 ms
root INFO [41ce728d-0449-4f9c-b848-c16ab349d1c2] Start of 0 plugins took: 93.9 ms
root INFO [nsfw-watcher: 25309] root INFO options.projectPath is/opt
root INFO [nsfw-watcher: 25309] root INFO this.projectPath is/opt/workspace/github/theia-as-binary-nexe-production-2/theia
root INFO [nsfw-watcher: 25309] root INFO Configuring to accept webviews on '.+.webview..+' hostname.
root INFO resolutionPaths_1 is /opt/workspace/github/theia-as-binary-nexe-production-2/theia/package.json
root INFO [nsfw-watcher: 25309] root INFO Deploy plugins list took: 2.5 ms
root ERROR [nsfw-watcher: 25309] Received message which is neither a response nor a notification message:
{
    "port": 3000,
    "address": "127.0.0.1"
}
root INFO [nsfw-watcher: 25309] root INFO Theia app listening on http://localhost:3000.
root INFO [33fe5dd1-6c6b-4c4f-9156-54ed5411b44f] Sync of 0 plugins took: 30.2 ms
root INFO [33fe5dd1-6c6b-4c4f-9156-54ed5411b44f] Load contributions of 0 plugins took: 0.0 ms
root INFO [33fe5dd1-6c6b-4c4f-9156-54ed5411b44f] Start of 0 plugins took: 111.8 ms

Notice how it said that it's listening on http://0.0.0.0:8888 at first, and then said Theia app listening on http://localhost:3000 while I was trying to run it on http://0.0.0.0:8888.

root INFO Deploy plugins list took: 1.9 ms
root ERROR [nsfw-watcher: 25351] root ERROR Failed to start the backend application.
root ERROR [nsfw-watcher: 25351] root ERROR Error: listen EADDRINUSE: address already in use 127.0.0.1:3000
    at Server.setupListenHandle [as _listen2] (net.js:1280:14)
    at listenInCluster (net.js:1328:12)
    at GetAddrInfoReqWrap.doListen [as callback] (net.js:1461:7)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:61:10)
root ERROR [nsfw-watcher: 25351] root ERROR Uncaught Exception:  Error: listen EADDRINUSE: address already in use 127.0.0.1:3000
root ERROR [nsfw-watcher: 25351] root ERROR Error: listen EADDRINUSE: address already in use 127.0.0.1:3000
    at Server.setupListenHandle [as _listen2] (net.js:1280:14)
    at listenInCluster (net.js:1328:12)
    at GetAddrInfoReqWrap.doListen [as callback] (net.js:1461:7)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:61:10)
root ERROR JSON: root ERROR Failed to start the backend application.

root ERROR JSON: root ERROR Error: listen EADDRINUSE: address already in use 127.0.0.1:3000
    at Server.setupListenHandle [as _listen2] (net.js:1280:14)
    at listenInCluster (net.js:1328:12)
    at GetAddrInfoReqWrap.doListen [as callback] (net.js:1461:7)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:61:10)

root ERROR JSON: root ERROR Uncaught Exception:  Error: listen EADDRINUSE: address already in use 127.0.0.1:3000

root ERROR JSON: root ERROR Error: listen EADDRINUSE: address already in use 127.0.0.1:3000
    at Server.setupListenHandle [as _listen2] (net.js:1280:14)
    at listenInCluster (net.js:1328:12)
    at GetAddrInfoReqWrap.doListen [as callback] (net.js:1461:7)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:61:10)

Does anyone have any guesses on what's going here?

[akosyakov]

Does anyone have any guesses on what's going here?

Maybe cli arguments are not applied? but sorry no experience with packaging like that.

[a1994846931931]

Does anyone have any guesses on what's going here?

Maybe cli arguments are not applied?

But it says Theia app listening on http://0.0.0.0:8888 at first, which means it must have parsed the cli argument.

[a1994846931931]

Just figured out why - it's because Theia forks ipc-bootstrap.js.

[dono1986]

@a1994846931931
Can you please share your working configuration for a nbin package?

[jgbradley1]

It would be interesting if someone came up with a way to package Theia as a python package. Making it pip installable would likely improve adoption as well since docker is not available in many enterprise networks due to the requirement to run as root.

The best example of this that I can think of is JupyterLab. That project has figured out how to wrap an npm project into a python package. More instructions here - note the jlpm command is just a locked down version of yarn. Perhaps poking around in their codebase will lead to some insight.

Although Theia is a platform to build IDEs and is not really the final product, having more support for different distribution mechanisms (i.e. the deb and rpm examples linked above) could definitely be improved. Jupyterlab (and the whole jupyter ecosystem) is a great proof of concept that shows the more we streamline the path to distribution, the more widespread adoption we could expect...and therefore more devs contributing to the project overall.

[a1994846931931]

@a1994846931931
Can you please share your working configuration for a nbin package?

@dono1986 I'm still working on it... But what I know so far is that if we want to make binary of theia with nbin, we should probably avoid using fs-extra. I don't know if the change is acceptable to @akosyakov ?

[akosyakov]

@a1994846931931 What's wrong with fs-extra?

[a1994846931931]

@a1994846931931 What's wrong with fs-extra?

Because nbin could only hack (override) the internal fs and if a nbin packaged binary needs to read from its internal filesystem, it must use the overridden fs.

https://github.com/cdr/nbin/blob/42b249e5def2d8d71fb3bfd33ff5a8c59eb54a0c/src/patches/fs.ts#L30

It doesn't mean, however, that we need to get rid of fs-extra. Instead, only those codes that may read files from the internal virtual filesystem (that is the files packaged into the binary) should use fs. For example, if I packaged the built-in extensions into the binary, the plugin loader that relies on fs-extra will fail to get access to these extension files.

[akosyakov]

@a1994846931931 Could you provide a list of such places?

[a1994846931931]

@a1994846931931 Could you provide a list of such places?

@akosyakov That would include:

• packages/debug/src/node/vscode/vscode-debug-adapter-contribution.ts
• packages/plugin-ext/src/hosted/node/plugin-manifest-loader.ts
• packages/plugin-ext/src/hosted/node/plugin-reader.ts

And packages/core/src/node/backend-application.ts as well if one wants to package ssl files into the binary as well.

[akosyakov]

I will add it to the dev meeting? Do you think you can join to explain? One concern that we need a way to ensure that we don't break it with new changes.

[a1994846931931]

I will add it to the dev meeting? Do you think you can join to explain? One concern that we need a way to ensure that we don't break it with new changes.

@akosyakov I'd love to join in but, I may not be able to join in the meeting in office hours due to my company's network setting and I must be careful not to break the "safety red line". It's kind of tricky because all the work is done within the company's computer and I cannot show you how it works at home.

[akosyakov]

@a1994846931931 I have not tried nbin. Does it use webpack under the hood to bundle the server code?

[a1994846931931]

@a1994846931931 I have not tried nbin. Does it use webpack under the hood to bundle the server code?

@akosyakov No, I think it's just appending buffers of the content of all the files one specifies along with the file's meta info (absolute path in the building machine, size, etc.) to a patched node binary:

https://github.com/cdr/nbin/blob/42b249e5def2d8d71fb3bfd33ff5a8c59eb54a0c/src/api/bundler.ts#L19

Then the patched node executable will index all the files or directories within itself, and try to read the content when an fs method is called. The way it tells whether the file should be read from it's inner virtual system is based on the file path:

https://github.com/cdr/nbin/blob/42b249e5def2d8d71fb3bfd33ff5a8c59eb54a0c/src/patches/fs.ts#L47

The pathname will be compared with the recorded path. So yes, the pathname, which depends on where you initially build this executable, will be write into the entry js file before building:

https://github.com/cdr/code-server/blob/80241443810dff1f974ffc1382306c486ff9fb04/scripts/build.ts#L318

But my understanding of these is based on an early version of code-server. I can't find anything related to nbin in the latest version. Is it that they have stopped building executable version based nbin?

[akosyakov]

I think it would be beneficial to webpack the backend first and then use a tool to create binary from bundled content. It should be simple then and webpack allow to reduce the size of binary by obfuscating and minifying code. I think we don't need to worry about fs-extra then.

[nothingismagick]

For tauri-theia, we are currently using pkg - but are a little dissatisfied with the bundle size. IMHO 25MB for a binary bundle is still WAY too big.

Also there are a bunch of warnings when running pkg...
tauri-apps/tauri-theia#2

[paul-marechal]

I managed to get something working for the Theia Blueprint project: #4985 (comment)

[tongji1907]

@nothingismagick I'm also try to build theia with tauri, any suggestions?