Think about how to integrate the electron-builder delta update mechansim

[probonopd]

Looks like the electron-builder delta update mechansim is downloading binary deltas automatically in the background, and replaces the running application with the new version:

$ ls Downloads/
Bitwarden-0.0.15-x86_64.AppImage

$ Downloads/Bitwarden-0.0.15-x86_64.AppImage 
6211
[16:34:04.315] [info] Checking for update
[16:34:04.501] [info] Generated new staging user ID: 64f28813-9306-500e-a48d-9876811ca83b
[16:34:06.360] [info] Found version 0.0.16 (url: Bitwarden-0.0.16-x86_64.AppImage)
[16:34:06.362] [info] Downloading update from Bitwarden-0.0.16-x86_64.AppImage
[16:34:07.020] [info] File has 158 changed blocks
[16:34:07.020] [debug] [
  {
    "kind": 0,
    "start": 0,
    "end": 82292
  },
  {
    "kind": 1,
    "start": 82292,
    "end": 115060
  },
  {
    "kind": 0,
    "start": 115060,
    "end": 50854106
  },
  {
    "kind": 1,
    "start": 50854106,
    "end": 54138640
  },
  {
    "kind": 0,
    "start": 54138912,
    "end": 54749978
  },
  {
    "kind": 1,
    "start": 54749706,
    "end": 54793666
  },
  {
    "kind": 0,
    "start": 54793939,
    "end": 54905680
  },
  {
    "kind": 1,
    "start": 54905407,
    "end": 54924984
  }
]
[16:34:07.023] [info] Full: 53,694.4 KB, To download: 3,301.6 KB (6%)
[16:34:07.029] [info] Differential download: https://github.com/bitwarden/desktop/releases/download/v0.0.16/Bitwarden-0.0.16-x86_64.AppImage
[16:34:07.036] [debug] effective url: , range: bytes=82292-115059
[16:34:07.204] [info] Redirect to https://github-production-release-asset-2e65be.s3.amazonaws.com/76930145/b07d73c4-1a13-11e8-8d82-58a2153a36f7
[16:34:07.917] [debug] effective url: https://github-production-release-asset-2e65be.s3.amazonaws.com/76930145/b07d73c4-1a13-11e8-8d82-58a2153a36f7, range: bytes=50854106-54138639
[16:34:09.634] [debug] effective url: https://github-production-release-asset-2e65be.s3.amazonaws.com/76930145/b07d73c4-1a13-11e8-8d82-58a2153a36f7, range: bytes=54749706-54793665
[16:34:09.868] [debug] effective url: https://github-production-release-asset-2e65be.s3.amazonaws.com/76930145/b07d73c4-1a13-11e8-8d82-58a2153a36f7, range: bytes=54905407-54924983
[16:34:10.016] [info] New version 0.0.16 has been downloaded to /tmp/up-y1XSAk/Bitwarden-0.0.16-x86_64.AppImage
[16:34:25.136] [info] Auto install update on quit
[16:34:25.137] [info] Install: isSilent: true, isRunAfter: false

$ ls Downloads/
Bitwarden-0.0.16-x86_64.AppImage 

Questions:

• Can the automatic updates be prevented, e.g., when running on a machine with limited/expensive network connectivity? (In my opinion, it is bad taste to blindly use the user's Internet connection without the user initiating anything)
• What mechanism is used?
• Is there an implementation for C/C++ available?
• Is this more/less efficient than zsync?
• Should we integrate this mechanism into AppImageUpdate, so that such application can also be updated through e.g., software centers?
• ...

[TheAssassin]

There's a couple of questions to solve before discussing about any kind of integration.

First of all: why? I don't really see a reason to implement this in AppImageUpdate. They built their homebrew update system, and didn't seem to be interested in supporting AppImage specific features such as the update information. That's totally okay, don't get me wrong, but they have their own ecosystem, and "integrating" or even (re-)"implementing" would only help app store devs to update such AppImages. I am strongly opposed to implementing any special treatment for these AppImages, as they add a lot of complexity to the method that reads the update information, with little use for us, but a lot of maintenance overhead.

What mechanism is used?

You remember that ticket? It was something based on the NSIS mechanism.

Is there an implementation for C/C++ available?

I would only consider integrating it with subprocesses at this point. Maintaining any other kind of implementation is too much work. Coding something based on NSIS's codebase or anything like that would be redundant. We should try to use the upstream code. Then, we don't have to react on changes in their code base, we just need to update some upstream tool or something alike.

Is this more/less efficient than zsync?

According to the Electron dev who's implemented the cross-platform update mechanisms, they work similar to zsync, so I'd say they're more or like equally efficient.

Should we integrate this mechanism into AppImageUpdate, so that such application can also be updated through e.g., software centers?

That should be the first question. The current code base allows us to integrate other update systems. The question is more, what tools do Electron offer that could be used to implement these things?

@probonopd first of all, you need to find a (CLI) tool that can be used to perform an update on a given AppImage, or, if the former isn't possible, something like a parameter or environment varaible that needs to be passed to such an AppImage to cause an immediate update (although an external tool would always be less error prone).
If there is such a tool, you need to talk to the Electron devs to embed some kind of update information (can be a static string, e.g., electron), so that libappimageupdate knows it needs to delegat e the update to a subprocess.
Then, we can continue to implement support in libappimage.

Perhaps you should highlight the developer of the Electron update system here, so we can discuss the issue with them.

[probonopd]

Maybe I should have put the user story first: "As the user, I want to be able to update all of the AppImages on my system using a unified tool."

Not saying this is a super high priority for now, but we should keep it on the radar screen.

cc @develar

[develar-vacation]

1. All features of electron-updater are user-driven. I am not aware of AppImage daemon requests or user stories. Maybe because I am not fully involved into Linux world, but I guess the real reason is that what Electron developers wants — do not think about distribution. Packed into distributable format app should just works.
2. So, it is a reason why AppImage daemon not supported. Technically — block map generation implemented in Go. Downloading in Typescript. As stated in that ticket, our implementation roughly equals to ZSync. Just because no magic and there is no possibility to invent something more efficient than CDC (and still so easy maintainable).
3. Small technical difference — blockmap embedded into AppImage, there is no separate zsync file.
4. Electron-updater doesn’t dictate UI or behavior. Update can be blind (and we promote this style) or explicit).

I don’t think that AppImage daemon should support electron updater format in general. For me is strange and comical — instead, electron updater should support ecosystem, not ecosystem :) But there is one big and stopper tecnical issue — our block map generator (see repo develar/app-builder) implemented in go and we can build app-builder executable very easy on any OS for any OS and arch. But zsync only on Linux.

Even this issue is not reported by user, but by you, author of AppImage.

[develar-vacation]

Since appimage daemon is not preinstalled, we were forced to did the same as on Windows — use our own updater. We cannot preinstall AppImage daemon because it will lead to increased maintenance costs and end users questions (like what? Why app brings some another daemon to my system).

So that’s why currently electron-updater doesn’t support AppImage daemon.

[TheAssassin]

@develar-vacation I think you got this wrong. This repository doesn't provide any kind of daemon. And there is no "AppImage daemon" -- there's only appimaged, which isn't responsible for any kind of updates, but only desktop integration (which works already with Electron AppImages).

Furthermore, desktop integration could've been performed completely independent of appimaged.

But, anyway, we are talking about giving people the ability to update AppImages built with Electron-builder.

As stated in my last comment, I wouldn't try to build your JS-based code at all. Instead, I'd rely on whatever tools you provide, given there's such tools. What we would have to do to support this, though, is to make Electron based apps embed a so-called update information. Since this is about using an external tool with a custom format, a static string would suffice.
To do so, all you need to do is call appimagetool with -u electron-updater or something like that. All you need to do is add a single parameter to that call. I guess that's doable for you.

[probonopd]

Thank you @develar-vacation. I understand what you mean when you are saying "Packed into distributable format app should just works". And I completely share that vision.

"Linux people" in general want to "manage the system" all at once. The question is whether we want to (and can) support that in addition. So, use the mechanism inside the AppImage (that you have now) but in addition allow an optional external updater to keep all the AppImages in the entire system updated.

But now: Happy vacation!

[develar-vacation]

@TheAssassin under “AppImage daemon” I mean some central solution, I don’t know exactly how do you call it.

keep all the AppImages in the entire system updated.

Interesting, I am aware of “Update” context menu action for a single app, but I don’t know about centralized solution.

Electron builder uses old AppImage desktop integration script and all installed apps has vendor (exactly as it was in the provided script) prefix, so, it is easy to find all apps on a user machine.
I hate Squirrel.Windows way because I am tired to answer “you must handle Squirrel events or updater will not work”. But in this case we can delegate update call directly to electron-updater without explicit modification of app code (not yet fully clear for me how, but I think it is possible).

So, we need to introduce some CLI hidden API, to call AppImage file with some arg to auto update, right?

Electron updater is a more feature rich (set of supported providers, e.g. GitHub and so on), there are known user custom private providers or customly configured SSL certs. That’s why I doubt that electron builder produced AppImage file can be updated externally (although I am agree that external updater will be better, it will not work for all and it will lead to maintaince hell).

Happy vacation!

Thanks :)

[TheAssassin]

keep all the AppImages in the entire system updated.

This is an idea about creating a daemon using libappimageupdate, searching for updates for all AppImages on your system. It's nothing real, yet.

AppImageUpdate is a library and standalone application to update single AppImages. It would be nice if it was capable of updating any AppImage it is used for, therefore @probonopd suggested to add support for Electron's update mechanism. This doesn't mean we want to implement your mechanism, but we'd like to be able to trigger an update at any time.

libappimageupdate is also supported by a couple of app store apps. But you can't update anything Node-based right now, since those aren't built with support for the existing zsync-based mechanisms. Therefore I suggested to add support for external tools (like the CLI tool you mentioned), by adding a new type of update information.

Electron updater is a more feature rich (set of supported providers, e.g. GitHub and so on), there are known user custom private providers or customly configured SSL certs.

Well, our update mechanism has a couple of features, too, e.g., there's a GitHub and Bintray provider. See https://github.com/AppImage/AppImageSpec/blob/master/draft.md#update-information.

That’s why I doubt that electron builder produced AppImage file can be updated externally (although I am agree that external updater will be better, it will not work for all and it will lead to maintaince hell).

How is the "Electron update information" embedded at the moment? Can it be read by reading a file inside the AppImage? If yes, it doesn't seem to be too problematic to build such an update mechanism. You could mount the existing AppImage (or we implement some kind of "--appimage-extract-file" parameter in the runtime), then read that file, and perform the update. The updater itself might even be contained in there.

When you're back from your vacation, feel free to join #AppImage on Freenode, and we'll work something out.

Enjoy the rest of your vacation!

[develar-vacation]

How is the "Electron update information" embedded at the moment?

See electron-userland/electron-builder#1138 (comment) (resources/app-update.yml)

Blockmap https://github.com/develar/app-builder/blob/master/pkg/blockmap/blockmap.go#L107
Last 4 bytes it is length of blockmap deflate compressed json data. https://github.com/electron-userland/electron-builder/blob/master/packages/electron-updater/src/differentialDownloader/FileWithEmbeddedBlockMapDifferentialDownloader.ts#L10

[TheAssassin]

So, in order to develop a "standalone" updater, you need to read that file, and run those routines. Seems doable.

@probonopd I'll consider my suggestion with the external tool seems like the way to go. We'll have to wait for the Electron team to release such a tool (as long as they're interested in better app store support through our libraries).

@develar-vacation where in the Electron-build code is appimagetool called? I'll send a PR with the missing parameter then.

We'll continue here as soon as the Electron tools are ready.