Download the OTA update as individual files

[colenso]

Would help if the OTA updates could be downloaded as individual files rather than one big .zip file. Especially helpful with app having a bigger OTA update size and users downloading the OTA on Mobile Data.

[anbraten]

As the client apps are sending there current version to the server there could also be some system generating diff zips. So in case the user had for example the previous version and now updates to the latest version he could just get a to file large diff containing the two updated files. If another user is just installing the app he would get the full zip. The diffed zips could be cached based on from and to version identifiers.

Or something like a diff -ruaBN previous new > test1.diff

[riderx]

@anbraten since user can be in any current previous version, that would mean produce a diff zip for all previous version, plus Capgo will need to understand what is a diff in your build who seems very complex since project can be react/svelte/angular/vue/js/etc
a believe for now a easier upgrade would be using better the background job feature from iOS and android. to have more than 30 sec in background to finish the work.

[anbraten]

I guess creating the diff would be somehow possible without knowing what the details are, similar to how git, diff work, but sure the server side "calculation" whats the difference between two version would require some computation power even if the result of an upgrade / downgrade between two versions would be cached.

Doing that in a background job seems to be a nice idea in general.

One benefit of sending something like a diff, could be that you save a lot of network bandwidth which would be nice for the client, but also for your server if you really have a lot of clients.

Just wanted to do some brain dump in general as I read this issue 😃. I really like your lib and system.

[riderx]

I totally agree if i found a easy way to do diff i would love to have that, sending duplicated file is not nice

[csotiMatyas]

A comment on this, i was also thinking about sending just some files or folders in the bundle folder. My problem is having an asssets folder being around 1mb, a total waste to send on every download. So i was looking around in the code trying to figure out how the bundle swap happens, so maybe instead of a bundle "replace" i could just make a "copy and replace all" so i could just send what i want, and just override the needed folders/ files. But i can't really find the place where it happens on android.

[riderx]

the files are saved in a new folder and then the capgo define the capacitor web view new default location to it, so for your idea to work, it needs to copy previous app in new folder and then override certain files, the problem is: It's difficult to be sure to have one solution to fit all, since some people in web use unique file name to flush cache

[riderx]

/bounty $500

[0x0elliot]

hey, i would like to help out with this issue. would you mind giving me some context and assigning me to it? :)

[riderx]

Hey @0x0elliot i did assign you !
Currently, the updates are sent from the CLI as a single zip and downloaded by Android and iOS devices as single file.
This is causing issue for big update (more than 10 MB).
So, we have a high level of fail update from 10% to 50% for app used in location where internet is less stable.

What needs to be done:

• Update the CLI to send multiple zip/files instead of one to store.
• Update the backend to understand multi files by update.
• Update the plugin code in android to download multi files updates.
• Update the plugin code in IOS to download multi files updates.

To note:

• There is a current end-to-end encryption system who makes this job a bit harder

The job is pretty big, and I believe it has to be split to be done.
Are you interested in doing any of this task?

[0x0elliot]

i am interested. if you would be comfortable, we can split the bounty into two tasks and 4 PRs:

Part 1

• Update the backend to understand multi files by update.
• Update the CLI to send multiple zip/files instead of one to store.

Part 2

• Update the plugin code in android to download multi files updates.
• Update the plugin code in IOS to download multi files updates.

In chronological order.

I am of course very new to this. Would snoop around in the discord to figure it out properly.

Immediate bottlenecks

To begin with, some questions, which project exactly is the backend?

to help save some time, would you mind helping me figure out which files should i look into? i am thinking of creating another API endpoint (or the equivalent in this case) for this feature as that's the cleaner way to proceed further.

[riderx]

yes, of course, I can adjust the bounty accordingly.
you are right, the backend is in the repo https://github.com/Cap-go/capgo
in the Supabase folder.
I have to create a reproducible env i believe for this task

[0x0elliot]

okay. let me know of the .env file then. why would you require it?

[riderx]

if you want to be able to try end to end, otherwise you can run in local

[0x0elliot]

been trying to set it up locally. it has surely been a bottleneck to set it up. would be less of one if you can manage to create one with some instructions so that i can just focus on the issue.

[0x0elliot]

having quite a difficult time with migrations here :') @riderx
this should be a one time effort help though!

[Tallyb]

Some notes:
Diff should be made on the md5 of the file, as this is a reliable way to know it has changes.
Lost of modern tools use hash when building web output, e.g. webpack, for caching reasons, so file name is likely to be different.

[riderx]

Hey @0x0elliot sorry for the delay it's now way more easy to have local env, i made the seeb db and migration file.
https://github.com/Cap-go/capgo#start-local-supabase-db

[ayewo]

Hey @riderx

Still open? I'm thinking I might want to take a stab at this ...

[riderx]

It's still open yes !

[ayewo]

@riderx Great!

I would like to tenatively adopt the same approach proposed above:

i am interested. if you would be comfortable, we can split the bounty into two tasks and 4 PRs:

Part 1

• Update the backend to understand multi files by update.
• Update the CLI to send multiple zip/files instead of one to store.

Part 2

• Update the plugin code in android to download multi files updates.
• Update the plugin code in IOS to download multi files updates.

Before I start work on Part 1, I'd like to assess if the scope is commensurate with the bounty on offer.
Once I'm done with my initial assessment, I'll ping you here or in Discord so you can officially assign me to the issue.

[ayewo]

... or negotiate the terms before being assigned.

[riderx]

ok no problem mate, just to let you know as the same time someone else contacted me in private and is also doing a proposal on how to do the work

[ayewo]

@riderx Haha, no problem. Thanks for the heads up. (Sounds like that person was subscribed to be notified of any activity on this issue 😀).

Quick question

When I tried to stand up the DB using supabase start as per your README instructions, I get the following error related to migrations:

Applying migration 20230321023005_create_base_db.sql...
ERROR: schema "supabase_migrations" already exists (SQLSTATE 42P06)
At statement 242: --
-- Name: supabase_migrations; Type: SCHEMA; Schema: -; Owner: postgres
--

CREATE SCHEMA supabase_migrations

Seesm like the latest Supabase docker image now ships with the supabase_migrations schema already in place which is why there's a conflict?

[ayewo]

@riderx Nevermind, I figured it out 😎

You can assign me to the issue 🙂.

[AitorG]

@ayewo could be nice, as a first version, to update only those files included in zip and do not delete others in order to save memory and transfer data (it could be setted by config value), just saying :)

[ayewo]

@AitorG yea it would nice, but I haven't been assigned to this issue, yet 😀.

Someone else proposed something similar above to which @riderx replied with the following comment:

the files are saved in a new folder and then the capgo define the capacitor web view new default location to it, so for your idea to work, it needs to copy previous app in new folder and then override certain files, the problem is: It's difficult to be sure to have one solution to fit all, since some people in web use unique file name to flush cache

[riderx]

You have it here: #238

[ayewo]

@riderx sure 👍

/attempt #119

Options

• Cancel my attempt

[triforcely]

Hey @ayewo ,

I was the "incognito" contributor that @riderx mentioned before, and no, I wasn't subscribed to this thread, I have emails with dates to prove, haha 😁

I won't be available for the upcoming couple of weeks, so the issue is fully yours. I thought that I would share some findings/ideas that we've come up with @riderx - it is how I was thinking about approaching the problem with some corrections from @riderx. Of course, you are free to do it your way; it would be a waste of time not to share my findings.

Good luck!

Notes

**New file format for OTA updates**

Some resources about how others handle this issue
· https://docs.expo.dev/technical-specs/expo-updates-1/
· https://docs.expo.dev/eas-update/how-it-works/
· https://docs.expo.dev/eas-update/deployment-patterns/

 
We need to introduce a new (optional) file format based on the bundle manifest to support incremental updates. The regular full-size bundle will be uploaded first when uploading the app update bundle - this is required to ensure that:
· Regular, complete APP updates are continuing to work to support older clients. This ensures that the API changes that we make are not breaking.
· Clients who do not enable incremental package updates are not interrupted.

Bundle manifest file will be generated and uploaded to the Capgo (or generated by any other means when not using Capgo) This manifest is required so the clients can decide which files must be downloaded. 
 
This manifest could look like this:
  
{
  "index.html": {
    "hash": "",
    "url": "<signed url of apps/${apikey.user_id}/${body.app_id}/versions/${body.bucket_id}/incremental/[...]>"
  },
  "js/main-18skch2.chunk.js": {
    "hash": "",
    " url": "<signed url of apps/${apikey.user_id}/${body.app_id}/versions/${body.bucket_id}/incremental/[...]>"
  }
  // and so on
}In case somebody is self hosting, they can provide their own urls in the manifest.
 
Based on the manifest, the client can decide the following:
 
· Which files should be kept (cloned) from the active bundle
· Based on the file name
· Based on checksum (to avoid re-downloading large static resources)
· Which files are new and have to be downloaded
 
The CLI will handle this new bundling behavior out of the box.

https://api.capgo.app/updates endpoint is going to be extended with the „manifest” property but everything else is going to be kept the same to not break existing versions of the plugin.

Extending capacitor-updater

The new capacitor-updater will need to handle new, manifest base bundles gracefully. To support this case following changes should be implemented for Android and iOS:
· When checking if updates are available, it should look at the „manifest” prop first
· When exists: continue "incremental download" logic
· When not exists: continue "full bundle download" logic
· The "apply update" path is updated to handle both scenarios.
· In case of issues with the bundle, e.g. checksums of downloads are different than what the manifest has stated, we should fall back to the full bundle download process.
· There may be transport issues or a packaging mistake from the plugin users' side.

const version = await CapacitorUpdater.download({
  manifest: {
    // full manifest object here
  }
})
await CapacitorUpdater.set(version);

Extending CLI

CLI should optional support incremental updates. 

To do:
· Update the regular package bundle to support additional flags to push incremental update files.
· Add new command to transform the raw web bundle to a structure compatible with capacitor-updater incremental updates, generate a file manifest and upload the manifest with separately encrypted parts of the bundle

Extending Capgo

Capgo backend should be updated in a way that is compatible with existing clients.

To do:
· Implement changes required to support new requirements of CLI and capacitor-updater

[WcaleNieWolny]

Hey, is this still up? I have some experience in setting up S3 with the CLI so I belive I can start by handling sending updates by the CLI and then move onto the updater plugin. I have some ideas how to make this work and I would love to work on this. Here are some of my ideas:

I believe setting up manifest as JSON is a bad idea. If we would to do the manifest something like this:
app_versions

id	manifest
10	{"index.html": { "hash": "", "url": "" }, "js/main.js": { "hash": "", " url": "" } }

then diffing 2 versions would not be possible. How do you figure out what files have already been uploaded?

I believe a better solution would be to setup our mantifest like this:

app_versions

id	manifest
10	array of ids from manifest_entries

manifest_entries

id	file_name	checksum	url	size (?)	storage
10	main.js	something	URL	129	r2 (see note)

about size I am not sure, but I do think it is going to be needed to generate the metadata for the version
and for storage, we might not need this column as I do not believe

Then when we attempt to upload new versions we would create a tree of files to upload like this:

main.js:
    checksum: something
dir/haha.txt:
    checksum: different
nested/deep/file/body.html:
    checksum: checksum

(This will be transformed into json)

and we send this to some edge function (either an existing one or a modified one)
and the backed would attempt to construct the manifest by running the following steps:

• Iterate over all of the files send by the CLI
• check if checksum of said file exists in manifest_entries
• if it does create an entry like this:

main.js:
   needUpload: false
   url: null

and it would add id of said manifest_entries into the manifest of the version

• else if it does not:

main.js
   needUpload: true
   url: upload_url

then the server would create a new manifest_entry and insert it into the manifest

• all of the entries are send to the client (CLI)
• then the client uploads all the missing files into S3
• after an upload to S3 we can have an edge fn run to set the size of a file, however we can also just trust the CLI to send both size and checksum

This however creates some problems:

Deleting version would not be easy. If the have version A and version B where version B depends on files from version A we need to make sure we do not delete these files from S3
We would achieve this by running an SQL query (as SQl is a very powerful tool) that selects all manifest_entries from version A that are used ONLY in version A
After this we would either delete them from S3 and send another query to remove them from supabase or if possible we would select and remove them in the first query, essentially running 1 query to delete data from S3

For old clients (IOS/Android plugin) without progresive deploy we still need the zip file so we still likely need to upload it into S3 which sucks as it creates duplicates of the data

My solution is definitely not perfect, but nevertheless I would love to attempt this.
Please let me know if someone is working on this.

[ayewo]

@WcaleNieWolny yes, I’m still working on this.

Your proposed solution is a bit too complicated though.

For instance,

For old clients (IOS/Android plugin) without progresive deploy we still need the zip file so we still likely need to upload it into S3 which sucks as it creates duplicates of the data

I’m proposing to solve this in a simple way: since this feature will likely require a new flag in capacitor.config.js which we could call ”enablePartialUpdates” = true, if this config is set, the plugin could use a newer version of the updateURL for updates.

So, existing apps will continue to work with the current updateURL of https://api.capgo.app/updates¹ but new apps that enable the feature will use v2 of that URL i.e. https://api.capgo.app/v2/updates

Footnotes

1. Typing on my phone so my recollection of the URL may not be accurate. ↩

[WcaleNieWolny]

That does not change the fact that we still need to keep zip file and individual files. I understand that this is going to have to be enabled in the config. However how do we handle the migration to this feature? If there are already versions (of the updater plugin) deployed without this partial upadate and versions with them we need to have both files. Then when everyone migrates to the version in the app store that has this new feature you will be able to disable the old zip files

[WcaleNieWolny]

Imagine a situation like this:

• This features go live
• Someone enables it in an existing codebase
• He/she uploads the new version to the app store/google play
• He/she deploys a small change (for example a css change)
• People that have still not migrated to the latest store version (ones without individual file download support) have no way to get the latest css change unless a zip file is provided. The old endpoint would essentially have to lie and tell that there are no updates available

[WcaleNieWolny]

I would be happy to work together on this. Let me know if that is okay with you

[ayewo]

Imagine a situation like this:

• This features go live
• Someone enables it in an existing codebase
• He/she uploads the new version to the app store/google play
• He/she deploys a small change (for example a css change)
• People that have still not migrated to the latest store version (ones without individual file download support) have no way to get the latest css change unless a zip file is provided. The old endpoint would essentially have to lie and tell that there are no updates available

That’s easy, the CapUpdater plugin will continue to work as it currenty does today: it will download the full zip file from the updateURL. (Note I
never said anything about retiring the current URL).

Newer apps OTOH can enjoy bandwidth savings on cellular by only downloading a partial update. Large updates tend to fail on cellular so bandwidth savings is what motiving this feature.

As I said, no need to overthink this when a simpler solution exists.

[WcaleNieWolny]

Okay, but are you satisfied with the rest of the proposal? How do you intend to implement this?

[riderx]

@ayewo i would prefer to have the same endpoint and just send in the body that you allow manifest in the request:
likemanifest: true the the backend answer with manifest.

[ayewo]

@riderx yea, sure. I thought about it after I posted that.

[ayewo]

I would be happy to work together on this. Let me know if that is okay with you

@WcaleNieWolny I'm backed up a little bit so wondering: are you still game 😉?

If yes, I'll share a spec for the 3rd pull request needed to wrap up this feature.

[riderx]

/tip 100 @ayewo

[algora-pbc]

🎉🎈 @ayewo has been awarded $100! 🎈🎊

• Share on socials
• Give feedback

[WcaleNieWolny]

I will be attempting this

Cap-go/capgo#601
Cap-go/CLI#236

/attempt #119

Algora profile	Completed bounties	Tech	Active attempts	Options
@WcaleNieWolny	62 Capgo bounties + 2 bounties from 1 project	TypeScript, Java, Rust & more	#266	Cancel attempt

[riderx]

It's finally done you can try it with version 6.3.0 and Capgo cloud, use the bundle upload with --manifest option