Skip to content

base64_encode embedded data files #14526

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 25, 2021
Merged

base64_encode embedded data files #14526

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f787a1e
base64_encode embedded data files
crudelios Jun 23, 2021
1f4597a
Fix CircleCI errors, add myself to AUTHORS
crudelios Jun 23, 2021
e6ba876
Attempt to fix test failures
crudelios Jun 24, 2021
d1fc879
Add myself to AUTHORS (for real this time)
crudelios Jun 25, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AUTHORS
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -572,3 +572,4 @@ a license to everyone to use it as detailed in LICENSE.)
* Camillo Lugaresi <camillol@google.com> (copyright owned by Google LLC)
* Chris Craig <emscripten@goldwave.com>
* Le Yao <le.yao@intel.com> (copyright owned by Intel Corporation)
* José Cadete <crudelios@gmail.com>
2 changes: 1 addition & 1 deletion src/jsifier.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -398,7 +398,7 @@ function JSify(functionsOnly) {
print(preprocess(read('arrayUtils.js')));
}

if (SUPPORT_BASE64_EMBEDDING && !MINIMAL_RUNTIME) {
if ((SUPPORT_BASE64_EMBEDDING || FORCE_FILESYSTEM) && !MINIMAL_RUNTIME) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about replacing this specific change with this:

diff --git a/emcc.py b/emcc.py
index eaf519567..269e5cebb 100755
--- a/emcc.py
+++ b/emcc.py
@@ -1877,6 +1877,10 @@ def phase_linker_setup(options, state, newargs, settings_map):
   else:
     settings.SYSTEM_JS_LIBRARIES.append((0, shared.path_from_root('src', 'library_pthread_stub.js')))
 
+  if settings.FORCE_FILESYSTEM:
+    # enable base64 decoding for file packager data
+    settings.SUPPORT_BASE64_EMBEDDING = 1
+
   if settings.FORCE_FILESYSTEM and not settings.MINIMAL_RUNTIME:
     # when the filesystem is forced, we export by default methods that filesystem usage
     # may need, including filesystem usage from standalone file packager output (i.e.

?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only reason I didn't do it that way is because using SUPPORT_BASE64_EMBEDDING has side effects which I'm not sure are needed in this case. Specifically, enabling SUPPORT_BASE64_EMBEDDING adds some calls in the code to tryParseAsDataURI, which wouldn't be called otherwise.

However, if that's OK, then sure,it's easier to just enable SUPPORT_BASE64_EMBEDDING 👍

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, very good point. Yes, thinking on this more carefully, I agree with you.

print(preprocess(read('base64Utils.js')));
}

Expand Down
21 changes: 9 additions & 12 deletions tools/file_packager.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@
subdir\file, in JS it will be subdir/file. For simplicity we treat the web platform as a *NIX.
"""

import base64
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice that python makes this easy...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah is was rather easy to encode the files. Adding decoding was a little trickier but not by much, it's a small change with a visible impact.

import os
import sys
import shutil
Expand Down Expand Up @@ -95,6 +96,11 @@
new_data_files = []


def base64_encode(b):
b64 = base64.b64encode(b)
return b64.decode('ascii')


def has_hidden_attribute(filepath):
"""Win32 code to test whether the given file has the hidden property set."""

Expand Down Expand Up @@ -464,18 +470,9 @@ def was_seen(name):
basename = os.path.basename(filename)
if file_['mode'] == 'embed':
# Embed
data = list(bytearray(utils.read_binary(file_['srcpath'])))
code += '''var fileData%d = [];\n''' % counter
if data:
parts = []
chunk_size = 10240
start = 0
while start < len(data):
parts.append('''fileData%d.push.apply(fileData%d, %s);\n'''
% (counter, counter, str(data[start:start + chunk_size])))
start += chunk_size
code += ''.join(parts)
code += ('''Module['FS_createDataFile']('%s', '%s', fileData%d, true, true, false);\n'''
data = base64_encode(utils.read_binary(file_['srcpath']))
code += '''var fileData%d = '%s';\n''' % (counter, data)
code += ('''Module['FS_createDataFile']('%s', '%s', decodeBase64(fileData%d), true, true, false);\n'''
% (dirname, basename, counter))
counter += 1
elif file_['mode'] == 'preload':
Expand Down