Skip to content

Multithreading 1/N: JS scripts via Blob URLs #5526

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Sep 11, 2017
Merged

Multithreading 1/N: JS scripts via Blob URLs #5526

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d758422
Add minimal C example for a pthread enabled application.
juj Aug 16, 2017
3be2e96
Implement support and add a test that pthreads can be used from main …
juj Aug 16, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion src/library_pthread.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -321,7 +321,12 @@ var LibraryPThread = {
// Ask the new worker to load up the Emscripten-compiled page. This is a heavy operation.
worker.postMessage({
cmd: 'load',
url: currentScriptUrl,
// If the application main .js file was loaded from a Blob, then it is not possible
// to access the URL of the current script that could be passed to a Web Worker so that
// it could load up the same file. In that case, developer must either deliver the Blob
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we always create a blob here and send that over?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically we could, but doing that has disadvantages, the URL changes from human readable one to a blob UUID one, which affects CSP and CORS behavior, profiling, printed callstacks and logged messages, and we'd have to add a new extra code to do the XHRs in. It would be preferred to allow users to keep loading the .js files when they want to do that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, lgtm.

// object in Module['mainScriptUrlOrBlob'], or a URL to it, so that pthread Workers can
// independently load up the same main application file.
urlOrBlob: Module['mainScriptUrlOrBlob'] || currentScriptUrl,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be a full URL, or a partial URL relative to Module['scriptDirectory']? (#5484)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Module['mainScriptUrlOrBlob'] can be a relative URL, in which case it will be treated with respect to Module['scriptDirectory'], which, if relative, will be treated relative to https://developer.mozilla.org/en-US/docs/Web/HTML/Element/base.

buffer: HEAPU8.buffer,
tempDoublePtr: tempDoublePtr,
TOTAL_MEMORY: TOTAL_MEMORY,
Expand Down
8 changes: 7 additions & 1 deletion src/pthread-main.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,13 @@ this.onmessage = function(e) {
DYNAMICTOP_PTR = e.data.DYNAMICTOP_PTR;

PthreadWorkerInit = e.data.PthreadWorkerInit;
importScripts(e.data.url);
if (typeof e.data.urlOrBlob === 'string') {
importScripts(e.data.urlOrBlob);
} else {
var objectUrl = URL.createObjectURL(e.data.urlOrBlob);
importScripts(objectUrl);
URL.revokeObjectURL(objectUrl);
}
if (typeof FS !== 'undefined') FS.createStandardStreams();
postMessage({ cmd: 'loaded' });
} else if (e.data.cmd === 'objectTransfer') {
Expand Down
18 changes: 18 additions & 0 deletions tests/pthread/hello_thread.c
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
#include <pthread.h>
#include <emscripten.h>

void *thread_main(void *arg)
{
EM_ASM(Module.print('hello from thread!'));
#ifdef REPORT_RESULT
REPORT_RESULT(1);
#endif
return 0;
}

int main()
{
pthread_t thread;
pthread_create(&thread, NULL, thread_main, NULL);
EM_ASM(Module['noExitRuntime']=true);
}
182 changes: 182 additions & 0 deletions tests/pthread/main_js_as_blob_loader.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,182 @@
<!doctype html>
<html lang="en-us">
<head>
<meta charset="utf-8">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Emscripten-Generated Code</title>
<style>
.emscripten { padding-right: 0; margin-left: auto; margin-right: auto; display: block; }
textarea.emscripten { font-family: monospace; width: 80%; }
div.emscripten { text-align: center; }
div.emscripten_border { border: 1px solid black; }
/* the canvas *must not* have any border or padding, or mouse coords will be wrong */
canvas.emscripten { border: 0px none; background-color: black; }

.spinner {
height: 50px;
width: 50px;
margin: 0px auto;
-webkit-animation: rotation .8s linear infinite;
-moz-animation: rotation .8s linear infinite;
-o-animation: rotation .8s linear infinite;
animation: rotation 0.8s linear infinite;
border-left: 10px solid rgb(0,150,240);
border-right: 10px solid rgb(0,150,240);
border-bottom: 10px solid rgb(0,150,240);
border-top: 10px solid rgb(100,0,200);
border-radius: 100%;
background-color: rgb(200,100,250);
}
@-webkit-keyframes rotation {
from {-webkit-transform: rotate(0deg);}
to {-webkit-transform: rotate(360deg);}
}
@-moz-keyframes rotation {
from {-moz-transform: rotate(0deg);}
to {-moz-transform: rotate(360deg);}
}
@-o-keyframes rotation {
from {-o-transform: rotate(0deg);}
to {-o-transform: rotate(360deg);}
}
@keyframes rotation {
from {transform: rotate(0deg);}
to {transform: rotate(360deg);}
}

</style>
</head>
<body>
<hr/>
<figure style="overflow:visible;" id="spinner"><div class="spinner"></div><center style="margin-top:0.5em"><strong>emscripten</strong></center></figure>
<div class="emscripten" id="status">Downloading...</div>
<div class="emscripten">
<progress value="0" max="100" id="progress" hidden=1></progress>
</div>
<div class="emscripten_border">
<canvas class="emscripten" id="canvas" oncontextmenu="event.preventDefault()"></canvas>
</div>
<hr/>
<div class="emscripten">
<input type="checkbox" id="resize">Resize canvas
<input type="checkbox" id="pointerLock" checked>Lock/hide mouse pointer
&nbsp;&nbsp;&nbsp;
<input type="button" value="Fullscreen" onclick="Module.requestFullscreen(document.getElementById('pointerLock').checked,
document.getElementById('resize').checked)">
</div>

<hr/>
<textarea class="emscripten" id="output" rows="8"></textarea>
<hr>
<script type='text/javascript'>
var statusElement = document.getElementById('status');
var progressElement = document.getElementById('progress');
var spinnerElement = document.getElementById('spinner');

var Module = {
preRun: [],
postRun: [],
print: (function() {
var element = document.getElementById('output');
if (element) element.value = ''; // clear browser cache
return function(text) {
if (arguments.length > 1) text = Array.prototype.slice.call(arguments).join(' ');
// These replacements are necessary if you render to raw HTML
//text = text.replace(/&/g, "&amp;");
//text = text.replace(/</g, "&lt;");
//text = text.replace(/>/g, "&gt;");
//text = text.replace('\n', '<br>', 'g');
console.log(text);
if (element) {
element.value += text + "\n";
element.scrollTop = element.scrollHeight; // focus on bottom
}
};
})(),
printErr: function(text) {
if (arguments.length > 1) text = Array.prototype.slice.call(arguments).join(' ');
if (0) { // XXX disabled for safety typeof dump == 'function') {
dump(text + '\n'); // fast, straight to the real console
} else {
console.error(text);
}
},
canvas: (function() {
var canvas = document.getElementById('canvas');

// As a default initial behavior, pop up an alert when webgl context is lost. To make your
// application robust, you may want to override this behavior before shipping!
// See http://www.khronos.org/registry/webgl/specs/latest/1.0/#5.15.2
canvas.addEventListener("webglcontextlost", function(e) { alert('WebGL context lost. You will need to reload the page.'); e.preventDefault(); }, false);

return canvas;
})(),
setStatus: function(text) {
if (!Module.setStatus.last) Module.setStatus.last = { time: Date.now(), text: '' };
if (text === Module.setStatus.text) return;
var m = text.match(/([^(]+)\((\d+(\.\d+)?)\/(\d+)\)/);
var now = Date.now();
if (m && now - Date.now() < 30) return; // if this is a progress update, skip it if too soon
if (m) {
text = m[1];
progressElement.value = parseInt(m[2])*100;
progressElement.max = parseInt(m[4])*100;
progressElement.hidden = false;
spinnerElement.hidden = false;
} else {
progressElement.value = null;
progressElement.max = null;
progressElement.hidden = true;
if (!text) spinnerElement.hidden = true;
}
statusElement.innerHTML = text;
},
totalDependencies: 0,
monitorRunDependencies: function(left) {
this.totalDependencies = Math.max(this.totalDependencies, left);
Module.setStatus(left ? 'Preparing... (' + (this.totalDependencies-left) + '/' + this.totalDependencies + ')' : 'All downloads complete.');
}
};
Module.setStatus('Downloading...');
window.onerror = function() {
Module.setStatus('Exception thrown, see JavaScript console');
spinnerElement.style.display = 'none';
Module.setStatus = function(text) {
if (text) Module.printErr('[post-exception status] ' + text);
};
};
</script>

<script>

function download(url, responseType) {
return new Promise(function(resolve, reject) {
var xhr = new XMLHttpRequest();
xhr.open('GET', url, true);
xhr.responseType = 'blob';
xhr.onload = function() {
resolve(xhr.response);
};
xhr.send(null);
});
}

function addToDom(scriptCodeBlob) {
return new Promise(function(resolve, reject) {
var script = document.createElement('script');
var objectUrl = URL.createObjectURL(scriptCodeBlob);
script.src = objectUrl;
script.onload = function() {
Module['mainScriptUrlOrBlob'] = scriptCodeBlob;
URL.revokeObjectURL(objectUrl);
resolve();
}
document.body.appendChild(script);
});
}

download('hello_thread_with_blob_url.js').then(addToDom);

</script>
</body>
</html>
15 changes: 14 additions & 1 deletion tests/test_browser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3623,4 +3623,17 @@ def test_pthread_locale(self):

# Tests the Emscripten HTML5 API emscripten_set_canvas_element_size() and emscripten_get_canvas_element_size() functionality in singlethreaded programs.
def test_emscripten_set_canvas_element_size(self):
self.btest('emscripten_set_canvas_element_size.c', expected='1')
self.btest('emscripten_set_canvas_element_size.c', expected='1')

# Tests the absolute minimum pthread-enabled application.
def test_hello_thread(self):
self.btest(path_from_root('tests', 'pthread', 'hello_thread.c'), expected='1', args=['-s', 'USE_PTHREADS=1'])

# Tests that it is possible to load the main .js file of the application manually via a Blob URL, and still use pthreads.
def test_load_js_from_blob_with_pthreads(self):
src = os.path.join(self.get_dir(), 'src.c')
open(src, 'w').write(self.with_report_result(open(path_from_root('tests', 'pthread', 'hello_thread.c')).read()))

Popen([PYTHON, EMCC, 'src.c', '-s', 'USE_PTHREADS=1', '-o', 'hello_thread_with_blob_url.js']).communicate()
shutil.copyfile(path_from_root('tests', 'pthread', 'main_js_as_blob_loader.html'), os.path.join(self.get_dir(), 'hello_thread_with_blob_url.html'))
self.run_browser('hello_thread_with_blob_url.html', 'hello from thread!', '/report_result?1')