Skip to content

Use a specific lock file to avoid temp directory contention #13371

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 29, 2021
Merged

Use a specific lock file to avoid temp directory contention #13371

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 0 additions & 9 deletions emcc.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@
"""


import atexit
import json
import logging
import os
Expand Down Expand Up @@ -1927,14 +1926,6 @@ def get_full_import_name(name):
# exit block 'parse arguments and setup'
log_time('parse arguments and setup')

if DEBUG:
# we are about to start using temp dirs. serialize access to the temp dir
# when using EMCC_DEBUG, since we don't want multiple processes would to
# use it at once, they might collide if they happen to use the same
# tempfile names
shared.Cache.acquire_cache_lock()
atexit.register(shared.Cache.release_cache_lock)

if options.post_link:
process_libraries(libs, lib_dirs, temp_files)
if len(input_files) != 1:
Expand Down
15 changes: 15 additions & 0 deletions tools/shared.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
from . import cache, tempfiles, colored_logger
from . import diagnostics
from . import config
from . import filelock


DEBUG = int(os.environ.get('EMCC_DEBUG', '0'))
Expand Down Expand Up @@ -396,6 +397,20 @@ def __init__(self):
except Exception as e:
exit_with_error(str(e) + 'Could not create canonical temp dir. Check definition of TEMP_DIR in ' + config.config_file_location())

# Since the cannonical temp directory is, by definition, the same
# between all processes that run in DEBUG mode we need to use a multi
# process lock to prevent more than one process from writing to it.
# This is because emcc assumes that it can use non-unique names inside
# the temp directory.
# In the case where we run emcc recurively to populate the cache we
# do (sadly) still need to ignore this lock, in the same way we do for
# the cache lock.
if 'EM_EXCLUSIVE_CACHE_ACCESS' not in os.environ:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is a little unfortunate to mention EM_EXCLUSIVE_CACHE_ACCESS here as it would be the first time outside of cache.py. Perhaps we could call a helper function in that file?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a little sad that need this exception here.. but I'm OK with hardcoding it for now. Hopefully in the long run we can completely remove EM_EXCLUSIVE_CACHE_ACCESS.

filelock_name = os.path.join(self.EMSCRIPTEN_TEMP_DIR, 'emscripten.lock')
lock = filelock.FileLock(filelock_name)
lock.acquire()
atexit.register(lock.release)

def get_temp_files(self):
return tempfiles.TempFiles(
tmp=self.TEMP_DIR if not DEBUG else get_emscripten_temp_dir(),
Expand Down