Skip to content

Commit

Permalink
src: use STL containers instead of v8 values for static module data
Browse files Browse the repository at this point in the history
Instead of putting the source code and the cache in v8::Objects,
put them in per-process std::maps. This has the following benefits:

- It's slightly lighter in weight compared to storing things on the
  v8 heap. Also it may be slightly faster since the preivous v8::Object
  is already in dictionary mode - though the difference is very small
  given the number of native modules is limited.
- The source and code cache generation templates are now much simpler
  since they just initialize static arrays and manipulate STL
  constructs.
- The static native module data can be accessed independently of any
  Environment or Isolate, and it's easy to look them up from the
  C++'s side.
- It's now impossible to mutate the source code used to compile
  native modules from the JS land since it's completely separate
  from the v8 heap. We can still get the constant strings from
  process.binding('natives') but that's all.

A few drive-by fixes:

- Remove DecorateErrorStack in LookupAndCompile - We don't need to
  capture the exception to decorate when we encounter
  errors during native module compilation, as those errors should be
  syntax errors and v8 is able to decorate them well. We use
  CompileFunctionInContext so there is no need to worry about
  wrappers either.
- The code cache could be rejected when node is started with v8 flags.
  Instead of aborting in that case, simply keep a record in the
  native_module_without_cache set.
- Refactor js2c.py a bit, reduce code duplication and inline Render()
  to make the one-byte/two-byte special treatment easier to read.

PR-URL: #24384
Fixes: https://github.com/Remove
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Franziska Hinkelmann <franziska.hinkelmann@gmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
  • Loading branch information
joyeecheung authored and rvagg committed Nov 28, 2018
1 parent 0a08cd7 commit 21e9aa2
Show file tree
Hide file tree
Showing 16 changed files with 457 additions and 419 deletions.
3 changes: 2 additions & 1 deletion lib/internal/bootstrap/cache.js
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,12 @@ const {
NativeModule
} = require('internal/bootstrap/loaders');
const {
source,
getSource,
compileCodeCache
} = internalBinding('native_module');
const { hasTracing } = process.binding('config');

const source = getSource();
const depsModule = Object.keys(source).filter(
(key) => NativeModule.isDepsModule(key) || key.startsWith('internal/deps')
);
Expand Down
3 changes: 1 addition & 2 deletions node.gyp
Original file line number Diff line number Diff line change
Expand Up @@ -410,15 +410,13 @@
'src/node_api.h',
'src/node_api_types.h',
'src/node_buffer.h',
'src/node_code_cache.h',
'src/node_constants.h',
'src/node_contextify.h',
'src/node_errors.h',
'src/node_file.h',
'src/node_http2.h',
'src/node_http2_state.h',
'src/node_internals.h',
'src/node_javascript.h',
'src/node_messaging.h',
'src/node_mutex.h',
'src/node_native_module.h',
Expand All @@ -430,6 +428,7 @@
'src/node_persistent.h',
'src/node_platform.h',
'src/node_root_certs.h',
'src/node_union_bytes.h',
'src/node_version.h',
'src/node_watchdog.h',
'src/node_revert.h',
Expand Down
9 changes: 5 additions & 4 deletions src/env.cc
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
#include "node_internals.h"
#include "async_wrap.h"
#include "v8-profiler.h"
#include "node_buffer.h"
#include "node_platform.h"
#include "node_file.h"
#include "node_context_data.h"
#include "node_file.h"
#include "node_internals.h"
#include "node_native_module.h"
#include "node_platform.h"
#include "node_worker.h"
#include "tracing/agent.h"
#include "tracing/traced_value.h"
#include "v8-profiler.h"

#include <stdio.h>
#include <algorithm>
Expand Down
15 changes: 6 additions & 9 deletions src/env.h
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,13 @@
#include "inspector_agent.h"
#endif
#include "handle_wrap.h"
#include "node.h"
#include "node_http2_state.h"
#include "node_options.h"
#include "req_wrap.h"
#include "util.h"
#include "uv.h"
#include "v8.h"
#include "node.h"
#include "node_options.h"
#include "node_http2_state.h"

#include <list>
#include <stdint.h>
Expand Down Expand Up @@ -347,12 +347,6 @@ constexpr size_t kFsStatsBufferLength = kFsStatsFieldsNumber * 2;
V(libuv_stream_wrap_ctor_template, v8::FunctionTemplate) \
V(message_port, v8::Object) \
V(message_port_constructor_template, v8::FunctionTemplate) \
V(native_modules_code_cache, v8::Object) \
V(native_modules_code_cache_hash, v8::Object) \
V(native_modules_source, v8::Object) \
V(native_modules_source_hash, v8::Object) \
V(native_modules_with_cache, v8::Set) \
V(native_modules_without_cache, v8::Set) \
V(performance_entry_callback, v8::Function) \
V(performance_entry_template, v8::Function) \
V(pipe_constructor_template, v8::FunctionTemplate) \
Expand Down Expand Up @@ -684,6 +678,9 @@ class Environment {
// List of id's that have been destroyed and need the destroy() cb called.
inline std::vector<double>* destroy_async_id_list();

std::set<std::string> native_modules_with_cache;
std::set<std::string> native_modules_without_cache;

std::unordered_multimap<int, loader::ModuleWrap*> hash_to_module_map;
std::unordered_map<uint32_t, loader::ModuleWrap*> id_to_module_map;
std::unordered_map<uint32_t, contextify::ContextifyScript*>
Expand Down
36 changes: 19 additions & 17 deletions src/node.cc
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@
#include "node_context_data.h"
#include "node_errors.h"
#include "node_internals.h"
#include "node_javascript.h"
#include "node_native_module.h"
#include "node_perf.h"
#include "node_platform.h"
Expand Down Expand Up @@ -130,7 +129,7 @@ typedef int mode_t;

namespace node {

using native_module::NativeModule;
using native_module::NativeModuleLoader;
using options_parser::kAllowedInEnvironment;
using options_parser::kDisallowedInEnvironment;
using v8::Array;
Expand Down Expand Up @@ -212,7 +211,7 @@ double prog_start_time;
Mutex per_process_opts_mutex;
std::shared_ptr<PerProcessOptions> per_process_opts {
new PerProcessOptions() };

NativeModuleLoader per_process_loader;
static Mutex node_isolate_mutex;
static Isolate* node_isolate;

Expand Down Expand Up @@ -1243,8 +1242,7 @@ static void GetInternalBinding(const FunctionCallbackInfo<Value>& args) {
Null(env->isolate())).FromJust());
DefineConstants(env->isolate(), exports);
} else if (!strcmp(*module_v, "natives")) {
exports = Object::New(env->isolate());
NativeModule::GetNatives(env, exports);
exports = per_process_loader.GetSourceObject(env->context());
} else {
return ThrowIfNoSuchModule(env, *module_v);
}
Expand Down Expand Up @@ -1780,18 +1778,24 @@ void LoadEnvironment(Environment* env) {

// The bootstrapper scripts are lib/internal/bootstrap/loaders.js and
// lib/internal/bootstrap/node.js, each included as a static C string
// defined in node_javascript.h, generated in node_javascript.cc by
// node_js2c.
// generated in node_javascript.cc by node_js2c.

// TODO(joyeecheung): use NativeModule::Compile
// TODO(joyeecheung): use NativeModuleLoader::Compile
// We duplicate the string literals here since once we refactor the bootstrap
// compilation out to NativeModuleLoader none of this is going to matter
Isolate* isolate = env->isolate();
Local<String> loaders_name =
FIXED_ONE_BYTE_STRING(env->isolate(), "internal/bootstrap/loaders.js");
FIXED_ONE_BYTE_STRING(isolate, "internal/bootstrap/loaders.js");
Local<String> loaders_source =
per_process_loader.GetSource(isolate, "internal/bootstrap/loaders");
MaybeLocal<Function> loaders_bootstrapper =
GetBootstrapper(env, LoadersBootstrapperSource(env), loaders_name);
GetBootstrapper(env, loaders_source, loaders_name);
Local<String> node_name =
FIXED_ONE_BYTE_STRING(env->isolate(), "internal/bootstrap/node.js");
FIXED_ONE_BYTE_STRING(isolate, "internal/bootstrap/node.js");
Local<String> node_source =
per_process_loader.GetSource(isolate, "internal/bootstrap/node");
MaybeLocal<Function> node_bootstrapper =
GetBootstrapper(env, NodeBootstrapperSource(env), node_name);
GetBootstrapper(env, node_source, node_name);

if (loaders_bootstrapper.IsEmpty() || node_bootstrapper.IsEmpty()) {
// Execution was interrupted.
Expand Down Expand Up @@ -1843,8 +1847,6 @@ void LoadEnvironment(Environment* env) {
env->options()->debug_options->break_node_first_line)
};

NativeModule::LoadBindings(env);

// Bootstrap internal loaders
Local<Value> bootstrapped_loaders;
if (!ExecuteBootstrapper(env, loaders_bootstrapper.ToLocalChecked(),
Expand Down Expand Up @@ -2485,7 +2487,6 @@ void FreePlatform(MultiIsolatePlatform* platform) {
delete platform;
}


Local<Context> NewContext(Isolate* isolate,
Local<ObjectTemplate> object_template) {
auto context = Context::New(isolate, nullptr, object_template);
Expand All @@ -2499,8 +2500,9 @@ Local<Context> NewContext(Isolate* isolate,
// Run lib/internal/per_context.js
Context::Scope context_scope(context);

// TODO(joyeecheung): use NativeModule::Compile
Local<String> per_context = NodePerContextSource(isolate);
// TODO(joyeecheung): use NativeModuleLoader::Compile
Local<String> per_context =
per_process_loader.GetSource(isolate, "internal/per_context");
ScriptCompiler::Source per_context_src(per_context, nullptr);
Local<Script> s = ScriptCompiler::Compile(
context,
Expand Down
19 changes: 0 additions & 19 deletions src/node_code_cache.h

This file was deleted.

22 changes: 9 additions & 13 deletions src/node_code_cache_stub.cc
Original file line number Diff line number Diff line change
@@ -1,23 +1,19 @@

#include "node_code_cache.h"
#include "node_native_module.h"

// This is supposed to be generated by tools/generate_code_cache.js
// The stub here is used when configure is run without `--code-cache-path`

namespace node {
namespace native_module {

const bool native_module_has_code_cache = false;
// The generated source code would insert <std::string, UnionString> pairs
// into native_module_loader.code_cache_.
void NativeModuleLoader::LoadCodeCache() {}

void DefineCodeCache(Environment* env, v8::Local<v8::Object> target) {
// When we do not produce code cache for builtin modules,
// `internalBinding('code_cache')` returns an empty object
// (here as `target`) so this is a noop.
}

void DefineCodeCacheHash(Environment* env, v8::Local<v8::Object> target) {
// When we do not produce code cache for builtin modules,
// `internalBinding('code_cache_hash')` returns an empty object
// (here as `target`) so this is a noop.
}
// The generated source code would instert <std::string, std::string> pairs
// into native_module_loader.code_cache_hash_.
void NativeModuleLoader::LoadCodeCacheHash() {}

} // namespace native_module
} // namespace node
5 changes: 5 additions & 0 deletions src/node_internals.h
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,10 @@ struct sockaddr;

namespace node {

namespace native_module {
class NativeModuleLoader;
}

extern Mutex process_mutex;
extern Mutex environ_mutex;

Expand All @@ -179,6 +183,7 @@ extern bool v8_initialized;

extern Mutex per_process_opts_mutex;
extern std::shared_ptr<PerProcessOptions> per_process_opts;
extern native_module::NativeModuleLoader per_process_loader;

// Forward declaration
class Environment;
Expand Down
41 changes: 0 additions & 41 deletions src/node_javascript.h

This file was deleted.

Loading

0 comments on commit 21e9aa2

Please sign in to comment.