Porting spidermonkey60 support to fdb-layer

.gitignore

@@ -9,6 +9,8 @@
 .venv
 .DS_Store
 .rebar/
+.rebar3/
+.erlfdb/
 .eunit/
 log
 apache-couchdb-*/
@@ -33,34 +35,37 @@ share/www
 src/b64url/
 src/bear/
 src/config/
-src/couch/priv/couch_js/config.h
+src/couch/priv/couch_js/**/config.h
 src/couch/priv/couchjs
 src/couch/priv/couchspawnkillable
 src/couch/priv/couch_ejson_compare/couch_ejson_compare.d
-src/couch/priv/couch_js/http.d
-src/couch/priv/couch_js/main.d
-src/couch/priv/couch_js/utf8.d
-src/couch/priv/couch_js/util.d
+src/couch/priv/couch_js/**/*.d
 src/couch/priv/icu_driver/couch_icu_driver.d
 src/mango/src/mango_cursor_text.nocompile
 src/docs/
+src/erlfdb/
 src/ets_lru/
 src/fauxton/
 src/folsom/
 src/hqueue/
 src/hyper/
 src/ibrowse/
 src/ioq/
+src/hqueue/
+src/jaeger_passage/
 src/jiffy/
 src/ken/
 src/khash/
+src/local/
 src/meck/
 src/mochiweb/
 src/oauth/
+src/passage/
 src/proper/
 src/rebar/
 src/smoosh/
 src/snappy/
+src/thrift_protocol/
 src/triq/
 tmp/
 

FDB_NOTES.md

@@ -0,0 +1,57 @@
+Things of Note
+===
+
+
+1. If a replication sends us two revisions A and B where one is an
+   ancestor of the other, we likely have divergent behavior. However,
+   this should never happen In Theory.
+
+2. Multiple updates to the same document in a _bulk_docs (or if they
+   just happen to be in the same update batch in non-fdb CouchDB)
+   we likely have subtly different behavior.
+
+3. I'm relying on repeated reads in an fdb transaction to be "cheap"
+   in that the reads would be cached in the fdb_transaction object.
+   This needs to be checked for certainty but that appeared to
+   be how things behaved in testing.
+
+4. When attempting to create a doc from scratch in an interacitve_edit
+   update, with revisions specified *and* attachment stubs, the reported
+   error is now a conflict. Previously the missing_stubs error was
+   raised earlier.
+
+5. There may be a difference in behavior if a) there are no VDU functions
+   set on a db and no design documents in a batch. This is because in
+   this situation we don't run the prep_and_validate code on pre-fdb
+   CouchDB. The new code always checks stubs before merging revision trees.
+   I'm sure the old way would fail somehow, but it would fail further on
+   which means we may have failed with a different reason (conflict, etc)
+   before we got to the next place we check for missing stubs.
+
+6. For multi-doc updates we'll need to investigate user versions on
+   versionstamps within a transaction. Also this likely prevents the
+   ability to have multiple updates to the same doc in a single
+   _bulk_docs transaction
+
+7. Document body storage needs to be implemented beyond the single
+   key/value approach.
+
+8. We'll want to look at how we currently apply open options to individual
+    elements of an open_revs call. Might turn out that we have to grab a
+    full FDI even if we could look up a rev directly. (i.e., revs_info
+    would require us having the entire FDI, however it'd be wasteful to return
+    all of that in an open_revs call, but bug compatibility ftw!)
+
+9. Is it possible that a server_admin can delete a db without being able
+    to open it? If so that's probably changed behavior.
+
+10. All docs on large active databases might be a thing getting the doc
+    count. If we allow range requests up to 5s, and we continue to return
+    the doc count total we may have to play games with snapshot reads on
+    the doc count key or else it'll whack any _all_docs range requests
+
+11. Revision infos need to track their size f we want to maintain a database
+    size counter we'll want to store the size of a given doc body for each
+    revision so that we don't have to read the old body when updating the tree.
+
+12. Update sequences do not yet include an incarnation value.

LICENSE

@@ -850,6 +850,61 @@ FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
 OTHER DEALINGS IN THE FONT SOFTWARE.
 
 
+for share/server/60/esprima.js
+Based on https://github.com/jquery/esprima
+
+BSD License
+
+Copyright JS Foundation and other contributors, https://js.foundation/
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+  * Redistributions of source code must retain the above copyright
+    notice, this list of conditions and the following disclaimer.
+  * Redistributions in binary form must reproduce the above copyright
+    notice, this list of conditions and the following disclaimer in the
+    documentation and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY
+DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+
+share/server/60/escodegen.js
+Based on https://github.com/estools/escodegen
+
+BSD License
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+  * Redistributions of source code must retain the above copyright
+    notice, this list of conditions and the following disclaimer.
+  * Redistributions in binary form must reproduce the above copyright
+    notice, this list of conditions and the following disclaimer in the
+    documentation and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY
+DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
 for src/fauxton/assets/less/bootstrap/font-awesome/*
 for share/www/css/*
 

Makefile

@@ -234,7 +234,11 @@ python-black-update: .venv/bin/black
 .PHONY: elixir
 elixir: export MIX_ENV=integration
 elixir: elixir-init elixir-check-formatted elixir-credo devclean
-	@dev/run -a adm:pass --no-eval 'mix test --trace --exclude without_quorum_test --exclude with_quorum_test $(EXUNIT_OPTS)'
+	@dev/run --erlang-config=rel/files/eunit.config -n 1 -a adm:pass --no-eval 'mix test --trace --exclude without_quorum_test --exclude with_quorum_test $(EXUNIT_OPTS)'
+
+.PHONY: elixir-only
+elixir-only: devclean
+	@dev/run --erlang-config=rel/files/eunit.config -n 1 -a adm:pass --no-eval 'mix test --trace --exclude without_quorum_test --exclude with_quorum_test $(EXUNIT_OPTS)'
 
 .PHONY: elixir-init
 elixir-init:

NOTICE

@@ -178,6 +178,14 @@ This product also includes the following third-party components:
 
   Copyright (c) 2015 Twitter, Inc.
 
+* esprima.js (https://github.com/jquery/esprima)
+
+  Copyright JS Foundation and other contributors, https://js.foundation/
+
+* escodegen.js (https://github.com/estools/escodegen)
+
+  Copyright (C) 2012 Yusuke Suzuki (twitter: @Constellation) and other contributors.
+
 * hyper
 
   Copyright (c) 2014 Game Analytics ApS

configure

@@ -30,6 +30,7 @@ ERLANG_MD5="false"
 SKIP_DEPS=0
 
 COUCHDB_USER="$(whoami 2>/dev/null || echo couchdb)"
+SM_VSN="1.8.5"
 
 . ${rootdir}/version.mk
 COUCHDB_VERSION=${vsn_major}.${vsn_minor}.${vsn_patch}
@@ -50,6 +51,7 @@ Options:
   --disable-docs              do not build any documentation or manpages
   --erlang-md5                use erlang for md5 hash operations
   --dev                       alias for --with-curl --disable-docs --disable-fauxton
+  --spidermonkey-version VSN  specify the version of SpiderMonkey to use (defaults to $SM_VSN)
   --skip-deps                 do not update erlang dependencies
   --rebar=PATH                use rebar by specified path (version >=2.6.0 && <3.0 required)
 EOF
@@ -140,6 +142,24 @@ parse_opts() {
                 printf 'ERROR: "--user" requires a non-empty argument.\n' >&2
                 exit 1
                 ;;
+
+            --spidermonkey-version)
+                if [ -n "$2" ]; then
+                    eval SM_VSN=$2
+                    shift 2
+                    continue
+                else
+                    printf 'ERROR: "--spidermonkey-version" requires a non-empty argument.\n' >&2
+                    exit 1
+                fi
+                ;;
+            --spidermonkey-version=?*)
+                eval SM_VSN=${1#*=}
+                ;;
+            --spidermonkey-version=)
+                printf 'ERROR: "--spidermonkey-version" requires a non-empty argument.\n' >&2
+                exit 1
+                ;;
             --) # End of options
                 shift
                 break
@@ -180,6 +200,7 @@ cat > rel/couchdb.config << EOF
 {log_file, "$LOG_FILE"}.
 {fauxton_root, "./share/www"}.
 {user, "$COUCHDB_USER"}.
+{spidermonkey_version, "$SM_VSN"}.
 {node_name, "-name couchdb@127.0.0.1"}.
 {cluster_port, 5984}.
 {backend_port, 5986}.
@@ -206,12 +227,14 @@ with_fauxton = $WITH_FAUXTON
 with_docs = $WITH_DOCS
 
 user = $COUCHDB_USER
+spidermonkey_version = $SM_VSN
 EOF
 
 cat > $rootdir/config.erl << EOF
 {with_curl, $WITH_CURL}.
 {with_proper, $WITH_PROPER}.
 {erlang_md5, $ERLANG_MD5}.
+{spidermonkey_version, "$SM_VSN"}.
 EOF
 
 install_local_rebar() {

configure.ps1

@@ -10,6 +10,7 @@
   -DisableDocs               request build process skip building documentation (default false)
   -SkipDeps                  do not update Erlang dependencies (default false)
   -CouchDBUser USER          set the username to run as (defaults to current user)
+  -SpiderMonkeyVersion VSN   select the version of SpiderMonkey to use (defaults to 1.8.5)
 
   Installation directories:
   -Prefix PREFIX             install architecture-independent files in PREFIX
@@ -49,6 +50,8 @@ Param(
     [ValidateNotNullOrEmpty()]
     [string]$CouchDBUser = [Environment]::UserName, # set the username to run as (defaults to current user)
     [ValidateNotNullOrEmpty()]
+    [string]$SpiderMonkeyVersion = "1.8.5", # select the version of SpiderMonkey to use (default 1.8.5)
+    [ValidateNotNullOrEmpty()]
     [string]$Prefix = "C:\Program Files\Apache\CouchDB", # install architecture-independent file location (default C:\Program Files\Apache\CouchDB)
     [ValidateNotNullOrEmpty()]
     [string]$ExecPrefix = $Prefix, # install architecture-dependent file location (default C:\Program Files\Apache\CouchDB)
@@ -133,6 +136,7 @@ $CouchDBConfig = @"
 {log_file, ""}.
 {fauxton_root, "./share/www"}.
 {user, "$CouchDBUser"}.
+{spidermonkey_version, "$SpiderMonkeyVersion"}.
 {node_name, "-name couchdb@localhost"}.
 {cluster_port, 5984}.
 {backend_port, 5986}.
@@ -144,7 +148,7 @@ $InstallMk = @"
 # Licensed under the Apache License, Version 2.0 (the "License"); you may not
 # use this file except in compliance with the License. You may obtain a copy of
 # the License at
-# 
+#
 #   http://www.apache.org/licenses/LICENSE-2.0
 #
 # Unless required by applicable law or agreed to in writing, software
@@ -175,12 +179,14 @@ with_fauxton = $BuildFauxton
 with_docs = $BuildDocs
 
 user = $CouchDBUser
+spidermonkey_version = $SpiderMonkeyVersion
 "@
 $InstallMk | Out-File "$rootdir\install.mk" -encoding ascii
 
 $lowercurl = "$WithCurl".ToLower()
 $ConfigERL = @"
 {with_curl, $lowercurl}.
+{spidermonkey_version, "$SpiderMonkeyVersion"}.
 "@
 $ConfigERL | Out-File "$rootdir\config.erl" -encoding ascii
 

dev/run

@@ -110,6 +110,11 @@ def setup_logging(ctx):
 
 
 def setup_argparse():
+    parser = get_args_parser()
+    return parser.parse_args()
+
+
+def get_args_parser():
     parser = optparse.OptionParser(description="Runs CouchDB 2.0 dev cluster")
     parser.add_option(
         "-a",
@@ -180,6 +185,12 @@ def setup_argparse():
         default=[],
         help="Optional key=val config overrides. Can be repeated",
     )
+    parser.add_option(
+        "--erlang-config",
+        dest="erlang_config",
+        default="rel/files/sys.config",
+        help="Specify an alternative Erlang application configuration"
+    )
     parser.add_option(
         "--degrade-cluster",
         dest="degrade_cluster",
@@ -200,7 +211,13 @@ def setup_argparse():
         action="store_true",
         help="Select available ports for nodes automatically",
     )
-    return parser.parse_args()
+    parser.add_option(
+        "--extra_args",
+        dest="extra_args",
+        default=None,
+        help="Extra arguments to pass to beam process",
+    )
+    return parser
 
 
 def setup_context(opts, args):
@@ -222,7 +239,9 @@ def setup_context(opts, args):
         "haproxy": opts.haproxy,
         "haproxy_port": opts.haproxy_port,
         "config_overrides": opts.config_overrides,
+        "erlang_config": opts.erlang_config,
         "no_eval": opts.no_eval,
+        "extra_args": opts.extra_args,
         "reset_logs": True,
         "procs": [],
         "auto_ports": opts.auto_ports,
@@ -559,7 +578,7 @@ def boot_node(ctx, node):
         "-args_file",
         os.path.join(node_etcdir, "vm.args"),
         "-config",
-        os.path.join(reldir, "files", "sys"),
+        os.path.join(ctx["rootdir"], ctx["erlang_config"]),
         "-couch_ini",
         os.path.join(node_etcdir, "default.ini"),
         os.path.join(node_etcdir, "local.ini"),
@@ -578,6 +597,8 @@ def boot_node(ctx, node):
         mode = "r+b"
     logfname = os.path.join(ctx["devdir"], "logs", "%s.log" % node)
     log = open(logfname, mode)
+    if "extra_args" in ctx and ctx["extra_args"]:
+        cmd += ctx["extra_args"].split(" ")
     cmd = [toposixpath(x) for x in cmd]
     return sp.Popen(cmd, stdin=sp.PIPE, stdout=log, stderr=sp.STDOUT, env=env)
 

mix.exs

@@ -35,7 +35,7 @@ defmodule CouchDBTest.Mixfile do
   # Run "mix help deps" to learn about dependencies.
   defp deps() do
     [
-      {:httpotion, "~> 3.0", only: [:dev, :test, :integration], runtime: false},
+      {:httpotion, ">= 3.1.3", only: [:dev, :test, :integration], runtime: false},
       {:jiffy, path: Path.expand("src/jiffy", __DIR__)},
       {:ibrowse,
        path: Path.expand("src/ibrowse", __DIR__), override: true, compile: false},

mix.lock

@@ -1,7 +1,7 @@
 %{
   "bunt": {:hex, :bunt, "0.2.0", "951c6e801e8b1d2cbe58ebbd3e616a869061ddadcc4863d0a2182541acae9a38", [:mix], [], "hexpm"},
   "credo": {:hex, :credo, "1.0.5", "fdea745579f8845315fe6a3b43e2f9f8866839cfbc8562bb72778e9fdaa94214", [:mix], [{:bunt, "~> 0.2.0", [hex: :bunt, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm"},
-  "httpotion": {:hex, :httpotion, "3.1.2", "50e3e559c2ffe8c8908c97e4ffb01efc1c18e8547cc7ce5dd173c9cf0a573a3b", [:mix], [{:ibrowse, "== 4.4.0", [hex: :ibrowse, repo: "hexpm", optional: false]}], "hexpm"},
+  "httpotion": {:hex, :httpotion, "3.1.3", "fdaf1e16b9318dcb722de57e75ac368c93d4c6e3c9125f93e960f953a750fb77", [:mix], [{:ibrowse, "== 4.4.0", [hex: :ibrowse, repo: "hexpm", optional: false]}], "hexpm"},
   "ibrowse": {:hex, :ibrowse, "4.4.0", "2d923325efe0d2cb09b9c6a047b2835a5eda69d8a47ed6ff8bc03628b764e991", [:rebar3], [], "hexpm"},
   "jason": {:hex, :jason, "1.1.2", "b03dedea67a99223a2eaf9f1264ce37154564de899fd3d8b9a21b1a6fd64afe7", [:mix], [{:decimal, "~> 1.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm"},
   "jiffy": {:hex, :jiffy, "0.15.2", "de266c390111fd4ea28b9302f0bc3d7472468f3b8e0aceabfbefa26d08cd73b7", [:rebar3], [], "hexpm"},