build prototype XS-based VatWorker process, define kernel-vatworker protocol

[warner]

Yesterday @dckc @michaelfig and I walked through a way to get started on the #1107 #1127 plan. The first step is probably to define the protocol to be spoken between the kernel and the VatWorker (something that can be sent over pipes). Then the next one is to build a VatWorker process: something that speaks this protocol over stdin/stdout, knows how to load a Bundle of vat code, can attach an instance of liveslots to it, and then wait for instructions on stdin.

This process will expect dispatch messages on stdin. Each one turns into a dispatch.something call into liveslots (which then invokes methods on various objects exported by the vat code). When liveslots calls syscall.something, that turns into a syscall message sent back over stdout. The process then blocks waiting for stdin to return the syscall results (which are almost always "undefined"). When the results come back, the process returns from the syscall, and liveslots continues.

The process is responsible for using setImmediate to detect when the vat becomes quiescent. At that point, the process should finally respond to the dispatch message over stdout. After that, the process goes back to waiting for the next dispatch.

This process should probably be written in Node first, but the goal is to quickly get it written in XS instead.

The idea is that we could eventually change the kernel's vatManager to spawn off one of these processes, instead of running the vat in the same process as the kernel.

The other idea is to start doing some meaningful integration with XS, so we can all keep learning about it. Our current XS prototype starts "from the top", putting the entire swingset (in fact the whole ag-solo, I think) in XS. But we know that our most important XS goals (safer TCB, checkpointing vats, efficient metering) will all be better satisfied "from the bottom", one vat at a time.

We could get to a point where we run some vats, perhaps the dynamic vats created to host contracts, in separate XS-based processes, while the static vats and the kernel continue to run in a single shared Node.js process. Or we might rewrite the kernel in a different language (Rust being my perennial favorite), and put the vats in something different, that is capable of running JS (WASM boxes running XS being my other favorite). Or we might start with VatWorkers being merely separate Workers in Node (which will probably help a lot with debugging, since DevTools knows a lot more about Workers than it does about bespoke child processes).

All of those cases will be helped by isolating the code necessary to run a vat, and separating it from the rest of the kernel. We don't necessarily want to run each vat in a separate process, in the long run, but if we can get to that point, then we can easily move that code back into a shared process later. The important thing is figuring out how to draw the line between vat and kernel.

We're not aiming to build the kernel side of this tool right now. This ticket is about building the vatWorker side. We'll drive it with a mock kernel (perhaps just a shell script that copies things to stdin at the right times, and checks the stdout responses for a match). For now, the only vat is has to handle is the demo one (vat-target in the gist below), and the only messages are the ones in the demo transcript.

kernel-vatworker protocol

I can think of three messages that we need, plus acks:

• kernel->vat loadVat(bundle), plus ack
• kernel->vat dispatch(type, args), plus (eventual) ack
• vat->kernel syscall(type, args), plus (prompt but async) response

loadVat

The first message the vatWorker receives is loadVat. This will include the "vat bundle", in the same format that dynamicVat.js receives them (the result of calling bundleSource(filename, 'nestedEvaluate')). The bundle is a JSON-serializable object; if the kernel-vatworker protocol just JSONs a message like {type: 'loadVat', bundle}, then the serialization is trivial.

The vatWorker will need to have a copy of liveSlots available. The vat bundle yields a buildRootObject function, which should be passed to makeLiveSlots.

Look at dynamicVat.js for some structure. If written in Node, this will use importBundle.

The vatWorker should send back a loadVat-specific ACK when the vat has been loaded successfully. If the bundle cannot be loaded (syntax error, runtime exception whne calling buildRootObject), the NAK should include the error details.

dispatch

On each crank, the kernel will send a dispatch message to the vatWorker. This will be one of dispatch.deliver, dispatch.notifyFulfillToData, dispatch.notifyFulfillToPresence, or dispatch.notifyReject. The vatWorker will parse the message and invoke the vat's dispatch object (really the liveslots dispatch object). The vatWorker won't ack the dispatch until the vat becomes quiescent (which will require a setImmediate check). In the meantime, the vat may make one or more syscalls. The protocol must be able to distinguish between outbound syscalls and the dispatch ack.

The dispatch might NAK if the vat explodes during processing. This could be a metering fault, or some deeper problem in the JS engine. The dispatch function may throw an exception: that is not grounds for a NAK (although it'd help to log the problem somewhere, maybe stderr).

Once the dispatch is ACKed, the vatWorker is expected to remain idle until the next dispatch arrives. The vat code is not given access to IO or the timer queue, so it should have no way to regain control until the next dispatch wakes it up.

syscall

While a dispatch is running, the vat can make syscalls (really the vat performs eventual-sends and resolves Promises, and liveslots turns these into syscalls). When liveslots invokes syscall.send or syscall.fulfillToDat or whatever, the vatWorker should serialize the details into a syscall message and write it to stdout, where the kernel will receive the syscall request and execute it. The vatWorker should perform a blocking read of stdin, waiting for the results of the syscall. We need liveslots to believe that it is making a synchronous invocation of the syscall method. No further vat code or liveslots code should execute until the kernel responds to the syscall request with the results.

The results of a syscall are almost always empty, just undefined. The exceptions are device invocations (syscall.callNow), and we aren't yet trying to support any of those in the vatWorker. We're also trying to get rid of synchronous syscalls entirely, but there are a lot of barriers. But, for now, the vatWorker should pretend that there are important results to be had; it should wait patiently for the kernel to respond to the syscall request, then deserialize the results and return them to the liveslots caller.

finishing the dispatch

The vatWorker must not ACK the kernel's dispatch message until the vat is idle. The vat might have returned from the initial dispatch call, but it might have performed local Promise operations that leave an arbitrary number of callbacks sitting on the promise queue. So the real test of the vat being idle is that a setImmediate callback can fire.

In the regular kernel, we callsetImmediate, then invoke the vat's dispatch function (in a Promise chain), then use the setImmediate to trigger the callback chain that allows the kernel to move on to the next message. It is left as an exercise to the reader to figure out how to make this work well in a vatWorker communicating over stdin/stdout. The blocking read we use to let liveslots think it has a synchronous syscall might not play nicely with the idea of using setImmediate to detect an idle vat.

demo transcript

https://gist.github.com/warner/ee5168d1afc4679ef92ce5164cb982b7 has a simple vat-target and a bootstrap function which sends it a series of five dispatches. The vat-target reacts to these with a set of syscalls. Combined, they exercise all of the vat/kernel interactions we're trying to implement for this vatWorker experiment. The gist also has a copy of the transcript this generates. We should be able to parse that in a simple program that feeds in the simulated kernel-vatworker protocol messages, and checks that the responses are correct.

pieces

A Node.js-based vatworker will need to include a copy of liveslots, importBundle, some mechanism to detect the vat being idle, some protocol handlers, and something to get the whole process started.

I'd bet that the simplest protocol to pursue is to JSON-serialize an object containing the message type, and the message body as a copy of the arguments of the original dispatch function. For example, the original kernel/vat boundary dispatch method has a signature of dispatch.deliver(target, method, argsdata, result) (where target and method are both strings, result is either a string or undefined/null, and argsdata is a "capdata" structure which is a string .body and an array-of-strings .slots). When the kernel wants to make the VatWorker do this, it could send:

JSON.stringify({ msgtype: 'dispatch', type: 'deliver', method, argsdata, result });

and when the kernel wants to ACK a syscall, it could send:

JSON.stringify({ msgtype: 'syscall-ack', results });

At the beginning of the connection, the kernel would also send:

JSON.stringify({ msgtype: 'load-bundle', bundle });

and the VatWorker would eventually send:

JSON.stringify({ msgtype: 'load-bundle-ack' });
// or
JSON.stringify({ msgtype: 'load-bundle-nak', error });

or something.

VatPowers

The real kernel provides a couple of extra objects to the vat's buildRootObject() function, in an argument named vatPowers. This includes a pair of string-to-string transformation functions (transformTildot and transformMetering), which are powerless but require Babel to build, and don't run so well in a powerless SES compartment. We don't need these for this prototype, and transformMetering will be removed from vatPowers soon anyways. The VatWorker will be responsible for transformMetering, but we can figure out how to bake that into the Compartments it builds later.

There are also some other vatPowers related to metering that we don't need to support for now.

subsequent steps

I'm going to see if I can split the kernel/vatManager code up to facilitate the VatWorker connection. My goal is to make the code that builds a static or dynamic vat provide a way to specify where that new vat ought to live. This is the place that could fork off a new Node.js or XS process, tell it to load the vat bundle, and then start sending dispatch messages to it. It could also create a new in-process Worker to host the vat, communicating via structured-clone postMessage channels (same kernel-vatworker protocol, but not serialized all the way down into strings).

[dckc]

progress:

1. 2020-07-18 0c23955 kernelSimulator sends load-bundle message (to nobody)
2. 2020-07-18 2c5e4fa vatWorker: read JSON lines
3. 2020-07-18 811b90d vatWorker: handle load-bundle
4. 2020-07-18 d636d26 vatWorker: toward dispatch (CHECKPOINT)

https://gist.github.com/dckc/f8e0b5d838079a994784d599c282cce7

for reference: git log --date=short --pretty='1. %ad %h %s'|tac

[dckc]

@warner how is syscall.subscribe supposed to work?

I started on dispatch. At first I had syscall as a function (that just threw) but I got syscall.subscribe is not a function. So I turned syscall into an object with stub methods. This is what I've got so far:

$ node -r esm kernelSimulator.js >,halfDuplex
$ node -r esm vatWorker.js <,halfDuplex 
loaded bundle with methods Array <Array <[Object: null prototype] {}>>(1) [ 'buildRootObject' ]
{"msgtype":"load-bundle-ack"}
{"msgtype":"dispatch-ack"}
{"msgtype":"dispatch-ack"}
{"msgtype":"dispatch-ack"}
{"msgtype":"dispatch-ack"}
{"msgtype":"dispatch-ack"}
(node:20832) UnhandledPromiseRejectionWarning: Error: TODO: syscall.subscribe(p-60)
    at Object.subscribe (/home/connolly/projects/agoric/agoric-sdk/make-vat-transcript/vatWorker.js:99:19)
    at convertSlotToVal (/home/connolly/projects/agoric/agoric-sdk/packages/SwingSet/src/kernel/liveSlots.js:246:17)

...

1. 2020-07-18 d636d26 vatWorker: toward dispatch (CHECKPOINT)
2. 2020-07-18 851ac09 kernelSimulator: prune vestigial saveVat
3. 2020-07-18 5fc4170 vatWorker: dispatch up to syscall.subscribe

[dckc]

1st achievement unlocked.

kernelSimulator.js is only half-duplex so far. full duplex might expose shortcomings in vatWorker.js.

$ node -r esm kernelSimulator.js >,halfDuplex
$ cat ,halfDuplex
{"msgtype":"load-bundle","bundle":{"source":"function getExportWithNestedEvaluate(filePrefix) {\n  'use strict';\n  // Serialised sources.\n
...
{"msgtype":"dispatch","type":"deliver","args":["o+0","zero",{"body":"[{\"@qclass\":\"slot\",\"index\":0},{\"@qclass\":\"slot\",\"index\":1},{\"@qclass\":\"slot\",\"index\":2}]","slots":["o-50","p-60","p-61"]},"p-62"]}
{"msgtype":"syscall-ack","response":null}
{"msgtype":"syscall-ack","response":null}
{"msgtype":"syscall-ack","response":null}
{"msgtype":"syscall-ack","response":null}
{"msgtype":"syscall-ack","response":null}
{"msgtype":"dispatch","type":"deliver","args":["o+0","one",{"body":"[]","slots":[]},"p-63"]}
{"msgtype":"syscall-ack","response":null}
{"msgtype":"syscall-ack","response":null}
{"msgtype":"syscall-ack","response":null}
{"msgtype":"dispatch","type":"notifyFulfillToPresence","args":["p-60","o-50"]}
{"msgtype":"dispatch","type":"notifyReject","args":["p-61",{"body":"{\"@qclass\":\"error\",\"name\":\"Error\",\"message\":\"four\"}","slots":[]}]}
{"msgtype":"dispatch","type":"notifyFulfillToData","args":["p+5",{"body":"[\"data\",{\"@qclass\":\"slot\",\"index\":0}]","slots":["o-50"]}]}


$ node -r esm vatWorker.js ./fifo <,halfDuplex 
{"msgtype":"load-bundle-ack"}
{"msgtype":"syscall","method":"subscribe","args":["p-60"]}
{"msgtype":"syscall","method":"subscribe","args":["p-61"]}
{"msgtype":"syscall","method":"send","args":["o-50","callback",{"body":"[11,12]","slots":[]},"p+5"]}
{"msgtype":"syscall","method":"subscribe","args":["p+5"]}
{"msgtype":"syscall","method":"fulfillToData","args":["p-62",{"body":"[{\"@qclass\":\"slot\",\"index\":0},{\"@qclass\":\"slot\",\"index\":1}]","slots":["p+6","p+7"]}]}
{"msgtype":"dispatch-ack"}
{"msgtype":"syscall","method":"fulfillToPresence","args":["p+6","o-50"]}
{"msgtype":"syscall","method":"reject","args":["p+7",{"body":"{\"@qclass\":\"error\",\"name\":\"Error\",\"message\":\"oops\"}","slots":[]}]}
{"msgtype":"syscall","method":"fulfillToData","args":["p-63",{"body":"1","slots":[]}]}
{"msgtype":"dispatch-ack"}
{"msgtype":"dispatch-ack"}
{"msgtype":"dispatch-ack"}
{"msgtype":"dispatch-ack"}

1. 2020-07-18 5fc4170 vatWorker: dispatch up to syscall.subscribe
2. 2020-07-18 1ef5484 vatWorker: UnhandledPromiseRejectionWarning flailing WIP
3. 2020-07-18 f99ff93 vatWorker: refactor eachLine as LinePipe.readline
4. 2020-07-18 5a9dd78 syscall stubbed out, less sync/async trick
5. 2020-07-18 6d9e1f6 vatWorker works in 1 case
   https://gist.github.com/dckc/f8e0b5d838079a994784d599c282cce7

[dtribble]

🔥

[dckc]

made substantial progress on porting to xs. fwrite to stdout is failing. diagnostics at the js level are no good so the next step is gdb. Not sure I have the energy for that just now.

1. 2020-07-18 6d9e1f6 vatWorker works in 1 case
2. 2020-07-19 d98b55a eslint config etc. from new-repo, SwingSet
3. 2020-07-19 6582a54 lint-fix - WIP
4. 2020-07-19 dbd3047 xs console
5. 2020-07-19 4ac81dd links to liveSlots and related kernel sources
6. 2020-07-19 fd8c4bb harden() using xs deepfreeze Object.freeze(x, true)
7. 2020-07-19 5fe032c misc @Agoric packages by symlink
8. 2020-07-19 bbf0c0a xs-run gets to ... setImmediate: undefined ...
9. 2020-07-19 86945e2 lint config, tweaks
10. 2020-07-19 bbd09da load-bundle to: modFile.c:141: exception: (host): file write failed!

[dckc]

tee hee... fwrite failed because I didn't open the file for writing

@warner @michaelfig any chance you can reproduce my results?

$ cd agoric-swingset
$ git clone https://gist.github.com/dckc/f8e0b5d838079a994784d599c282cce7 make-vat-transcript
$ cd make-vat-transcript
$ echo $MODDABLE
/home/connolly/projects/moddable
$ (pushd $MODDABLE; git log -n 1)
~/projects/moddable ~/projects/agoric/agoric-sdk/make-vat-transcript
commit e3d70c0b746c653e06aa6602d7e88d87bae71ec4 (HEAD -> ag-linux-cli, dckc/ag-linux-cli)
Merge: c5abcd43 928e4e62
Author: Dan Connolly <dckc@madmode.com>
Date:   Mon May 4 21:15:06 2020 -0500

    Merge branch 'public' into ag-linux-cli
$ yarn xs-build-release
...
$ ./build/bin/lin/release/make-vat-transcript <,halfDuplex
lin_xs_cli: loading top-level main.js
 lin_xs_cli: loaded
lin_xs_cli: invoking main(argv)
 lin_xs_cli: main() returned a promise; entering event loop
{"msgtype":"load-bundle-ack"}
{"msgtype":"syscall","method":"subscribe","args":["p-60"]}
{"msgtype":"syscall","method":"subscribe","args":["p-61"]}
{"msgtype":"syscall","method":"send","args":["o-50","callback",{"body":"[11,12]","slots":[]},"p+5"]}
{"msgtype":"syscall","method":"subscribe","args":["p+5"]}
{"msgtype":"syscall","method":"fulfillToData","args":["p-62",{"body":"[{\"@qclass\":\"slot\",\"index\":0},{\"@qclass\":\"slot\",\"index\":1}]","slots":["p+6","p+7"]}]}
{"msgtype":"dispatch-ack"}
{"msgtype":"syscall","method":"fulfillToPresence","args":["p+6","o-50"]}
{"msgtype":"syscall","method":"reject","args":["p+7",{"body":"{\"@qclass\":\"error\",\"name\":\"Error\",\"message\":\"oops\"}","slots":[]}]}
{"msgtype":"syscall","method":"fulfillToData","args":["p-63",{"body":"1","slots":[]}]}
{"msgtype":"dispatch-ack"}
{"msgtype":"dispatch-ack"}
{"msgtype":"dispatch-ack"}
{"msgtype":"dispatch-ack"}

1. 2020-07-19 bbd09da load-bundle to: modFile.c:141: exception: (host): file write failed!
2. 2020-07-19 51fd20b vatWorker runs on xs in 1 case

[dckc]

Ugh... I back-ported the synchronous readLine API to node...

1. 2020-07-20 8122682 node driver for vatWorker using readSync, sleep (WIP)

but now (EDIT: unless I move node_modules out of the way) I'm losing thusly:

$ node -r esm index.js <,halfDuplex 
{"msgtype":"load-bundle-ack"}
{"msgtype":"syscall","method":"subscribe","args":["p-60"]}
{"msgtype":"syscall","method":"subscribe","args":["p-61"]}
{"msgtype":"syscall","method":"fulfillToData","args":["p-62",{"body":"[{\"@qclass\":\"slot\",\"index\":0},{\"@qclass\":\"slot\",\"index\":1}]","slots":["p+5","p+6"]}]}
(node:13290) UnhandledPromiseRejectionWarning: TypeError: o["callback"] is not a function
    at /home/connolly/projects/agoric/agoric-sdk/make-vat-transcript/node_modules/@agoric/eventual-send/src/index.js:425:15
    at Object.applyMethod (/home/connolly/projects/agoric/agoric-sdk/make-vat-transcript/node_modules/@agoric/eventual-send/src/index.js:412:14)
    at win (/home/connolly/projects/agoric/agoric-sdk/make-vat-transcript/node_modules/@agoric/eventual-send/src/index.js:444:52)
    at /home/connolly/projects/agoric/agoric-sdk/make-vat-transcript/node_modules/@agoric/eventual-send/src/index.js:461:20
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at process.runNextTicks [as _tickCallback] (internal/process/task_queues.js:66:3)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:13290) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
{"msgtype":"dispatch-ack"}
unexpected msgtype syscall-ack
unexpected msgtype syscall-ack

[michaelfig]

Try searching for .callback(

[warner]

the demo vat makes an E(callbackObj).callback(), to a presence that arrives in the dispatch.deliver() that invokes zero(). It sounds like liveslots and/or HandledPromise is not producing a functioning Presence. The IDs of the fulfillToData body (p+5 and p+6) should have been p+6 and p+7, which supports the idea that the E().callback() didn't happen (it failed to allocate the result promise as p+5, leaving that ID available for the response). It didn't throw an exception in the invocation of zero(), though, because if it did, the fulfillToData would have been a reject.

I'd look at what callbackObj is, and whether it also appears in the WeakMap maintained by eventual-send.

[dckc]

The o["callback'] thing goes away when I get rid of node_modules; it's pretty clearly version skew, so I'm not worrying about it.

[dckc]

It's working end-to-end now...

1. 2020-07-21 a2256bc add stdio module
2. 2020-07-21 b684fc2 kernelSimulator: spawn worker; full-duplex protocol
   https://gist.github.com/dckc/f8e0b5d838079a994784d599c282cce7

Perhaps tomorrow we can see if one of you can reproduce my work... and then maybe get bold and try non-trivial vats.

[warner]

Tonight we were able to get this working on my machine too. We tested it against the encouragement bot demo (one message) and it worked. We then tried to test it against the zoe test (30 messages), and something is failing as it tries to parse the bundle (which is about 320kB, much larger than the other demos).

One task is to get a better readline() in C (XS is using ftell to try and measure the size of the file it is reading from, except it's really a pipe, so it falls back to reading only one character at a time, and appending a JS string each time, which is O(JS^2)). We got part way through writing something better, but it's not clear that it works. For reference, our modFile.c wound up looking like:

static int bytes_read = 0;

void xs_file_read(xsMachine *the)
{
    fprintf(stderr, "in xs_file_read\n");
    FILE *file = getFile(the);
    int32_t result;
    int argc = xsmcArgc;
    int dstLen = (argc < 2) ? -1 : xsmcToInteger(xsArg(1));
    fprintf(stderr, " dstLen is %d\n", dstLen);
    void *dst;
    xsSlot *s1, *s2;
    struct stat buf;
    int32_t position = ftell(file);
    fprintf(stderr, " ftell says %d\n", position);

    fstat(fileno(file), &buf);
    if ((-1 == dstLen) || (buf.st_size < (position + dstLen))) {
        if (position >= buf.st_size) {
            fprintf(stderr, " read past end of file\n");
            xsUnknownError("read past end of file");
        }
        dstLen = buf.st_size - position;
        dstLen = 10000;
        fprintf(stderr, " dstLen is now %d\n", dstLen);
    }

    char *inbuf = malloc(10000);
    int offset;
    for (offset=0; offset < 10000; offset++) {
        char intc = fgetc(file);
        if (intc == '\n') {
            fprintf(stderr, " got LF at offset %d", offset);
            inbuf[offset] = intc;
            offset += 1;
            break;
        }
        if (intc == EOF) {
            fprintf(stderr, " got EOF at offset %d", offset);
            break;
        }
        inbuf[offset] = intc;
    }
    fprintf(stderr, " finished loop with offset %d", offset);
    dstLen = offset;

    s1 = &xsArg(0);

    xsmcVars(1);
    xsmcGet(xsVar(0), xsGlobal, xsID_String);
    s2 = &xsVar(0);
    if (s1->data[2] == s2->data[2]) {
        fprintf(stderr, " option 1\n");
        //xsResult = xsStringBuffer(NULL, dstLen);
        //dst = xsmcToString(xsResult);
    }
    else {
        fprintf(stderr, " option 2\n");
        xsmcSetArrayBuffer(xsResult, NULL, dstLen);
        dst = xsmcToArrayBuffer(xsResult);
    }

    xsResult = xsStringBuffer(inbuf, dstLen);
    dst = xsmcToString(xsResult);

    //result = fread(dst, 1, dstLen, file);
    fprintf(stderr, " result %d %.100s %d\n", dstLen, (char *)dst, bytes_read);
    bytes_read += dstLen;
    //if (result != dstLen)
    //    xsUnknownError("file read failed");
}

To generate the zoe bundle, I started from packages/zoe/test/swingsetTests/zoe, edited bootstrap to override the ARGV values:

      
      const argv2 = ['automaticRefundOk', 
                     [[3, 0, 0],
                      [0, 17, 0]],
                   ];
      const [testName, startingValues] = argv2;

      const { aliceP, bobP, carolP, daveP } = makeVats(
        log,
        vats,
        zoe,
        installations,
        startingValues,
      );

and ran swingset-runner: ../packages/swingset-runner/bin/runner --filedb run as described in my gist's run.sh. I think we looked at vat-zoe, which was v5. We used a short python snippet to sort the transcript lines numerically:

import sys
lines = open("transcript-zoe.txt").readlines()
lines.sort(key=lambda line: int(line.split(" ", 1)[0].split(".")[2]))
for line in lines:
    sys.stdout.write(line)

We had to update the XS manifest because I'd added compartment-wrapper.js to the import-bundle package since @dckc's work: "@agoric/compartment-wrapper": "xs_modules/@agoric/import-bundle/compartment-wrapper",, and also add a symlink from the xs_modules/ directory.

I'm continuing to work on refactoring the kernel so that we can plug this in.

[michaelfig]

Is using netstrings the answer, or just another problem to add to this one?

[warner]

I'm inclined towards netstrings, but @dckc said he was going to hack on a line-oriented reader a bit longer.

[dckc]

getline(3) fits pretty well

@warner writes:

One task is to get a better readline() in C ...

getline(3) seems to be just what the doctor ordered as long as we can rely on \n serving (only) as a message delimiter.

It works in at least one case:

1. 2020-07-23 ebd715c push readLine down to C, where we find getline(3) handy

Considering netstrings

netstrings start with a variable length string of digits, so we'd still have to use fgetc() or the like, plus ... is fread() guaranteed to read the whole thing at once?

Is there a canonical C implementation of netstrings in a dozen lines or so? I suppose so... Netstrings 19970201 by Bernstein includes:

The following C code reads a netstring and decodes it into a
dynamically allocated buffer buf of length len.

      if (scanf("%9lu",&len) < 1) barf();  /* >999999999 bytes is bad */
      if (getchar() != ':') barf();
      buf = malloc(len + 1);       /* malloc(0) is not portable */
      if (!buf) barf();
      if (fread(buf,1,len,stdin) < len) barf();
      if (getchar() != ',') barf();

The lack of free(buf) makes me wonder a bit. But that does seem to answer the fread question.

[michaelfig]

getline(3) seems to be just what the doctor ordered as long as we can rely on \n serving (only) as a message delimiter.

If you use JSON.stringify(obj) on the producing end, it is guaranteed not to produce any newlines.

It works in at least one case:

🎉

The lack of free(buf) makes me wonder a bit.

djb's netstring decoder mallocs buf and fills it with the decoded data. It's up to the caller of the decoder to do something with buf and to free(buf) when you're done with it.

[dckc]

sure... it's just that barf() is used before and after malloc() -- does barf() check whether buf is NULL before it calls free()? Is barf() actually supposed to be real C code? How does it affect control flow? So it just makes me wonder if this code is actually supposed to be used as-is in production settings.

[dckc]

zoe test fails with no target o+1

I've got the zoe test running with a node-powered vatWorker...

1. 2020-07-23 ebd715c push readLine down to C, where we find getline(3) handy
2. 2020-07-23 017dcd7 vatWorker: combine handle() and makeWorker() layers
3. 2020-07-23 0a96c6f kernelSimulator: log syscall requests
4. 2020-07-23 9b0d16d add finish to protocol
5. 2020-07-23 b5b477c integrate node vatWorker driver in full duplex mode
6. 2020-07-23 4959b9b toward zoe example: convert run.sh to Makefile
7. 2020-07-23 c29d29f automate zoe test setup
8. 2020-07-23 57fd89f capture generated transcript-zoe.txt

$ TRANSCRIPT=transcript-zoe.txt WORKERBIN=./node-vat-worker node -r esm kernelSimulator.js 
spawning Object <[Object: null prototype] {}> { workerBin: './node-vat-worker' }
loadVat done.
dispatching: v5.t.0
syscall request Object <[Object: null prototype] {}> {
  msgtype: 'syscall',
  method: 'reject',
  args: Array <Array <[Object: null prototype] {}>>(2) [
    'p-60',
    Object <[Object: null prototype] {}> {
      body: '{"@qclass":"error","name":"TypeError","message":"target[getZoe] does not exist, has zero,one"}',
      slots: Array <Complex prototype>(0) []
    }
  ]
}
dispatch done: v5.t.0
dispatching: v5.t.1
Object <[Object: null prototype] {}> {
  msgtype: 'dispatch-nak',
  error: 'no target o+1'
}
^C

[warner]

that looks like it's still using the demo bundle, not the zoe bundle.. try adding VAT1=...../packages/zoe/test/swingsetTests/zoe/vat-zoe.js

[dckc]

ok... fixed VAT1

1. 2020-07-24 dcbd1b5 fix VAT1 for zoe test

Now I get makeRoot is not a function.

$ make zoe
VAT1=../zoe/test/swingsetTests/zoe/vat-zoe.js TRANSCRIPT=transcript-zoe.txt WORKERBIN=./node-vat-worker node -r esm kernelSimulator.js
spawning Object <[Object: null prototype] {}> { workerBin: './node-vat-worker' }
load-bundle failed: TypeError <Object <Object <[Object: null prototype] {}>>>: makeRoot is not a function
    at build (/home/connolly/projects/agoric/agoric-sdk/packages/SwingSet/src/kernel/liveSlots.js:513:22)
    at makeLiveSlots (/home/connolly/projects/agoric/agoric-sdk/packages/SwingSet/src/kernel/liveSlots.js:572:7)
    at loadBundle (/home/connolly/projects/agoric/agoric-sdk/packages/xs-vat-worker/vatWorker.js:85:16)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at process.runNextTicks [as _tickCallback] (internal/process/task_queues.js:66:3)
    at /home/connolly/projects/agoric/agoric-sdk/node_modules/esm/esm.js:1:34535
    at /home/connolly/projects/agoric/agoric-sdk/node_modules/esm/esm.js:1:34176
    at process.<anonymous> (/home/connolly/projects/agoric/agoric-sdk/node_modules/esm/esm.js:1:34506)
    at Function.<anonymous> (/home/connolly/projects/agoric/agoric-sdk/node_modules/esm/esm.js:1:296856)
    at Function.<anonymous> (/home/connolly/projects/agoric/agoric-sdk/node_modules/esm/esm.js:1:296555)
Object <[Object: null prototype] {}> {
  msgtype: 'load-bundle-nak',
  error: Object <[Object: null prototype] {}> {}
}

This is with a little extra logging that I'm not committing just yet because I don't have console.error in xs yet:

$ git diff
diff --git a/vatWorker.js b/vatWorker.js
index 86680fc..bf654df 100644
--- a/vatWorker.js
+++ b/vatWorker.js
@@ -105,7 +105,7 @@ function makeWorker(io, setImmediate) {
         try {
           await loadBundle(name, message.bundle);
         } catch (error) {
-          // console.log('load-bundle failed:', error);
+          console.error('load-bundle failed:', error);
           io.writeLine(format({ msgtype: 'load-bundle-nak', error }));
           break;
         }

[michaelfig]

My liveSlots.js doesn't make any references to makeRoot. Is yours maybe outdated?

[dckc]

liveSlots version skew

yes, I think my liveSlots.js was older than the one @warner used; and of course that one is out of date now too. I'm now at efbf359 with yarn test Done in 543.82s.

>64K JSON

I think xs or my manifest.json has a 64K limit somewhere.

This looks like an argument for netstrings:

dispatching: v5.t.12
bad JSON  65536  chars: [ {"msgtype":"syscall","method":"send","args":["o-50","createVat",{"body":"[{\"source\":\"function getExportWithNestedEvaluate(filePrefix) {\\n  'use strict';\\n  // Serialised sources.\\n  if (filePrefix === undefined) {\\n    filePrefix = \\\"/bundled-source\\\";\\n  }\\n  const moduleFormat = \\\"nestedEvaluate\\\";\\n  const entrypoint = \\\"packages/zoe/src/contractFacet.js\\\";\\n  const sourceBundle = {\\n  \\\"packages/zoe/src/contractFacet.js\\\": \\\"'use strict';Object.defineProperty(exports, '__esModule', { value: true });var assert = require('../../assert/src/assert.js');var index = require('../../eventual-send/src/index.js');var offerSafety = require('./offerSafety.js');var rightsConservation = require('./rightsConservation.js');var objArrayConversion = require('./objArrayConversion.js');var cleanProposal = require('./cleanProposal.js');var state = require('./state.js');var\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\nevalContractCode = require('./evalContractCode.js'); /* This is the Zoe contract facet. Each time we make a new instance of a*/ /**\\\\n

[michaelfig]

>64K JSON

I think xs or my manifest.json has a 64K limit somewhere.

This looks like an argument for netstrings:

Unfortunately, the multiple layers of json are not easily converted to netstrings. It may be necessary to lift the limit.

[dckc]

I'm sure I'll have to lift the limit, but just to diagnose what's going on, I'm trying netstrings.

I get invalid script because the 500K bundle is bigger than the 32K parser.buffer in the xs manifest.

$ make zoe
VAT1=../zoe/test/swingsetTests/zoe/vat-zoe.js TRANSCRIPT=transcript-zoe.txt node -r esm kernelSimulator.js
spawning Object <[Object: null prototype] {}> {
  workerBin: './build/bin/lin/release/xs-vat-worker'
}
lin_xs_cli: loading top-level main.js
 lin_xs_cli: loaded
lin_xs_cli: invoking main(argv)
 lin_xs_cli: main() returned a promise; entering event loop
Error <Object <[Object: null prototype] {}>>: load-bundle-nak: invalid script

When I increase the parser.buffer, I get:

kernelSimulator send size 588579
unexpected exit: Object <[Object: null prototype] {}> { code: null, signal: 'SIGSEGV' }

I guess I should use gdb to attach to the running xs vatWorker...

for reference:

1. 2020-07-24 dcbd1b5 fix VAT1 for zoe test
2. 2020-07-27 c942f6b regen zoe transcript
3. 2020-07-27 f4e647c use netstrings (WIP: no support for non-ASCII lengths)
4. 2020-07-27 80ab01c add compartment-wrapper to manifest (KLUDGE)
5. 2020-07-27 26d11c6 add build package target as expected by yarn build
6. 2020-07-27 ad9dddf kernelSimulator: report child signal such as SIGSEGV
7. 2020-07-27 7b1805d kernelSimulator: don't bury nak message

[dckc]

zoe-node loses at v5.t.12

with the vatrunner running under node, I get:

$ make zoe-node
VAT1=../zoe/test/swingsetTests/zoe/vat-zoe.js TRANSCRIPT=transcript-zoe.txt WORKERBIN=./node-vat-worker node -r esm kernelSimulator.js
spawning Object <[Object: null prototype] {}> { workerBin: './node-vat-worker' }
loadVat done.
dispatching: v5.t.0
syscall request Object <[Object: null prototype] {}> {
  msgtype: 'syscall',
  method: 'fulfillToPresence',
  args: Array <Array <[Object: null prototype] {}>>(2) [ 'p-60', 'o+1' ]
}
dispatch done: v5.t.0
...
dispatch done: v5.t.11
dispatching: v5.t.12
(node:31236) UnhandledPromiseRejectionWarning: TypeError: Unexpected own properties in error: errno,syscall,code
    at isPassByCopyError (/home/connolly/projects/agoric/agoric-sdk/packages/marshal/marshal.js:168:11)
    at passStyleOf (/home/connolly/projects/agoric/agoric-sdk/packages/marshal/marshal.js:321:11)
    at Object.replacer (/home/connolly/projects/agoric/agoric-sdk/packages/marshal/marshal.js:453:25)
    at JSON.stringify (<anonymous>)
    at Object.serialize (/home/connolly/projects/agoric/agoric-sdk/packages/marshal/marshal.js:541:18)
    at /home/connolly/projects/agoric/agoric-sdk/packages/SwingSet/src/kernel/liveSlots.js:452:21

[dckc]

zoe is intermittent around v5.t.12 (createVat)

I got it working in the nodejs vatWorker some of the time:

1. 2020-07-27 7b1805d kernelSimulator: don't bury nak message
2. 2020-07-27 ce51e20 Makefile: add zoe-node target
3. 2020-07-27 4c13a54 node vatWorker: handle EAGAIN from fs.writeSync
4. 2020-07-29 9264ed1 node netstrings: handle partial writeSync()

keep stdio free to debug SEGV

1. 2020-07-29 5c37f60 kernelSimulator: move usage info to top of file
2. 2020-07-29 0c46c2d kernelSimulator: handle non-ascii netstrings
3. 2020-07-29 10f6f86 kernelSimulator: factor netstring read / write out of makeWorker
4. 2020-07-29 b814e11 refactor eachTranscriptEvent as runTranscript
5. 2020-08-01 f3832aa kernelSimulator: fix missing await for error handling
6. 2020-08-01 b35eed7 kernelSimulator: better diagnostic
7. 2020-08-01 5c6d827 keep stdin / stdout free for diagnostics; use fd 3, 4

struggling to sync with revised require design

I was getting vatRequire unprepared to satisfy require(@agoric/harden) so I updated agoric-sdk from efbf359..b2c980a but now the simple transcript now fails with require: undefined variable!:

1. 2020-08-01 5b80235 agoric-sdk catch-up

$ rm -rf ./build/
$ yarn xs-build
$ WORKERBIN=./build/bin/lin/debug/xs-vat-worker node -r esm kernelSimulator.js 
spawning Object <[Object: null prototype] {}> {
  workerBin: './build/bin/lin/debug/xs-vat-worker'
}
lin_xs_cli: loading top-level main.js
 lin_xs_cli: loaded
lin_xs_cli: invoking main(argv)
/home/connolly/projects/agoric/agoric-sdk/packages/import-bundle/src/index.js:57: exception: getExportWithNestedEvaluate: get require: undefined variable!
 lin_xs_cli: main() returned a promise; entering event loop
/home/connolly/projects/agoric/agoric-sdk/packages/xs-vat-worker/vatWorker.js:74: exception: throw!
/home/connolly/projects/agoric/agoric-sdk/packages/xs-vat-worker/vatWorker.js:101: exception: throw!
Error <Object <[Object: null prototype] {}>>: expected load-bundle-ack; found: load-bundle-nak; error: getExportWithNestedEvaluate: get require: undefined variable [{"msgtype":"load-bundle-nak","error":"getExportWithNestedEvaluate: get require: undefined variable"}]
    at expect (/home/connolly/projects/agoric/agoric-sdk/packages/xs-vat-worker/kernelSimulator.js:84:13)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async runTranscript (/home/connolly/projects/agoric/agoric-sdk/packages/xs-vat-worker/kernelSimulator.js:113:3)
    at async main (/home/connolly/projects/agoric/agoric-sdk/packages/xs-vat-worker/kernelSimulator.js:166:3)

1. 2020-08-01 5b80235 agoric-sdk catch-up

[dckc]

got this part added to #1407 tonight, along with syncing with upstream moddable:

• this package will incorporate an agoric-labs/moddable tree via git submodule

• this package will have a special yarn build-something command that:

  • builds the XS toolset
  • builds an xs-vat-worker executable (which incorporates liveSlots.js and maybe other files from swingset)

• this package will also export a JS library, whose main function merely uses require.resolve to return the absolute pathname of the generated xs-vat-worker executable (or undefined if it doesn't exist yet, because yarn build-something wasn't run)

I'm inclined to switch to this design:

• this package might instead/also have an API that encapsulates spawning the xs-vat-worker executable and running the messages-over-pipes protocol.

[dckc]

@warner @FUDCo help?

I'm getting 'running.get(...)' is undefined at vatAdminWrapper.js:70:13

In the kernel, I see that vatSyscallHandler returns ['ok', data] but doSyscall strips off the ok and returns just data. What does liveslots expect syscall to return? data or ['ok', data]?

Or is the failure here from something else altogether?

run log: https://gist.github.com/dckc/3031523281d25fbf735bd2b69f7a8625

current code: https://github.com/dckc/agoric-sdk/tree/xs-vat-worker 5fe2224