VM as OS Concepts

[mwsltn]

Just some rambling thoughts per our conversation

Web-Based Architectures

• complex interactions between multiple pieces of software
• often an expert is required to "harden" a system with so many moving parts

VMs

• necessary for write once run everywhere
• multiple vms can compete for resources in ways that hide resource needs of individual processes (threads) contained within them

VM as OS

• Runs only on one combination of hardware as exposed by VirtualBox (or some other VM)
• Custom linux installation, missing significant software as compared to a standard linux system.
• login() which usually owns all other processes after boot, will be replaced with a ‘system()’ program which forks into an ‘aREPL’, asynchronous REPL, and an nginx (or some other webserver) implementation
• The system presents configuration options via the standard monitor output channels, these can be serialized to disk in between runs, eventually this can be exposed as a secure ncurses interface
• the ‘expert level’ configuration is provided by the language developers, potential flaw: each 0-day explit effects all systems
• the webserver reads a request an http put request to http://foo.bar/eval that looks like:
  {
  souce: Byte array representing source input string
  }

and returns a JSONs that looks like:
{
out: standard output channel
err: standard input channel
}
whenever the output buffer size is reached

The byte array source is passed to the aREPL where it’s read, evaluated and when nevessary things are printed back

Tying systems together

L should implement many web-spec. functions as basic language constructs implemented in C and linked against the standard libraries that provide those functions, things like:

scp, sendEmal, get, put, post, update, delete (by no means exhaustive)

as well as all the language features given in the spec

provide a modular way of loading new “sys-tasks” into the language

[brendanberg]

I'm working on a layer diagram to clarify what is a VM/runtime feature and what is buit on top of that. Here are my thoughts so far:

VM and runtime

• Provide the low-level features like memory management, process management, security, I/O, etc.
• Message passing and queuing are OS features.
• How about a decent event model that makes Obj-C style KVO possible?
• TCP/IP stack in here somewhere.

HTTP Utilities

• No Nginx; HTTP server written in L to take advantage of the process model. Look at Go's HTTP server or Tornado for inspiration.
• I like the idea of a REST API for loading processes. Sort of an S3 for computation. Something like:
  • POST /<USER_FINGERPRINT>/process to upload; returns the process's URI. Body can be source or binary.
  • GET /<PROCESS_ID> to inspect process; representation includes URIs for output streams and event queues.
  • GET /<PROCESS_ID>/<STREAM_NAME> to read output stream; use Comet / HTTP streaming for output streams (client opens an HTTP connection and just sits there).
• System configuration can include a callback URI: the server will make REST calls to another server to log certain events.
  • PubSubHubub or similar protocol.

---

Obviously I need to think some things over. I suggest we use the comments here to stage ideas and use the issue body (if it is in fact editable) for a draft spec that I'll move to the actual repo once we start to feel good about it.

Here are some open questions; I don't have answers, but maybe strong opinions. First, I recognize the effectiveness of using off-the-shelf components for getting a proof-of-concept together, but also remember that I'm a minimalist. We should weigh the amount of work required to cut away all the fat, vs. write components from scratch. Maybe my point is that we shouldn't be afraid to, for example, write a scheduler from scratch while using a minimally modified memory manager. Again, how much is in L and how much is compiled C is an open question.

[mwsltn]

The idea behind using a stripped down linux kernel and OTS components is that we can replace things with custom ones as we want / need to. There's a lot of moving parts in this -- so I figure we take as many ones that work as we can to sort of get it together and get a dev. environment going.

I've written a scheduler before (part of an OS course) and it's really not that difficult.

I think we should implement as much in L as possible -- but obviously there will be custom C components.

[brendanberg]

You're totally right. Using an OTS kernel, etc. makes sense. Though some components will definitely have to be customized. The scheduler, for example will be pretty custom. Eventually replacing lower and lower level components in the stack...

How about a BSD kernel?
How much can we avoid GNU components?