Small Clojure Interpreter
I want a limited dialect of Clojure for a single-purpose, scripted application. Sci will fit nicely.
— @tiagoluchini
(require '[sci.core :as sci])
(sci/eval-string "(inc 1)") => ;; 2
(sci/eval-string "(inc x)" {:namespaces {'user {'x 2}}}) ;;=> 3Try SCI in your browser at NextJournal.
For usage with GraalVM native-image check here.
You want to evaluate code from user input, or use Clojure for a DSL inside your
project, but eval isn't safe or simply doesn't work.
This library works with:
- Clojure on the JVM
- Clojure compiled with GraalVM native
- ClojureScript, even when compiled with
:advanced, and JavaScript
Sci is used in:
- 4ever-clojure. 4clojure as a static web page.
- Babashka. A Clojure scripting tool that plays well with Bash.
- Bootleg. An HTML templating CLI.
- Bytefield-svg. NodeJS library to generate byte field diagrams.
- Cardigan Bay. Wiki engine in Clojure.
- Chlorine. Socket-REPL and nREPL package for Atom editor.
- Clj-kondo. A Clojure linter that sparks joy.
- Closh. Bash-like shell based on Clojure. GraalVM port is work in progress.
- Dad. A configuration management tool.
- Datalevin. Durable Datalog database.
- Firn. Org-mode static site generator.
- For-science. Discord bot.
- Jet. CLI to convert between JSON, EDN and Transit.
- Logseq. A local-only outliner notebook which supports both Markdown and Org mode.
- Malli. Plain data Schemas for Clojure/Script.
- PCP. Clojure Processor (PHP replacement).
- PGMig. Fast Standalone PostgreSQL Migration Runner.
- Prose. Alternate syntax for Clojure, similar to what Pollen brings to Racket.
- SICMUtils. Computer Algebra System in Clojure, tailored for math and physics investigations.
- Spire. Pragmatic provisioning using Clojure.
- Zprint. Tool to beautifully format Clojure(script) code and data.
Are you using SCI in your company or projects? Let us know here.
Experimental. Breaking changes are expected to happen at this phase. They will be documented in the CHANGELOG.md.
Use as a dependency:
See the generated codox documentation.
The main API function is sci.core/eval-string which takes a string to evaluate
and an optional options map.
In sci, defn does not mutate the outside world, only the evaluation
context inside a call to sci/eval-string.
By default SCI only enables access to most of the Clojure core functions. More
functions can be enabled by using :namespaces and :classes. Normally you
would use SCI's version of println but here, for the purposes of
demonstration, we use use Clojure's version of println instead:
user=> (require '[sci.core :as sci])
user=> (sci/eval-string "(println \"hello\")" {:namespaces {'clojure.core {'println println}}})
hello
nilIt is also possible to provide namespaces which can be required inside a SCI program:
user=> (def opts {:namespaces {'foo.bar {'println println}}})
user=> (sci/eval-string "(require '[foo.bar :as lib]) (lib/println \"hello\")" opts)
hello
nilYou can provide a list of allowed symbols. Using other symbols causes an exception:
user=> (sci/eval-string "(inc 1)" {:allow '[inc]})
2
user=> (sci/eval-string "(dec 1)" {:allow '[inc]})
ExceptionInfo dec is not allowed! [at line 1, column 2] clojure.core/ex-info (core.clj:4739)Providing a list of disallowed symbols has the opposite effect:
user=> (sci/eval-string "(inc 1)" {:deny '[inc]})
ExceptionInfo inc is not allowed! [at line 1, column 2] clojure.core/ex-info (core.clj:4739)Providing a macro as a binding can be done by providing a normal function that:
- has
:sci/macroon the metadata set totrue - has two extra arguments at the start for
&formand&env:
user=> (def do-twice ^:sci/macro (fn [_&form _&env x] (list 'do x x)))
user=> (sci/eval-string "(do-twice (f))" {:bindings {'do-twice do-twice 'f #(println "hello")}})
hello
hello
nilAlternatively you can refer to the macro from the Clojure environment via the var (this only works in a JVM environment):
user=> (defmacro do-twice [x] (list 'do x x))
user=> (sci/eval-string "(do-twice (f))" {:namespaces {'user {'do-twice #'do-twice 'f #(println "hello")}}})To remain safe and sandboxed, SCI programs do not have access to Clojure vars, unless you explicitly provide that access. SCI has its own var type, distinguished from Clojure vars.
In a SCI program these vars are created with def and defn just like in
normal Clojure:
(def x 1)
(defn foo [] x)
(foo) ;;=> 1
(def x 2)
(foo) ;;=> 2Dynamic vars with thread-local bindings are also supported:
(def ^:dynamic *x* 1)
(binding [*x* 10] x) ;;=> 10
(binding [*x* 10] (set! x 12) x) ;;=> 12
x ;;=> 1Creating SCI vars from Clojure can be done using sci/new-var:
(def x (sci/new-var 'x 10))
(sci/eval-string "(inc x)" {:namespaces {'user {'x x}}}) ;;=> 11To create a dynamic SCI var you can set metadata or use sci/new-dynamic-var:
(def x1 (sci/new-var 'x 10 {:dynamic true}))
(sci/eval-string "(binding [*x* 12] (inc *x*))" {:namespaces {'user {'*x* x1}}}) ;;=> 13
(def x2 (sci/new-dynamic-var 'x 10))
(sci/eval-string "(binding [*x* 12] (inc *x*))" {:namespaces {'user {'*x* x2}}}) ;;=> 13SCI vars can be bound from Clojure using sci/binding:
(def x (sci/new-dynamic-var 'x 10))
(sci/binding [x 11] (sci/eval-string "(inc *x*)" {:namespaces {'user {'*x* x2}}})) ;;=> 11The dynamic vars *in*, *out*, *err* SCI a sci program correspond to the
dynamic sci vars sci/in, sci/out and sci/err in the API. These
vars can be rebound as well:
(def sw (java.io.StringWriter.))
(sci/binding [sci/out sw] (sci/eval-string "(println \"hello\")")) ;;=> nil
(str sw) ;;=> "hello\n"A shorthand for rebinding sci/out is sci/with-out-str:
(sci/with-out-str (sci/eval-string "(println \"hello\")")) ;;=> "hello\n"To copy the public vars of a Clojure namespace and to reify the Clojure vars into
corresponding SCI vars, you can use ns-publics in Clojure and the following API functions:
sci/create-ns: creates an object that identifies a SCI namespace and carries the metadata of a SCI namespacessci/copy-var: macro that copies a Clojure var to a SCI namespace (created throughsci/create-ns). Automatically converts dynamic vars and macros. Captures docstrings and arglists.
E.g. given the following Clojure namespace:
(ns foobar)
(defmacro do-twice [x] (list 'do x x))
(defn times-two [x]
(* x 2))you can re-create that namespace in a SCI context like this:
(require 'foobar)
(def fns (sci/create-ns 'foobar-ns nil))
(def foobar-ns {'do-twice (sci/copy-var foobar/do-twice fns)
'times-two (sci/copy-var foobar/times-two fns)})
(def ctx (sci/init {:namespaces {'foobar foobar-ns}}))
(sci/binding [sci/out *out*]
(sci/eval-string* ctx "(foobar/do-twice (prn :x))"))
:x
:x
nil
(sci/eval-string* ctx "(foobar/times-two 2)")
4To copy an entire namespace without enumerating all vars explicitly with
sci/copy-var you can use the following approach using ns-publics and sci/new-var:
(reduce (fn [ns-map [var-name var]]
(let [m (meta var)
no-doc (:no-doc m)
doc (:doc m)
arglists (:arglists m)]
(if no-doc ns-map
(assoc ns-map var-name
(sci/new-var (symbol var-name) @var
(cond-> {:ns fns
:name (:name m)}
(:macro m) (assoc :macro true)
doc (assoc :doc doc)
arglists (assoc :arglists arglists)))))))
{}
(ns-publics 'foobar))To enable printing to stdout and reading from stdin you can SCI-bind
sci/out and sci/in to *out* and *in* respectively:
(sci/binding [sci/out *out*
sci/in *in*]
(sci/eval-string "(print \"Type your name!\n> \")")
(sci/eval-string "(flush)")
(let [name (sci/eval-string "(read-line)")]
(sci/eval-string "(printf \"Hello %s!\" name)
(flush)"
{:bindings {'name name}})))
Type your name!
> Michiel
Hello Michiel!When adding a Clojure function to SCI that interacts with *out* (or *in* or *err*), you
can hook it up to SCI's context. For example, a Clojure function that writes to *out*
can be Clojure bound to SCI's out:
user=> (defn foo [] (println "yello!"))
#'user/foo
user=> ;; without binding *out* to sci's out, the Clojure function will use its default *out*:
user=> (sci/eval-string "(with-out-str (foo))" {:namespaces {'user {'foo foo}}})
yello!
""
;; Let's hook foo up to SCI's context:
user=> (defn wrapped-foo [] (binding [*out* @sci/out] (foo)))
#'user/wrapped-foo
user=> (sci/eval-string "(with-out-str (foo))" {:bindings {'foo wrapped-foo}})
"yello!\n"Creating threads with future and pmap is disabled by default, but can be
enabled by requiring sci.addons.future and applying the sci.addons.future/install function
to the sci options:
(ns my.sci.app
(:require
[sci.core :as sci]
[sci.addons.future :as future]))
(sci/eval-string "@(future (inc x))"
(-> {:namespaces {'user {'x 1}}}
(future/install)))
;;=> 2For conveying thread-local SCI bindings to an external future use
sci/future:
(ns my.sci.app
(:require
[sci.core :as sci]
[sci.addons.future :as future]))
(def x (sci/new-dynamic-var 'x 10))
@(sci/binding [x 11]
(sci/future
(sci/eval-string "@(future (inc x))"
(-> {:namespaces {'user {'x x}}}
(future/install)))))
;;=> 12Adding support for classes is done via the :classes option:
(sci/eval-string "(java.util.UUID/randomUUID)"
{:classes {'java.util.UUID java.util.UUID}})
;;=> #uuid "312ba519-37e2-4109-b164-97fb140b57b0"To make this work with GraalVM you will also need to add an entry to your
reflection
config
for this class. Also see reflection.json.
By default, SCI only lets you interop with classes explicitly provided in the
:classes config. When a method call returns an instance of a class that is not
in :classes you won't be able to to interop on that. You can disable this
safety measure with {:classes {:allow :all}}.
In JS hosts, to allow interop with anything, use the following config:
{:classes {'js goog/global :allow :all}}Sci uses a context (internally implemented using an atom) to keep track of state
changes like newly defined namespaces and vars. The contents of the context
should be considered implementation detail. Every call to eval-string creates
a fresh context. To preserve state over multiple evaluations, you can create a
context using the same options as those for sci/eval-string.
(def opts {:namespaces {'foo.bar {'x 1}}})
(def sci-ctx (sci/init opts))The SCI context can then be re-used over successive invocations of
sci/eval-string*:
(sci/eval-string* sci-ctx "foo.bar/x") ;;=> 1
(sci/eval-string* sci-ctx "(ns foo.bar) (def x 2) x") ;;=> 2
(sci/eval-string* sci-ctx "foo.bar/x") ;;=> 2In a multi-user environment it can be useful to give each user their own
context. This can already be achieved with eval-string, but for performance
reasons it may be desirable to initialize a shared context once. This shared
context can then be forked for each user so that changes in one user's context
aren't visible to other users:
(def forked (sci/fork sci-ctx))
(sci/eval-string* forked "(def forked 1)")
(sci/eval-string* forked "forked") ;;=> 1
(sci/eval-string* sci-ctx "forked") ;;=> Could not resolved symbol: forkedSCI supports loading code via a hook that is invoked by SCI's implementation of
require. The job of this function is to find and return the source code for
the requested namespace. This passed-in function will be called with a single
argument that is a hashmap with a key :namespace. The value for this key will
be the symbol of the requested namespace.
This function should return a map with keys :file (containing the filename to
be used in error messages) and :source (containing the source code text). SCI
will evaluate that source code to satisfy the call to require. Alternatively
the function can return nil which will result in SCI throwing an exception
that the namespace could not be found.
The load hook is passed as part of the SCI options via the :load-fn:
(defn load-fn [{:keys [namespace]}]
(when (= namespace 'foo)
{:file "foo.clj"
:source "(ns foo) (def val :foo)"}))
(sci/eval-string "(require '[foo :as fu]) fu/val" {:load-fn load-fn})
;;=> :fooNote that internally specified namespaces (either the default namespaces that
SCI provides itself or those provides via the :namespaces key) will be
considered first and if found there, :load-fn will not be called, unless
:reload or :reload-all are used:
(sci/eval-string
"(require '[foo :as fu])
fu/val"
{:load-fn load-fn
:namespaces {'foo {'val (sci/new-var 'val :internal)}}})
;;=> :internal
(sci/eval-string
"(require '[foo :as fu] :reload)
fu/val"
{:load-fn load-fn
:namespaces {'foo {'val (sci/new-var 'val :internal)}}})
;;=> :fooAnother option for loading code is to provide an implementation of
clojure.core/load-file. An example is presented here.
(ns my.sci.app
(:require [sci.core :as sci]
[clojure.java.io :as io]))
(spit "example1.clj" "(defn foo [] :foo)")
(spit "example2.clj" "(load-file \"example1.clj\")")
(let [env (atom {})
opts {:env env}
load-file (fn [file]
(let [file (io/file file)
source (slurp file)]
(sci/with-bindings
{sci/ns @sci/ns
sci/file (.getAbsolutePath file)}
(sci/eval-string source opts))))
opts (assoc-in opts [:namespaces 'clojure.core 'load-file] load-file)]
(sci/eval-string "(load-file \"example2.clj\") (foo)" opts))
;;=> :fooImplementing a REPL can be done using the following functions:
sci/reader: returns reader for parsing source code, either from a string orio/readersci/parse-next: returns next form from readersci/eval-form: evaluates form returned byparse-next.
See examples for examples for both Clojure and ClojureScript. Run instructions are included at the bottom of each example.
To include an nREPL server in your sci-based project, you can use babashka.nrepl.
For general information about Clojure and GraalVM, check out clj-graal-docs and graalvm-clojure.
To build native images with GraalVM it is recommended to use clojure 1.10.3 or
later.
To use SCI as a native shared library from e.g. C, C++, Rust, read this tutorial.
Currently SCI doesn't support deftype and definterface.
Forms evaluated by SCI can produce lazy sequences. In Clojure, dynamic vars and laziness can be a tricky combination and the same goes for dynamic SCI vars.
Consider the following example:
(let [sw (java.io.StringWriter.)
result (sci/binding [sci/out sw] (sci/eval-string "(map print (range 10))"))]
(println "Output:" (str sw))
(println "Result:" result))If the returned lazy seq was realized within the sci/binding scope, the output
would be:
Output: 0123456789
Result: (nil nil nil nil nil nil nil nil nil nil)
But because the result is only printed outside of sci/binding the result is:
Execution error (ClassCastException) at sci.impl.io/pr-on (io.cljc:44).
class sci.impl.vars.SciUnbound cannot be cast to class java.io.Writer (sci.impl.vars.SciUnbound is in unnamed module of loader clojure.lang.DynamicClassLoader @4c2af006; java.io.Writer is in module java.base of loader 'bootstrap')
This happens because by the time the lazy-seq is realized, the binding scope for
sci/out is no longer established, and as a result the lazy-seq can no longer
be realized (due to the delayed calls to println, a side-effecting call
dependends on the value of sci/out, set by sci/binding.
If the result is intented to be serialized as a string, then one could simply serialize while the binding is still in place:
(let [sw (java.io.StringWriter.)]
(sci/binding [sci/out sw]
(let [result (sci/eval-string "(map print (range 10))")]
(println "Result:" result)
(println "Output:" (str sw)))))Note that we moved (println "Result:" result) before (println "Output:" (str sw)), since the first call takes care of realization.
Required: lein, the clojure CLI and GraalVM.
To succesfully run the GraalVM tests, you will have to compile the binary first
with script/compile.
To run all tests:
script/test/all
For running individual tests, see the scripts in script/test.
Use clojure -M:bench to benchmark the various phases of sci on the JVM:
$ clojure -M:bench --complete --sexpr "(let [x 1 y 2] (+ x y))" --quick
BENCHMARKING EXPRESSION: (let [x 1 y 2] (+ x y))
PARSE:
-> (let [x 1 y 2] (+ x y))
Evaluation count : 605268 in 6 samples of 100878 calls.
Execution time mean : 1,105801 µs
Execution time std-deviation : 209,640200 ns
Execution time lower quantile : 934,602619 ns ( 2,5%)
Execution time upper quantile : 1,459172 µs (97,5%)
Overhead used : 8,140922 ns
Found 1 outliers in 6 samples (16,6667 %)
low-severe 1 (16,6667 %)
Variance from outliers : 48,1886 % Variance is moderately inflated by outliers
ANALYSIS:
Evaluation count : 82248 in 6 samples of 13708 calls.
Execution time mean : 8,562699 µs
Execution time std-deviation : 1,289355 µs
Execution time lower quantile : 7,557778 µs ( 2,5%)
Execution time upper quantile : 10,039290 µs (97,5%)
Overhead used : 8,140922 ns
EVALUATION:
-> 3
Evaluation count : 1607688 in 6 samples of 267948 calls.
Execution time mean : 433,635072 ns
Execution time std-deviation : 67,003007 ns
Execution time lower quantile : 378,876890 ns ( 2,5%)
Execution time upper quantile : 512,818448 ns (97,5%)
Overhead used : 8,140922 nsUse --parse, --evaluate and/or --analyze to bench individual phases
(--complete will bench all of them). Leaving out --quick will run
criterium/bench instead of criterium/quick-bench.
To benchmark an expression within GraalVM native-image, run script/compile and then run:
$ time ./sci "(loop [val 0 cnt 1000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val))"
1000000
./sci 0.92s user 0.08s system 99% cpu 1.003 total- adgoji for financial support.
- Clojurists Together for financial support.
- Lee Read for the logo.
- contributors and other users posting issues with bug reports and ideas
Copyright © 2019-2021 Michiel Borkent
Distributed under the Eclipse Public License 1.0. This project contains code from Clojure and ClojureScript which are also licensed under the EPL 1.0. See LICENSE.