Adding spores SIP

Author: heathermiller

sips/pending/_posts/2013-06-10-spores.md

@@ -0,0 +1,211 @@
+---
+layout: sip
+disqus: true
+title: SIP-21 - Spores
+---
+
+**By: Heather Miller, Martin Odersky, and Philipp Haller**
+
+Functional programming languages are regularly touted as an enabling force, as an increasing number of applications become concurrent and distributed. However, managing closures in a concurrent or distributed environment, or writing APIs to be used by clients in such an environment, remains considerably precarious-- complicated environments can be captured by these closures, which regularly leads to a whole host of potential hazards across libraries/frameworks in Scala's standard library and its ecosystem.
+
+Potential hazards when using closures incorrectly:
+
+- Memory leaks
+- Race conditions, due to capturing mutable references
+- Runtime serialization errors, due to unintended capture of references
+
+This SIP outlines an abstraction, called _spores_, which enables safer use of closures in concurrent and distributed environments. This is achieved by controlling the environment which a spore can capture. Using an _assignment-on-capture_ semantics, certain concurrency bugs due to capturing mutable references can be avoided.
+
+## Motivating Examples
+
+### Futures and Akka Actors
+
+In the following example, an Akka actor spawns a future to concurrently process incoming requests.
+
+**Example 1:**
+
+    def receive = {
+      case Request(data) =>
+        future {
+          val result = transform(data)
+          sender ! Response(result)
+        }
+    }
+
+Capturing `sender` in the above example is problematic, since it does not return a stable value. It is possible that the future's body is executed at a time when the actor has started processing the next `Request` message which could be originating from a different actor. As a result, the `Response` message of the future might be sent to the wrong receiver.
+
+
+### Serialization
+
+The following example uses Java Serialization to serialize a closure. However, serialization fails with a `NotSerializableException` due to the unintended capture of a reference to an enclosing object.
+
+**Example 2:**
+
+    case class Helper(name: String)
+
+    class Main {
+      val helper = Helper("the helper")
+
+      val fun: Int => Unit = (x: Int) => {
+        val result = x + " " + helper.toString
+        println("The result is: " + result)
+      }
+    }
+
+Given the above class definitions, serializing the `fun` member of an instance of `Main` throws a `NotSerializableException`. This is unexpected, since `fun` refers only to serializable objects: `x` (an `Int`) and `helper` (an instance of a case class).
+
+Here is an explanation of why the serialization of `fun` fails: since `helper` is a field, it is not actually copied when it is captured by the closure. Instead, when accessing helper its getter is invoked. This can be made explicit by replacing `helper.toString` by the invocation of its getter, `this.helper.toString`. Consequently, the `fun` closure captures `this`, not just a copy of `helper`. However, `this` is a reference to class `Main` which is not serializable.
+
+The above example is not the only possible situation in which a closure can capture a reference to `this` or to an enclosing object in an unintended way. Thus, runtime errors when serializing closures are common.
+
+## Design
+
+The main idea behind spores is to provide an alternative way to create closure-like objects, in a way where the environment is controlled.
+
+A spore is created as follows.
+
+**Example 3:**
+
+    val s = spore {
+      val h = helper
+      (x: Int) => {
+        val result = x + " " + h.toString
+        println("The result is: " + result)
+      }
+    }
+
+The body of a spore consists of two parts:
+
+1. a sequence of local value (val) declarations only, and
+2. a closure.
+
+In general, a `spore { ... }` expression has the following shape.
+
+Note that the value declarations described in point 1 above can be `implicit` but not `lazy`.
+
+**Figure 1:**
+
+    spore {
+      val x_1: T_1 = init_1
+      ...
+      val x_n: T_n = init_n
+      (p_1: S_1, ..., p_m: S_m) => {
+        <body>
+      }
+    }
+
+The types `T_1, ..., T_n` can also be inferred.
+
+The closure of a spore has to satisfy the following rule. All free variables of the closure body have to be either
+
+1. parameters of the closure, or
+2. declared in the preceding sequence of local value declarations.
+
+**Example 4:**
+
+    case class Person(name: String, age: Int)
+    val outer1 = 0
+    val outer2 = Person("Jim", 35)
+    val s = spore {
+      val inner = outer2
+      (x: Int) => {
+        s"The result is: ${x + inner.age + outer1}"
+      }
+    }
+
+In the above example, the spore's closure is invalid, and would be rejected during compilation. The reason is that the variable `outer1` is neither a parameter of the closure nor one of the spore's value declarations (the only value declaration is: `val inner = outer2`).
+
+### Spore Type
+
+The type of the spore is determined by the type and arity of the closure. If the closure has type
+`A => B`, then the spore has type `Spore[A, B]`. For convenience we also define spore types for two or more parameters.
+
+In example 3, the type of s is `Spore[Int, Unit]`.
+Implementation
+The spore construct is a macro which
+
+- performs the checking described above, and which
+- replaces the spore body so that it creates an instance of one of the Spore traits, according to the arity of the closure of the spore.
+
+The `Spore` trait for spores of arity 1 is declared as follows:
+
+    trait Spore[-T, +R] extends Function1[T, R]
+
+For each function arity there exists a corresponding `Spore` trait of the same arity (called `Spore2`, `Spore3`, etc.)
+
+### Macro Expansion
+
+An invocation of the spore macro expands the spore's body as follows. Given the general shape of a spore as shown above, the spore macro produces the following code:
+
+      new <spore implementation class>[S_1, ..., S_m, R]({
+        val x_1: T_1 = init_1
+        ...
+        val x_n: T_n = init_n
+        (p_1: S_1, ..., p_m: S_m) => {
+          <body>
+        }
+      })
+
+Note that, after checking, the spore macro need not do any further transformation, since implementation details such as unneeded remaining outer references are removed by the new backend intended for inclusion in Scala 2.11. It's also useful to note that in some cases these unwanted outer references are already removed by the existing backend.
+
+The spore implementation classes follow a simple pattern. For example, for arity 1, the implementation class is declared as follows:
+
+      class SporeImpl[-T, +R](f: T => R) extends Spore[T, R] {
+        def apply(x: T): R = f(x)
+      }
+
+### Type Inference
+
+Similar to regular functions and closures, the type of a spore should be inferred. Inferring the type of a spore amounts to inferring the type arguments when instantiating a spore implementation class:
+
+      new <spore implementation class>[S_1, ..., S_m, R]({
+        // ...
+      })
+
+In the above expression, the type arguments `S_1, ..., S_m`, and `R` should be inferred from the expected type.
+
+Our current proposal is to solve this type inference problem in the context of the integration of Java SAM closures into Scala. Given that it is planned to eventually support such closures, and to support type inference for these closures as well, we plan to piggyback on the work done on type inference for SAMs in general to achieve type inference for spores.
+
+## Motivating Examples, Revisited
+
+We now revisit the motivating examples we described in the above section, this time in the context of spores.
+
+### Futures and Akka actors
+
+Using spores, example 1 can be re-written as follows:
+
+    def receive = {
+      case Request(data) =>
+        val s = spore {
+          val from = sender
+          val d = data
+          () => {
+            val result = transform(d)
+            from ! Response(result)
+          }
+        }
+        future { s() }
+    }
+
+In this case, the problematic capturing of `this` is avoided, since the result of `this.sender` is assigned to the spore's local value `from` when the spore is created. The spore conformity checking ensures that within the spore's closure, only `from` and `d` are used.
+
+### Serialization
+
+Using spores, example 2 can be re-written as follows:
+
+    case class Helper(name: String)
+
+    class Main {
+      val helper = Helper("the helper")
+
+      val fun: Spore[Int, Unit] = spore {
+        val h = helper
+        (x: Int) => {
+          val result = x + " " + h.toString
+          println("The result is: " + result)
+        }
+      }
+    }
+
+Similar to example 1, the problematic capturing of `this` is avoided, since `helper` has to be assigned to a local value (here, `h`) so that it can be used inside the spore's closure. As a result, `fun` can now be serialized without runtime errors, since `h` refers to a serializable object (a case class instance).
+