Pickler.generate for type constructors (such as Pickler.generate[Either[L,R]])

[vergenzt]

I'm on scala-pickling_2.11-0.10.0, and have done the following:

import scala.pickling._
import binary._
// ...
val pickler = Pickler.generate[Either[Int,String]]

I get an error: error: Cannot generate a pickler for Either[Int,String]. Recompile with -Xlog-implicits for details. (I'm using Scala IDE and Gradle to build my project, and for some reason I have been unable to get -Xlog-implicits to actually work in either.)

Any thoughts?

[vergenzt]

It seems to be similar to #50, but I have no way of knowing until I can get the extra logging to work. -_- I'm working on that.

[havocp]

Is your pickler implicit? in the code you quoted the val pickler isnt implicit.

That error message is really misleading. It's the implicitNotFound message saying no implicit pickler was found when trying to call a method that takes an implicit pickler. If the implicit generator macros are in scope, not found does mean "cannot generate"; but if you aren't using those macros (and I wouldn't) then this error instead means that no implicit pickler is in scope.

[vergenzt]

The error occurs on the generate line. (Originally it occurred when I was trying to use a pickler, but I reduced the code to this.)

Picklers definitely exist for Int and String, and I thought the whole point of this was that scala-pickling can generate composite picklers from simpler types? It doesn't make sense to me that an already-existing pickler of type Either[Int,String] has to exist in scope in order to generate a pickler of type Either[Int,String].

[havocp]

no, that doesn't make sense. To generate that pickler you should need picklers in scope for Int, String, Left, and Right, and Either needs to be sealed. i.e. to generate a static pickler for a trait it has to be a sealed trait and you have to have picklers for all the subtypes. You may also have to list subtypes in a @directSubclasses annotation depending on how the compiler tackles your files.

It's a lot easier to debug this stuff if you do not import the implicit macros; those are this thing: https://github.com/scala/pickling/blob/0.10.x/core/src/main/scala/scala/pickling/pickler/GenPicklers.scala#L10 and the matching one for unpickler. They are included in Defaults._ so I'd avoid Defaults._. For debugging it's so much easier if you keep it all "static only" (import scala.pickling.static.StaticOnly) and then import from Defaults only the specific needed picklers like the int pickler. If you do it this way you should get more meaningful errors instead of the implicit-not-found. https://github.com/sbt/serialization is set up this way by default.

If you set it up that way you should get an error about the missing picklers, e.g. no pickler for Left or whatever, or an error about something like how the trait isn't sealed, or something else useful.

[vergenzt]

Okay, so I think I understand what you're saying to be: In order to generate a pickler for Either[L,R], (a) Either has to be sealed (it's a sealed abstract class); (b) I need a pickler for Left[L,R]; and (c) I need a pickler for Right[L,R].

The pickler generation works when I use Int and String as in the following (which compiles with no errors):

implicit val lPickler = Pickler.generate[Left[Int,String]]
implicit val rPickler = Pickler.generate[Right[Int,String]]
implicit val pickler = Pickler.generate[Either[Int,String]]

However, in my actual usage rather than Int and String I have type parameters L and R. I've constrained them both to have picklers, unpicklers, and typetags of their own, so implicit picklers should exist for L and R in scope, right? However, the following does not compile:

def myFunction[L: Pickler: Unpickler: FastTypeTag, R: Pickler: Unpickler: FastTypeTag] = {

  implicit val lPickler = Pickler.generate[Left[L,R]]   // error
  implicit val rPickler = Pickler.generate[Right[L,R]]  // error
  implicit val pickler = Pickler.generate[Either[L,R]]  // error

  // ...
}

For the left pickler generation line, I get the following errors (same thing for right, just substitute R for L):

1. class type required but L found
2. type mismatch; found : Class[?0] where type ?0 <: L required: ?{def ==(x$1: ? >: Class[Any](classOf[scala.Any])): ?} Note that Class extends Any, not AnyRef. Such types can participate in value classes, but instances cannot appear in singleton types or in reference comparisons.
3. type mismatch; found : Class[?0] where type ?0 <: L required: ?{def ==(x$1: ? >: Class[Any](classOf[scala.Any])): Boolean} Note that Class extends Any, not AnyRef. Such types can participate in value classes, but instances cannot appear in singleton types or in reference comparisons.

And then for the Either pickler generation, I get (1) that it can't generate picklers for Left[Any,Any] or Right[Any,Any] and (2) that it can't find the implicits for those picklers it can't generate.

I have no idea where to go with this.

[jsuereth]

I think we should probably provide a custom-baked Either pickler.

[havocp]

Agreed that Either[L,R] should be included out of the box.

I think the generate macro just isn't smart enough to autogenerate a generic pickler like that - right now when we have a type parameter, we seem to always be writing out the pickler by hand. In principle generate could be improved to create a pickler for Foo[T] given that it has a pickler for T, but it doesn't know how to do that right now apparently. So if you look at all the existing picklers for collections and stuff: https://github.com/scala/pickling/tree/0.10.x/core/src/main/scala/scala/pickling/pickler those are hand-written and not generated, it looks like to me.

I looked around trying to find a good analogous pickler that you could copy. The best I came up with so far is Attributed[T] here: https://github.com/sbt/sbt-remote-control/blob/master/commons/protocol/src/main/scala/sbt/protocol/Keys.scala#L125 so you'd write Left[T] and Right[T] pickler-returning functions with some code like that, and then the Either unpickler would look at the type tag and delegate to the Left/Right unpickler based on it, and Either pickler would do either match { case l: Left => leftPickler; case r: Right => rightPickler } (along those lines anyway).

[jsuereth]

you actually can't use the type tag in the event the types are the same (or it's a type that has subclasses), you need some other specifier (unfortunately).

[vergenzt]

I think the generate macro just isn't smart enough to autogenerate a generic pickler like that - right now when we have a type parameter, we seem to always be writing out the pickler by hand.

Ahh, I didn't realize that. Has implementing this been considered? It doesn't sound like it would be infeasible, and I'd love to try my hand at it if you have any tips.

you actually can't use the type tag in the event the types are the same (or it's a type that has subclasses), you need some other specifier (unfortunately).

What does this mean?

[phaller]

Hi @vergenzt, there is some support for generics already implemented. It'd probably be best to look at some examples for generic types in the test suite. If you're interested in improving the support for generics, that'd be appreciated, of course!

[phaller]

Here are two tests for generated picklers for generic types:
https://github.com/scala/pickling/blob/0.10.x/core/src/test/scala/pickling/run/generics-simple.scala
https://github.com/scala/pickling/blob/0.10.x/core/src/test/scala/pickling/run/generics-tough.scala

As you can see, the generic types in these tests are really simple. But maybe these tests are helpful for experimenting with the current implementation.

[havocp]

fwiw first thing I'd do is make versions of those tests "minus magic" (remove Defaults._, import StaticOnly).

[phaller]

I just want to second @havocp. It's usually the best way to approach extending the macros.