[Biplate Macro #2] New Uniplate Derive Macro

[niklasdewally]

This PR adds the new Uniplate derive macro.

The reason for this rewrite was to add Biplates and "multiple-type traversals"; this is discussed further in this wiki article: https://github.com/conjure-cp/conjure-oxide/wiki/2024%E2%80%9003-Implementing-Uniplates-and-Biplates-with-Structure-Preserving-Trees

Implementation

The library syn supports the creation of custom AST nodes. Using the Parse trait, one can implement parsers for these custom nodes by combining existing Rust AST nodes together as well as parsing Rust tokens by hand.

The macro code has two parts: parsing into a custom AST, and using this AST for code-gen. This simplifies code-gen, as only the data needed for Uniplate is encoded in the AST. This also allows for better error reporting, as Most errors are now reported during parsing, allowing errors to be associated with the specific tokens that caused them.

The biggest change in my AST compared to Rust's is the representation of types. Uniplate has three kinds of type: boxed pointer types, plateable types, and unpalatable types.

Each plateable types contains some wrapper types and a base type. For example, the input type Vec<Vec<A> would be split into wrappers Vec,Vec, and a base type A. This is useful as we want to generally derive implementations for A only. Vectors are implicitly unwrapped using default implementations.

Boxed pointer types represent smart pointers such as Box and Rc. These use the .borrow() function instead of & to pass the inner value to the Biplate instance. Generating instances for Box<A> and Rc<A> would mean that values are constantly moved from stack to heap - this avoids this.

Testing

I added trybuild, a library for UI testing derive macros. The Rust tester does not run at-all when test code does not compile, but we need to test whether our macro gives a compile error when it should. trybuild also checks that the error messages given by the compiler have not changed.

Based on #301

[niklasdewally]

@ozgurakgun @gskorokhod I think this is ready for a preliminary review.

Known issues:

• My Biplate implementation that unwraps vectors is not working for nested vectors and I don't know why.
• The macro currently assumes that all types inside an enum contain each-other - I need a way to say that (for example) a String cannot contain an int. This would probably be in the form of a parameter l specifying which types are recursive.

These errors can be seen by removing the comments in the examples/stmt.rs file.

I've got some ideas for how to fix these, and some examples I could writeup to show specifically the underlying problems, but I don't have the time to do this just yet. A general code review would still be helpful though.

[ChrisJefferson]

There seems to be a bunch of compile issues, I tried just running cargo check to start, and got errors about private crates, and lots of errors which look like:

error[E0599]: no method named `with_children` found for reference `&Expression` in the current scope
   --> crates/conjure_core/src/rules/minion.rs:293:22
    |
293 |                 expr.with_children(sub)
    |                      ^^^^^^^^^^^^^ method not found in `&Expression`
    |
    = help: items from traits can only be used if the trait is implemented and in scope
    = note: the following trait defines an item `with_children`, perhaps you need to implement it:
            candidate #1: `Uniplate`

[niklasdewally]

The entire project doesn't compile as I haven't refactored conjure oxide to use the new types yet.

The uniplate crate specifically should though!

[niklasdewally]

RFC @gskorokhod @ozgurakgun @lixitrixi @ChrisJefferson

This is the blocker for this PR; any help would be appreciated! I don't really know how to explain it very well in writing (especially as I last looked at this indepth 2 weeks ago), but hopefully this gets the core of the issue across!.

---

To decide which Biplates to generate, each type found inside the given type is added to a list. A Biplate instance is generated from the given type to each type in this list.

#[derive(Uniplate)]
enum A {
  B(F,G),
  C(H),
  D(i32),

Deriving Uniplate on A will generate the trait instances: Uniplate for A, Biplate<F> for A,Biplate<G> for A,Biplate<H> for A, Biplate<i32> for A.

Under the hood, Biplates and Uniplates are just recursive invocations of Biplate that walk towards the target type. Currently, for each field in an enum variant, the generated code calls the Biplate that goes from that fields’ type to the target type (<FieldTy as Biplate<TargetTy>>::biplate(self)).

Consider the instance Biplate<G> for B. For the B variant, it will call <F as Biplate<G>> and <G as Biplate<G>>. Suppose F does not contain G. In that case, there will not exist a Biplate instance from F to G (as instances are only generated for inner types of F); therefore, Rust will throw a compile error.

Another issue is that it will try to call <i32 as Biplate<G>>. i32 is a primitive type we don’t control and due to trait orphaning we cannot paste an instance for Biplate for it inside the users code.

The main blocker for this PR is that we need a UI to decide what instances to generate. I’m a bit stuck on what the best way to go about this would be.

Also note that traits specialisation does not exist, and generic traits in rust do not yet have mutually exclusive bounds, so what we can do with blanket implementations is very limited.

[ozgurakgun]

picking this up part by part, for <F as Biplate<G>> and

Suppose F does not contain G. In that case, there will not exist a Biplate instance from F to G (as instances are only generated for inner types of F); therefore, Rust will throw a compile error.

Why not generate that instance in the trivial way?

[ozgurakgun]

Another issue is that it will try to call <i32 as Biplate<G>>. i32 is a primitive type we don’t control and due to trait orphaning we cannot paste an instance for Biplate for it inside the users code.

we might have to special case primitive types. I see we already have derive_unplateable, do these create orphan instances then?

[niklasdewally]

picking this up part by part, for <F as Biplate<G>> and

Suppose F does not contain G. In that case, there will not exist a Biplate instance from F to G (as instances are only generated for inner types of F); therefore, Rust will throw a compile error.

Why not generate that instance in the trivial way?

I'll give a more concrete example:

#[derive(Eq, PartialEq, Clone, Debug, Uniplate)]
enum Stmt {
    Assign(String, Expr),
    Sequence(Vec<Stmt>),
    If(Expr, Box<Stmt>, Box<Stmt>),
    While(Expr, Box<Stmt>),
}

#[derive(Eq, PartialEq, Clone, Debug, Uniplate)]
enum Expr {
    Add(Box<Expr>, Box<Expr>),
    Sub(Box<Expr>, Box<Expr>),
    Mul(Box<Expr>, Box<Expr>),
    Div(Box<Expr>, Box<Expr>),
    Val(i32),
    Var(String),
    Neg(Box<Expr2>),
}

#[derive(Eq, PartialEq, Clone, Debug, Uniplate)]
enum Expr2 {
  Modulo(Box<Expr>)
}

When derive runs on Stmt it generates the instances:

<Stmt as Biplate<String>>
<Stmt as Biplate<Expr>>
<Stmt as Biplate<Stmt>>
<Stmt as Uniplate>

When derive runs on Expr it generates the instances:

<Expr as Biplate<i32>>
<Expr as Biplate<String>>
<Expr as Biplate<Expr>>
<Expr as Uniplate>

Here the problematic case is <Expr as Biplate<Stmt>>, which would be called in the generated code for <Stmt as Biplate<Stmt>>.

The macro runs on each type in isolation, generating biplates from that type to all detected inner types. If we generate all instances from each type and to each type in Expr, this may solve this issue, but we could get duplicate conflicting implementations of Biplate for a pair of types. Also, this would mean that we cannot use the type information of the from type properly during derivation.

We have no way of checking during macro execution if an implementation is missing or is already present, so we must instead allow the user to specify what additional instances to generate.

In the Haskell direct derive the user specifies all instances to generate by hand, using the compiler to tell them which ones are missing. I think we can do better here.

An example stack project for this is here: (uses dependencies too old for Apple Silicon unfortunately)
https://github.com/niklasdewally/uniplate-bits/tree/main/hs-derive-orig

[niklasdewally]

There seems to be a bunch of compile issues, I tried just running cargo check to start, and got errors about private crates, and lots of errors which look like:

error[E0599]: no method named `with_children` found for reference `&Expression` in the current scope
   --> crates/conjure_core/src/rules/minion.rs:293:22
    |
293 |                 expr.with_children(sub)
    |                      ^^^^^^^^^^^^^ method not found in `&Expression`
    |
    = help: items from traits can only be used if the trait is implemented and in scope
    = note: the following trait defines an item `with_children`, perhaps you need to implement it:
            candidate #1: `Uniplate`

I think that I will make another PR moving Uniplate on Conjure Oxide to a manual implementation using the new types. This would help me iterate on the macro implementation without breaking Conjure Oxide as often.

[niklasdewally]

Another issue is that it will try to call <i32 as Biplate<G>>. i32 is a primitive type we don’t control and due to trait orphaning we cannot paste an instance for Biplate for it inside the users code.

we might have to special case primitive types. I see we already have derive_unplateable, do these create orphan instances then?

Thats right - it is just a simple substitution filling in some predefined terminal instances impl<T> Biplate<T> for $ty

[niklasdewally]

Still work in progress, but I have added some syntax thats specifies what instances to generate. It fixes all issues to do with missing trait implementations; however, it requires the user to manually specify which types should be walked into when looking for children.

use uniplate::Uniplate;

#[derive(Eq, PartialEq, Clone, Debug, Uniplate)]
#[uniplate()]
#[biplate(to=String,walk_into=(Expr))]
enum Stmt {
    Assign(String, Expr),
    //Sequence(Vec<Stmt>),
    If(Expr, Box<Stmt>, Box<Stmt>),
    While(Expr, Box<Stmt>),
}

#[derive(Eq, PartialEq, Clone, Debug, Uniplate)]
#[uniplate()]
#[biplate(to=String)]
enum Expr {
    Add(Box<Expr>, Box<Expr>),
    Sub(Box<Expr>, Box<Expr>),
    Mul(Box<Expr>, Box<Expr>),
    Div(Box<Expr>, Box<Expr>),
    Val(i32),
    Var(String),
    Neg(Box<Expr>),
}

Forgetting to add a type to walk_into could cause annoying bugs, so I think an opt-out mechanism (don't generate instances for these two types) may be better than this. However, this is significantly simpler to implement.

[niklasdewally]

Working and ready for review @ozgurakgun 🚀

I think the documentation and UI of the macro still need some work, but I think merging this as is would be best.

[niklasdewally]

cc: @lixitrixi

[ozgurakgun]

brilliant. I think I'd like a walk through before we merge though. tomorrow afternoon? I can do 3.30.