Concept syntax revision proposal

[fredrikhr]

Opening this issue in order to discuss strengths and weaknesses of the concept syntax and to discuss other suggested alternatives.

TL;DR

Thesis: The current concept declaration syntax does not conform to other syntax elements in the language and the write-how-it-can-be-used style seems rather alien.

Examples

Current syntax:

type CanMoveWithTick = concept x
  x.move(0.0)

@PMunch's initial question to this syntax:

Is there a nicer way to specify that move takes a float?

Yes, there is:

type CanMoveWithTick = concept x
  x.move(float)

but according to @Araq this is a special rule for concept matching.

Alternate proposals

@Araq: ref. (alternately also without the leading proc in front of move)

type CanMoveWithTick = concept x
  proc move(self; ticks: float)

@couven92:

type CanMoveWithTick[T] = concept
  move: proc(m: T, ticks: float)

Features of current syntax

Generally, the discussion aims for finding a syntax that is most like the existing Nim syntax while considering the features, the current syntax has:

• In the current syntax, expression like x.len can be matched by a proc, a field, or a template with a matching signature, in the matching case x.len simply needs to be valid
• Any statically resolvable boolean expression can be put into the body of a concept

Questions

1. How do we integrate the statically resolvable boolean expressions into the alternative proposals shown above? Is it reasonable to keep the current syntax for that even though it is slightly dissimilar to Nim syntax in related AST expressions (e.g. object declarations).
2. Can all statically resolvable boolean expressions be replaced by field and proc signatures as suggested by @Araq and @couven92?
3. Is it wise to be able to match x.something to both a field inside the matching type as well as against a proc taking the matching type as its first argument?
4. How do I explicitly require that something MUST be a proc? How do I require it to be a field?
5. How do I require matching fields/procs to fulfill pragmas?
6. Is the syntax compatible with 'vref/vptr' @Varriount

---

This discussion started on IRC between @Varriount, @Araq, @PMunch and @couven92 and on @Araq's suggestion I am now summarizing the discussion here, so that we get a better overview over the different opinions and to decide whether the concept syntax should be changed or not. I am also tagging @zah as I understand he is the author of the current concept syntax.

[fredrikhr]

In my proposal above, I have designed the concept syntax to be as similar as possible to an object declaration.

@PMunch pointed out to me: In cases like the Container concept example in the manual, I really don't care and don't want to think about and don't want to restrict whether len is matched by a proc or a field (or sth. else).

To adress this, I propose the following: (amending the example from the manual)

type Container[T, TItem] = concept
  len: expr(cont: T): int
  `[]`: expr(cont: T): TItem
  items: iterator(cont: T): TItem

In case of len in the example above, the use of expr in this case means: An expression (i.e. a command or call invocation) that features a ident AST node called len and a T instance as parameter.

[PMunch]

The new syntax is definitely an improvement in terms of making Nim syntax more unified, the old stuff looked a bit out of place in a type block (however I would love to see it turned into a macro as it was a neat idea to just loosely describe usage).

The only thing I don't see mentioned in this (apart from the questions section), is the current functionality of evaluating static expressions. A simple case can be found in the JS FFI where a concept is defined to be anything but a string. This is of course only one of the uses for this but I can't think of a way to add something like that to the proposed concepts.

[andreaferretti]

In my opinion, the write as it is used syntax is clear and simple to understand.

What I do not like is when it deviates from usage, e.g.:

type CanMoveWithTick = concept x
  x.move(float)

For this to work, I would expect move to take atypedesc[float]. This special casing for concepts makes for shorter definitions, but it makes the language unnecessarily complex in my opinion.

I would write that as

type CanMoveWithTick = concept x
  var f: float
  x.move(f)

On the other hand, this may match if move takes a var float, which is more restrictive than desired.

I cannot understand the syntax

type Container[T, TItem] = concept
  len: expr(cont: T): int
  `[]`: expr(cont: T): TItem
  items: iterator(cont: T): TItem

If I interpret correctly, seq[int] would be of type Container[seq[int], int]], whereas I would expect Container[int]. It seems that this syntax adds an unnecessary type parameter representing the current type being tested.

In short, I vote to keep the current syntax, and if possible remove concept-specific special cases

[zah]

I'm sorry that I'll provide only a brief answer, but I've discussed this at length so many times before.

1. Concepts are not only about listing required procs:

a) As a minimum, any alternative proposal should also cover how associated types (e.g. Graph.Edge) and constants are defined (e.g. T.zero).

b) Concepts can express other properties of the type:

For example, here is a concept that will be satisfied for all types deriving from a certain base type.

type DerivedFrom[T] = concept type D
   var derived: ref D
   var base: ref T = derived

Or a concept checking if a type is serializable by requiring that each field is serializable:

type Serializable = concept x
   for f in fields(x):
      f.serialize(Stream)

2. Concepts can express interfaces with implementation-defined types

If you are familiar with many generic C++ code-bases, you should have noticed how often the informally defined concepts in C++ feature "implementation-defined" types. This is some opaque value, returned from a given proc, that must be passed to another proc as part of the required usage protocol. I'm very happy with how my syntax deals with such situations:

type TextureManager = concept tm
  var textureHandle = tm.allocTexture(int, int)
  bindTexture(Shader, Shader.getParam("param_name"), textureHandle)
  tm.releaseTexture(textureHandle)

3. All complaints focus on how the syntax is unnatural, which is quite subjective

I personally find the conciseness of the current syntax preferable (especially when operators are used) and "natural" is really in the eye of the beholder (I can claim the current syntax reads much more like English for example). For anyone suggesting a new syntax, here is a simple challenge - please try to go though every example in the manual and see for yourself if the new syntax is making any of the definitions simpler or prettier.

4. The current syntax offers some interesting possibilities for the future

This is strictly a speculation at this point, but the current syntax may be extended to provide ways to define temporal requirements in the usage protocols (e.g. proc foo must be called after init is called) or control-flow based requirements (proc foo must be called after certain state is tested in an if statement).

---

So, I'm sorry, but I'll have to put this to rest. As long as nobody is willing to put the effort to study how concepts are implemented and nobody provides a comprehensive alternative solution that has equivalent capabilities, the current syntax is here to stay.

[PMunch]

I must agree with 1 and 2, they are certainly a concern as I voiced earlier (even though this takes that concern and shows actual examples of what was missing).

As for 3 I must say that the syntax couven92 proposed above certainly seems more natural along with the other type annotations. It's not really about looking natural linguistically, just that it looks a bit out of place.

I think the issue that we've discovered with the current concept declaration is that it's too loose. It's not obvious if you've covered all the cases you meant to cover, simply because you've described a use and not some limits. As seen in andreaferreti's example with the var f: float; x.move(f) this would be a valid concept but might not have been the intended concept as it won't match with let or const. The point is that the current concept syntax is really versatile and has many good characteristics, but it might prove too hard and confusing to use. Leading to bugs that might've been easier to spot or impossible to make unconsciously with a stricter syntax.

EDIT: Not trying to attack the current syntax by the way. I actually kind of like it, but it does raise some concerns that we wanted to at least discuss.

[fredrikhr]

@andreaferretti you are totally right about the two type variables in my example of course. And it is even in my proposal unnecessary:

type Container[T] = concept
  len: expr(cont: Container[T]): int
  `[]`: expr(cont: Container[T]): T
  items: iterator(cont: Container[T]): T

[andreaferretti]

Now you are expressing a different thing. You miss a way to name the particular type that you are matching.

Written this way, it looks like a recursive defintion (a type S matches Container[T] if len is defined over an object of type Container[T])

[fredrikhr]

@zah, @PMunch, What do we think about introducing interface as some sort of lightweight concept? Similar to how template vs macro are lightweight and heavyweight AST modificaction features...

I understand @zah that concepts are MUCH more than what interfaces give us in e.g. C#. However, I have a hard time grasping the whole range of functionality of concepts, and I'd like a more simplified intermediate.

Having interface as a go-between would just mean that we need to add something to the Nim syntax, and that concepts can remain as they are...

[PMunch]

I think the point is that we want to use types everywhere, and not mix and match types and values. In the example x.move(float) it looks like we have an instance of x and a type float whereas with CanMoveWithTick.move(float) shows that move simply takes two parameters, one of type CanMoveWithTick and one with type float. The rest of couven92's proposed changes are simply just shuffling around things to make everything more type-description-like (for better or worse).

But zah has a good point in that concepts are much more powerful than simple type descriptions. But I feel that there should be a harder requirement for specifying them to remove ambiguity. Kind of like how Haskell has the error "Non-exhaustive patterns in x" I feel that concepts should have a stricter requirement for their definition. This is something which, unfortunately, feels really hard to do with the current syntax..

[krux02]

I like the original syntax, because it is just Nim code and therefore easy to teach. A very important property. I don't really like the idea of yet another declarative concept declaration language. But there is something in the original example that is missing. The the Nim equivalent of the c++ feature declval. declval is a construct to create a type expression when you don't want to actually create a value of certain type, you just need an expression of a certain type. This can be very useful for concepts:

type CanMoveWithTick = concept x
  x.move(declval(float))

So there is no need to change any syntax, just add a very small language feature that provides all you need and might even be useful for other parts in generic programming.

type Container[T] = concept x
  x.len is int
  x[declval(int)] is T
  for v in x.items:
    v is T

[Varriount]

@zah I don't really have an opinion on what, if any, change is made to concept syntax. I do have some questions regarding semantics though.

Currently it appears that all symbols in a concept body are open - wouldn't it make more sense for a concept body to follow the same symbol binding rules as a generic procedure body?

[zah]

@couven92, with concepts, defining simple interfaces should be simple, while defining complex requirements should still be possible. With VTable types, the simpler interfaces can also be turned into a run-time construct similar to C#'s interfaces. Because of this nice, gradual interplay between dynamic and static polymorphism, I don't think we need any additional intermediate constructs in the mix. If you fancy some particular alternative definition style, you can always resort to macros to create your own syntax.

Everyone contributing to this thread should take the time to read through the concepts spec in the manual, because I still see some basic misconceptions and questions that are already answered there:
https://github.com/nim-lang/Nim/blob/devel/doc/manual/generics.txt#L201

@andreaferretti, @krux02, I'm well aware of the possibility to use something like declval as an alternative to the current type-handling rule, but after spending months on the design and looking at many examples, I'm convinced that the current shortcut is worth it. I think your concerns are mostly based on a hunch (gut feeling) and after some more experience, you'll come to appreciate the rule as well. @andreaferretti, please note that without the type handling rule, introducing something like declval is necessary, because you have to be able to express a requirement like x.write(var Stream).

@PMunch, I'd like to see some specific examples of problems you have ran into:

I think the issue that we've discovered with the current concept declaration is that it's too loose

@Varriount, yes, handling the concept body in the same way as a generic proc body is a planned bugfix.

[krux02]

@zah Well I have a very bad gut feeling for the shortcut. I haven't digged into it too much, but what I am worried aboit is that cases where you actually want to pass the typedescriptor, not a value of that to the procedure

import typetraits

type
  MyTypeMapA = object
  MyTypeMapB = object

template map(a: typedesc[MyTypeMapA]; b: typedesc[int]): untyped = float
template map(a: typedesc[MyTypeMapA]; b: typedesc[float]): untyped = int
template map(a: typedesc[MyTypeMapB]; b: typedesc[int]): untyped = string
template map(a: typedesc[MyTypeMapB]; b: typedesc[float]): untyped = string

var a : MyTypeMapA.map(int)
var b : MyTypeMapA.map(float)
var c : MyTypeMapB.map(int)
var d : MyTypeMapB.map(float)

echo a.type.name # float
echo b.type.name # int
echo c.type.name # string
echo d.type.name # string

type
  MyValueMapA = object
  MyValueMapB = object

proc map(a: MyValueMapA; b: int):    float = float(b)
proc map(a: MyValueMapA; b: float):    int = int(b)
proc map(a: MyValueMapA; b: int):   string = $b
proc map(a: MyValueMapA; b: float): string = $b

The shortcut feels to me that it introduces ambiguety that is not necessary. I can be wrong because I reall haven't looked too deep into the specification, but that is my biggest concern here. How would the type map and the value map be distinctable in a concept?

[zah]

As I said, everybody should take the time to read the manual:
https://github.com/nim-lang/Nim/blob/devel/doc/manual/generics.txt#L250

In order to check for symbols accepting typedesc params, you must prefix
the type with an explicit type modifier. The named instance of the type,
following the concept keyword is also considered an explicit typedesc
value that will be matched only as a type.

It's best to see the examples in the manual, but to understand the paragraph above, you must also read the following:

The identifiers following the concept keyword represent instances of the
currently matched type. You can apply any of the standard type modifiers such
as var, ref, ptr and static to denote a more specific type of
instance. You can also apply the type modifier to create a named instance of
the type itself:

[krux02]

I well I guess then it works. But that doesn't mean that I need to like it. I don't like implicit type to value conversions. There are a lot of places in the Nim language that have to do with type and typedesc that let me cringe. But because I cannot really name what exactly goes wrong here, nor how to do it better without actually digging into the system (too lazy for that) and breaking a lot of stuff, I will accept it as it is for now.

[Araq]

For this to work, I would expect move to take atypedesc[float]. This special casing for concepts makes for shorter definitions, but it makes the language unnecessarily complex in my opinion.

I agree with this. It removes much of the elegance of the approach just to be able to avoid writing foo(witness(T)).

Syntax is also not entirely a matter of taste, for a syntax like proc op(x, y: int): bool it is clear that op is an open symbol and int is not whereas for op(int, int) is bool it is less clear.

But I can live with the existing syntax, but I still cannot accept "let's ignore typedesc in a concept to save some typing".

[krux02]

Well that just supports my gut feeling.

[evacchi]

this seems to works fine and requires no syntax change. Am I asserting something incorrect ?

type CanMoveWithTick = concept x,f
  x.move(f.float)

[Araq]

This RFC was succeeded by #168