v0.5 "cannot add methods to an abstract type" when overriding call

[dlfivefifty]

I'm updating ApproxFun and this caught me by surprise:

abstract Foo
julia> (f::Foo)(x)=x
ERROR: cannot add methods to an abstract type
 in eval(::Module, ::Any) at ./boot.jl:267

Is this intentional? Is there a way to get around this other than define for each subtype:

julia> immutable CFoo <: Foo end
julia> (f::CFoo)(x)=x

[JeffBezanson]

Yes, unfortunately this is the only capability I could not figure out how to cleanly preserve in #13412. How bad is this for you?

[dlfivefifty]

Not bad actually (only 3 concrete subtypes), but thought I’d double check before working around it.

[vtjnash]

grepping the source, it looks like the (only) problematic definition was: https://github.com/ApproxFun/ApproxFun.jl/blob/cb062d6714412bd342cd7e4682ba7ff9f788b40a/src/Multivariate/Multivariate.jl#L7

[wbhart]

We are hitting this. What is excluded here?

Previously, to define a function for all types that are subtypes of a given abstract type, this would be precisely the syntax used. That seems like a rather fundamental part of Julia.

Or is it only when this syntax is used in combination with an object call overload that is excluded?

Losing that would be rather a blow to us. As you may know, we cannot store all the information we need to define mathematical rings in the type, so we create objects that stand in as types, e.g.

R, x = PolynomialRing(ZZ, "x")

where R is now an object that contains information about the ring of all polynomials in ZZ[x].

However, the mathematical abstraction of a polynomial ring is implemented via a bunch of Julia types, all of which are subtypes of some abstract type.

So it's natural to define methods for all of them at once by overloading the call syntax, e.g.

R() returns 0 in the given ring, or R(1) coerces the integer 1 into the ring.

Before panicking I'd like to understand the change a bit better, and what workarounds there are. But this seems to be a bit of a worrisome and unexpected change.

[KristofferC]

I thought call just changed syntax: 0778a89#diff-5cb043c25dee87c0787a9e1bbbf935baL17

[wbhart]

I think I understand what is happening now. Rather than someone replying in detail, perhaps I can ask if my understanding is correct. I might also take the opportunity to discuss a very related issue which we are hitting, in case it is useful to know about (see further below).

I think call is changing syntax and rather than only being defined for types, it is still defined for objects. The comment in the PR about deprecating call and putting method tables inside types instead refers to implementing that which is formerly known as "call" by putting method tables inside types. This doesn't mean call will be restricted to types, but that objects of a given type will have their call syntax implemented by putting method tables inside the corresponding types. The advantage will be faster anonymous methods. Is this correct?

The only thing disappearing is the former ability to define a method for a given class of types, i.e. all types which are subtypes of a given abstract type. Is this because "call" is now a method and not a generic function?

One place where this will definitely affect us badly is when we want to write a single method that overloads call for all elements of all polynomial rings. E.g. the user may create at runtime R = ZZ[x], S = R[y], T = S[z]. The user may then create polynomials f, g, h each in one of these rings R, S or T. Each of the polynomials f, g, h necessarily has a different type, though all the types belong to the abstract type PolyElem{T} (where T is the type of the coefficients).

We use call to enable things like f(123) to give the value of the polynomial at 123, or f(M) to substitute a matrix M into the polynomial f, etc. It has to be a generic function because the user could in theory substitute absolutely anything into f, g or h.

It's therefore a very serious breakage that we can't overload call for all of these at once using call(R::PolyElem{T}, a).

I would also be quite concerned if a similar thing happened for overloading the array syntax []. We currently write generic functions to overload [] for our objects R, S, T above so that for example U = R["x"] is possible. This creates a polynomial ring over R with variable "x". We use it similarly for matrices.

On a related note, I'd like to take the opportunity to discuss an issue which we hit which we have been surprised has not hit anyone else yet. We accept there is likely nothing that can be done about this, but I want to mention it here in case it helps with future design decisions.

Consider the example of matrices over Z/nZ for multiprecision integer n. The initial thought is to have objects of type Z/nZ. As you know, putting n as a parameter of the type is not really possible, nor desirable (otherwise it would trigger recompilation again and again in multimodular algorithms that used many different values of n).

Therefore, the only option is to have the n in an object, not in the type. Let's call the type in question modint. So objects of type modint will somehow have the residue r associated with them, but also in some way the modulus n.

But now consider Julia's generic algorithms for matrices over a given type. Many of these algorithms require one to be able to create a zero object to insert into the matrix. But the zero function in Julia only takes the type modint as a parameter.

The problem is, the type modint contains no information about n, so it is in fact not able to create the zero object of this type, since it cannot set n.

I'm actually really surprised no one has hit this issue anywhere else. For this reason it would be really useful if functions like zero and one that are used so pervasively in Julia matrix algorithms could take more information from somewhere.

Note that the n really can't be stored in the type itself, since then things would need recompiling for every n.

As things are, we are not able to use any of the Julia generic matrix functionality in Nemo.

The only solution that I can see is to explicitly support using objects in the place of types in Julia, so that things like zero(R) will be used by Julia when constructing arrays over "parents" [1] like R which aren't implemented as types but as objects, e.g. arrays over R = IntegersMod(7).

I mention this only because it directly relates to the call syntax and calling of objects as though they were types.

In fact, in computer algebra, the distinction becomes blurred. Consider ideals A and B of a ring R. We can certainly perform arithmetic operations on A and B such as A*B. So in one sense they behave like objects or values. But we can also think of ideals as sets of elements in the same way that ZZ or a polynomial ring is. So they can also act as "parents". For example, the element 0 of an ideal could be obtained by zero(A) or A().

Of course we can define zero(A) and A() no problems. But this isn't the way the Julia generic matrix algorithms generate a zero object. They will use zero(typeof(A)) which is not sufficient to construct the zero object of "type" (actually parent) A.

As I say, I'm just mentioning it here since it seems like an opportune moment to do so. I'm not expecting anything, just hoping that it might be of use in future planning.

[1] "parents" is the name computer algebraists give to the objects that represent mathematical types such as polynomial rings or Z/nZ that can't be fully modeled using actual types in the programming language of choice.

[tknopp]

Is there a workaround for this? I was so happy to see this happening in 0.4 and now its gone again. In Gtk.jl this would have allowed to remove a lot of macros that are currently used to create widgets. JuliaGraphics/Gtk.jl#134

[vtjnash]

Gtk doesn't need this. In particular, the limitation is that you can't call an instance of a widget, but you can still define constructors for abstract subclasses.

[JeffreySarnoff]

Many things not yet done in Julia need to add methods to an abstract type.

If this capability were to become absent, fundamental expressiveness would be curtailed.
It is already too hard to gather together multifaceted realization as an abstraction, and to project from intrinsically coherent concept through abstraction into specific entityhoods.

Moreover, it is a substantial boon to clear human communication about Julia code.

[JeffreySarnoff]

@JeffBezanson @wbhart @vtjnash
For example, it has been an essential strength of Julia that one could think and code:

The norm of a number that belongs to a division algebra is the square root of the product of that number with the conjugate of that number.

import Base: conj, *, sqrt

abstract DivisionAlgebraNum <: Number
norm{T<:DivisionAlgebraNum}(x::T) = sqrt( x * conj(x) )

and then define specific types, knowing norm(x) does 'just work'.

immutable QuaternionNum{T<:Real} <: DivisionAlgebraNumber
   r::T;   i::T;   j::T; k::T
end
conj{T<:Real}(x::QuaternionNum{T}) = ...
*{T<:Real}(x::QuaternionNum{T}, y::QuaternionNum{T}) = ...
sqrt{T<:Real}(x::QuaternionNum{T}) = ...
...

immutable OctonionNum{T<:Real} <: DivisionAlgebraNumber ...
immutable ComplexNum{T<:Real} <: DivisionAlgebraNumber ...
immutable RealNum{T<:Real} <: DivisionAlgebraNumber ...

[wbhart]

@JeffreySarnoff There's no evidence ordinary generic functions like your norm function will stop working is there?

[JeffBezanson]

Yes, definitions like that still work fine. The change only affects what used to be the first argument of call.

[JeffreySarnoff]

So does this mean that I would not run into this using abstract types and generics on them to code some category theory as an abstraction and then use that to define some basic categories as working types?

[wbhart]

I suspect you would definitely hit it if you tried to define functors using (the replacement of) the call syntax. This could happen if your functors were treated as objects that had properties, which as you know happens. This is actually a very good example.

[wbhart]

One other area where this functionality is really needed is homological algebra. I'd be surprised if it wasn't also extremely valuable to homotopy theory and to group actions in representation theory.