Proposal: implement "operators" on tuples

[ianthehenry]

The Janet "operators" -- the functions that can be implemented as methods, like + and bxor and friends -- are not defined on tuples or arrays. And (as far as I know), there is no way for a Janet user to implement them on tuples or arrays.

I would like to propose that Janet implement them with "vector space" semantics, such that adding two tuples zips them together:

repl:1:> (+ [1 2 3] [4 5 6])
(5 7 9)

While adding an indexed and a non-indexed value applies the operation elementwise:

repl:2:> (+ 1 [1 2 3])
(2 3 4)

(And so on for * and / and friends.)

Upsides

A lot of what I do with Janet involves vectors, so I'm biased here, but this would be very convenient for working with Raylib or, theoretically, art playgrounds.

I have always overridden the default functions with custom versions that have these semantics, but there's some overhead to doing the extra typecheck that can add up in realtime animation wasm stuff.

I think that this is just nice behavior, but I usually work in a particular niche, so I wanted to see if this idea was compelling to a wider audience.

Downsides

At the moment these expressions can only error:

repl:1:> (+ 1 [1 2 3])
error: could not find method :+ for 1, or :r+ for <tuple 0x6000012082A0>
  in _thunk [repl] (tailcall) on line 1, column 1

But this expression in general might not:

repl:1:> (+ x [1 2 3])

If x implements a custom :+. Which is fine -- that will continue to work exactly as it does before. But this will change:

repl:1:> (+ [1 2 3] x)

Expressions that rely on :+ not being implemented for tuples will change. Previously this would take the custom :r+:

repl:1:> (+ [1 2] {:r+ (fn [a b] ["adding to" b])})
("adding to" (1 2))

But after this proposal, it will use the normal + defined on the tuple, and invoke :r+ for each element:

repl:1:> (+ [1 2] {:r+ (fn [a b] a)})
(("adding to" 1) ("adding to" 2))

This feels very low risk to me because I don't think it's very common to overload operators.

Questions

What if the operands are different lengths?

There are, I think, four reasonable options:

1. Error:

   repl:1:> (+ [1 2] [4 5 6])
   oh no
   

2. Take the longest common prefix:

   repl:1:> (+ [1 2] [4 5 6])
   (5 7)
   

3. Pad the shorter inputs with the identity element:

   repl:1:> (+ [1 2] [4 5 6])
   (5 7 6)
   

   (i.e., the same as (+ [1 2 0] [4 5 6]).)

4. Always return a result with the same length as the first indexed argument, choosing between (2) and (3):

   repl:1:> (+ [1 2] [4 5 6])
   (5 7)
   repl:2:> (+ [1 2 3] [4 5])
   (5 7 3)
   

I think that any of these are fine, but I would vote for (3). If you think of "adding" a two-dimensional vector and a three-dimensional vector, I think it makes the most sense to return a three-dimensional vector. And if you have, say, an [R G B A] vector, sometimes you want to manipulate the color components without touching the alpha channel.

Also maybe worth saying, but (3) is tractable because I'm not suggesting a general tuple-function-zipping thing, but only to define this for + and * and the like, where we can always pick the correct identity element.

That said, it's easy enough to pad a vector out -- (+ rgba [;rgb 0]) -- so none of the choices make a big difference.

Variadic?

Yeah of course:

repl:1:> (* [1 1 2] 10 [1 2 3])
(10 20 60)

What if the operands are different types?

I think the only reasonable answer is to return the first tuple type encountered. So:

repl:1:> (+ 1 @[4 5 6])
@[5 6 7]
repl:2:> (+ [1 2 3] @[4 5 6])
(5 7 9)

What about nested tuples?

I don't think there should be any special-casing. The rules of "adding exactly two indexed structures together zips, everything else maps" still applies correctly with nested tuples:

(+ [[1 2]] [3]) = [(+ [1 2] 3)] = [4 5]

What about other operations?

I think it's probably a bad idea to apply this to any function that is not already overloadable in the way that the operators are. i.e. I am not proposing that (math/pow [1 2 3] 2) should square each element of the "vector." I mean, it might be neat, but that's a separate proposal.

Do all abstract types / tables that override operators have to think about this?

Yes, if they want to apply that operator elementwise when they get a tuple as the second argument.

Wait so should abstract types have some way to opt into "I am indexable" so that if you add a tuple with your logically indexed abstract type it doesn't resort to elementwise addition?

That is to say:

(+ [1 2] my-custom-vector-type)

Would result in:

[(+ 1 my-custom-vector-type) (+ 2 my-custom-vector-type)]

When you could reasonably wish for it to be:

[(+ 1 (in my-custom-vector-type 0)) (+ 2 (in my-custom-vector-type 1))]

But this does not seem worth the complexity to support.

[CosmicToast]

And (as far as I know), there is no way for a Janet user to implement them on tuples or arrays.

There absolutely is, as janet has closures.

(defn myplus [& xs]
  (if (all indexed? xs)
    (print "hello from myplus")
    (+ ;xs)))
(def + myplus)
(print (+ 2 3))
(+ [1] [2])

You can capture + only to later redefine it.
You could also do a (def + (fn [&xs] ... for the same effect.

[ianthehenry]

Yeah, this is what I meant by:

I have always overridden the default functions with custom versions that have these semantics

But shadowing the function is not the same as implementing the method. Library functions that use + don’t use the custom functions that I define — I also have to redefine e.g. +=, ++, dec, product, etc. when I do this.

[CFiggers]

I can see the value of defining custom operator behaviors on a per-project basis. However, my instinct would be that defining language-wide defaults for things like + on arrays and tuples would be a mistake.

I obviously don't have hard data to back this up, but I would expect that the majority of the time, in the majority of simple programs, passing a data structure directly to + comes about because of a mistake, not a well-reasoned and intentional move. Consider programmers who are relatively new to Janet, or relatively new to thinking in Lisp, or even relatively new to programming in general. The majority of the time, I would expect, applying + to a tuple indicates an error on the part of the programmer.

If that is the case, then most of the time having + applied to arrays/tuples error out is actually the desired behavior. Having it silently succeed, but with completely different semantics than + usually has, would make programs tolerant of bugs introduced elsewhere in the program.

But we don't want to increase bug tolerance in the language—whenever possible, we want obvious bugs to cause errors, and cause them as close to the origin as possible, so the programmer is alerted to them early and can go understand and then fix them.

Having a way on a per-project basis to define custom operator behavior seems reasonable, and also like an effortful way of expressing "I know what I'm doing, so SUDO let me add tuples together." But as I stand right now, I would oppose defining any particular project-convenient behavior at the language level.

[ianthehenry]

Yeah! I think it's the semiring thing originally. It's nice to use the math symbols because you can use reason about PEGs using your existing intuition about numbers -- e.g. you know that (* A (+ B C)) is the same as (+ (* A B) (* A C)), because multiplication distributes. Except that this multiplication is not commutative. And, slightly confusingly, 0 in PEGs represents the mathematical "one" -- the multiplicative identity -- while there isn't really a short way to write mathematical zero (the PEG that never matches).

I think that's like the roundabout original reason for the convention of + meaning "or" and * meaning "and," which Janet follows. I've always understood the PEG operators by analogy with "sum" and "product" as the words are used in e.g. type systems -- the "sum" of A and B is the variant "A or B," and the "product" of A and B is the tuple "A and B." Which is a more direct analog than like integers, but maybe not widely used terms.

[sogaiu]

Thanks for those bits :)

Perhaps my brain will have an easier time working with * going forward!

[pepe]

I started with the hell yeah (as always :-D), but I also nodded my head on the @CFiggers arguments. Yesterday, before falling asleep had the idea about vector operators being a build-time option, which would be turned off by default. Would it work for you @ianthehenry?

[ianthehenry]

Having thought about it more, I think that making the behavior customizable is a bit worse than not having it at all.

I don't think there are any build-time options right now that meaningfully change runtime behavior, mostly just remove behavior, and it's kinda sad to say "this library is only compatible with Janets built with these flags." I think at that point it's effectively just a fork of the language -- I think of the build-time options as mostly useful for embedding and running on constrained platforms, not really for creating new dialects of Janet.

And allowing you to change the behavior of these operators at runtime reminds me of all of the problems of JavaScript's globally mutable prototypes -- you wouldn't be able to take advantage of vector addition in any library code, because then loading that library would subtly change the behavior of other code, and that sort of action at a distance is terrible.

I still think it would be net positive to be able to add and scale tuples easily, but I think that probably a better way to do this is to write a common library with sort of type-checking macro that allows me to statically overload the meaning of these operators without any runtime cost. It's not as nice as having it built-in to the language -- i.e. if the default operators worked this way, some other operations would work "for free" over vectors -- e.g. (mean [[1 2] [3 4]]) would give you [2 3] using the current definition in the stdlib instead of erroring. But I don't really think that's worth forking Janet into a separate Janet/J hybrid, if the consensus is against making Janet more APLish.

[pepe]

It makes sense. Thanks for the answer. With this in mind I am more open to the idea of including the operators in the standard Janet build.

[primo-ppcg]

The Janet "operators" -- the functions that can be implemented as methods, like + and bxor and friends -- are not defined on tuples or arrays. And (as far as I know), there is no way for a Janet user to implement them on tuples or arrays.

I like the idea, in principle. Adding two vectors adds the vectors, multiplying a vector by a scalar scales the vector. It makes sense, from a mathematical perspective.

However, this behavior can already be achieved with the current syntax, relatively concisely:

(map + [1 2 3] [4 5 6])
# ->
@[5 7 9]

The second use case is not so concise:

(map (partial * 10) [1 2 3])
# ->
@[10 20 30]

which I concede is somewhat agonizing to look at.

I think there's a few alternative options:

1. Create a function called vec, similar to map, that applies an operator to its arguments exactly as you describe. Reasonably, I think this could be added to the core language.
2. Create an abstract type, likely called vector, that implements arithmetic operators on itself and other indexed types. Possibly a good candidate for spork.
3. Make partial invocations less awkward (I've already attempted this). short-fn / |() can be used as an alternative, but not any nicer to look at, in my opinion.

[primo-ppcg]

I'd have to test to be sure, but the tuple creation appears to be the bottleneck: #1163 (comment)

I suspect that a C implementation would result in a noticeable speed up for each and :in, and a very noticeable speed up for eachp and :pairs.

EDIT: I'm wrong. The only use case that becomes faster is (eachp p ds), and the speed up is very minor. The rest become slower.

[primo-ppcg]

This is a somewhat inefficient but correct implmentation of vec:
Much more efficient:

(defn vec
  [op & inds]
  (var len 0)
  (def ninds (length inds))
  (def iters (array/new-filled ninds))
  (eachp [k ind] inds
    (unless (= nil (get ind 0))
      (set len (max len (length ind)))
      (put iters k true)))
  (if (> len 0)
    (do
      (def res @[])
      (def call-buffer (array/new-filled ninds))
      (forv i 0 len
        (eachp [k ind] inds
          (put call-buffer k (if (in iters k) (or (get ind i) (op)) ind)))
        (array/push res (vec op ;call-buffer)))
      res)
    (op ;inds)))

Usage:

(vec + [[1 2] 3] 4 [5 [6 7 8] 9])
# ->
@[@[10 11] @[13 14 15] 13]

[primo-ppcg]

@ianthehenry could you take a look at #1240, and see if it's sufficient?

[ianthehenry]

I don't really know what sufficient means in this context. That seems like a useful operator! I like that as a way to generalize arbitrary math functions, and it seems like a good addition to the standard library. (I think the fact that it always returns an array makes it less useful than it could be, but I get it.)

But the original proposal (which seems clearly not supported) is not really about power, but notation. Looking at some examples from the last Jaylib project I was playing with:

(defn get-mouse-local [camera]
  (- (/ mouse-position frame-scale)
     (* frame-size 0.5)))

(defn get-mouse-local [camera]
  (vec - (vec / mouse-position frame-scale)
     (vec * frame-size 0.5)))

That's not bad -- pure expressions, still pretty readable. My brain definitely reads the first one more easily, but I bet I could get used to it.

(+= (enemy :pos) (* (enemy :dir) (enemy :speed) time-elapsed))

(set (enemy :pos) (vec + (enemy :pos) (vec * (enemy :dir) (enemy :speed) time-elapsed)))

That's not a very nice substitute, but another macro thrown in could improve things:

(vec= + (enemy :pos) (vec * (enemy :dir) (enemy :speed) time-elapsed))

Which is basically fine, though my brain has more trouble with that.

I don't think that this looks bad, but I think that even with it in the standard library I'd still continue shadowing the built-ins because the overloaded versions just read better to me (if that's what you meant by the question).

For context this proposal originally came out of me turning these shadowed functions that I've copied around all my Jaylib toys into a real library, and realizing that it would be nicer if I didn't need library for this at all. But it seems, by all accounts, that I still will :)

[primo-ppcg]

I don't really know what sufficient means in this context.

I was asking mostly if there's a use case that isn't met. Allowing macros to be vectorized (e.g. (vec += (enemy :pos) ...)) is something that I think would be really useful. I'll see if I can work it in.

[subsetpark]

You might find it useful to look at a library I’ve written that embeds the J language into Janet: https://jnj.li/ J is an array language, and thus is particularly suited to the kind of array operations being discussed here, that is: operations which are a) declarative over any dimensionality of array ; b) more efficient than vanilla Janet when it comes to nested data structures.

…

On Aug 5, 2023 at 1:57 PM, <primo-ppcg ***@***.***)> wrote: This is a somewhat inefficient but correct implmentation of vec: (defn vec [op & inds] (def iters (map indexed? inds)) (if (any? iters) (do (def res @[]) (def ninds (length inds)) (def iter-keys (array/new-filled ninds)) (def call-buffer (array/new ninds)) (while (any? iters) (forv i 0 ninds (def ind (in inds i)) (if (indexed? ind) (if (in iters i) (if-let [key (next ind (in iter-keys i))] (array/push call-buffer (in ind (set (iter-keys i) key))) (put iters i false))) (array/push call-buffer ind))) (if (any? iters) (array/push res (vec op ;call-buffer))) (array/clear call-buffer)) res) (op ;inds))) Usage: (vec + [[1 2] 3] 4 [5 [6 7 8] 9]) # -> @***@***.*** 11] @[13 14 15] 13] — Reply to this email directly, view it on GitHub (#1227 (reply in thread)), or unsubscribe (https://github.com/notifications/unsubscribe-auth/AAAXT7NKIIZ7WUMQ34F5ALDXT2CPNANCNFSM6AAAAAA2PMPHFY). You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[ianthehenry]

I've seen that! I haven't found occasion to use it yet, but I think it's a very cool idea. (I have a passing interest in array programming but have not had occasion to do anything serious with it yet.)

It's kinda in the opposite direction of this proposal, though -- like, really this is about notational convenience, not really power or performance. I want to write expressions like (+= pos (* direction speed)) for 2D and 3D vectors, while also keeping the concise square bracket notation, and have that just Do What I Mean.

[greenfork]

I wonder what the opinion is on moving this to a separate C library that implements vector operations. Here is a rough API surface:

• v+, v-, v/, v*, v+=, v-=, v/=, v*= -- vector arithmetics
• v+n, v-n, v/n, vn, v+n=, v-n=, v/n=, vn=, n+v, n-v, n/v, nv, n+v=, n-v=, n/v=, nv= - vector and scalar, accept only 2 params
• v=, v>, v>=, v<, v<= -- vector comparison
• vbnot, vblshift, vbrshift, ... -- bitwise operations
• vmin, vmax, vneg, vsqrt, vsqure, vabs, ... -- element-wise functions
• dot, distance, cross, l-norm, p-norm, ... -- vector-specific math functions

So some examples:

(v+n [1 2 3] 1)

(v+= pos (v* direction speed))

(v+= (enemy :pos) (v* (enemy :dir) (enemy :speed) time-elapsed))

(defn get-mouse-local [camera]
  (v- (v/ mouse-position frame-scale)
     (v* frame-size 0.5)))

Several whys:

• Why new operator -- some languages treat single/multiple values differently, e.g. Perl/Raku uses variable name $var for single values and variable name @arr for multiple, Elvish does the same; Factor has prefixed with v operations for vector math (like in my comment, mostly copied from Factor btw).
• Why not overload + -- the surface of core utilities to implement is relatively small, why not just do it.
• Why write another letter -- we can optimize for reading instead of writing, seeing vector and vector+scalar operations clearly can be worth writing an extra letter.
• Why v+n and n+v instead of polymorphic v+ that works on both vectors and vector+scalar -- avoid dynamic dispatch to save time, vector operations tend to require optimized implementation.
• Why not implement an abstract data type vector that overloads operations like + -- I don't know, could be a viable alternative too. Ruby does that (with Vector class), Zig does that (Vector type). Can be paired with SIMD representation and instructions on them. Maybe if we want a high-performance vector operations, we should plainly abandon standard types as too slow?

My main question would be, what is the necessity to include it into core that can not be implemented outside? Convenience is one thing, is there anything else?

[pepe]

Something like https://neanderthal.uncomplicate.org/ <3.

[primo-ppcg]

v+, v-, v/, v*

These are equivalent to (map + ...), (map - ...), etc. This is roughly what would be happening under the hood anyway, and I don't think that readability is improved any.

v+n, v-n, v/n, v*n

I suppose these would make more sense for lanuages in which arithmetic operators are strictly dyadic. The equivalent for these is admittedly somewhat verbose, though. Something like (map |(+ $ n) v).

I think a function which vectorizes an operator would probably be a better idea. It encompasses all of the proposed operators (with exception to the mutators, and some vector specific functions), they remain variadic, you can mix scalars and vectors as you please, and could be used to vectorize any function, rather than a handful of operators.

[greenfork]

I personally would be more comfortable working with functions like v+ because they are shorter than map and more intuitive for me than vec. Avoiding dynamic dispatch could be a good way to avoid performance bottlenecks. Of course you could provide your own vision for your library with the API you like.

[primo-ppcg]

Avoiding dynamic dispatch could be a good way to avoid performance bottlenecks.

This is certainly true. Between 1.29 and 1.31 I made a number performance improvements to various functions by doing more or less precisely that. In many cases though, I think these could be compiled differently to the same effect. For example, (+ ;a) could/should compile to roughly the same byte code as (sum a), rather pushing the arguments to the stack and invoking +.

I don't really see a huge difference between ((vec +) n a b) and (v+ n a b) in terms of useability or readability. If implemented directly in C, the second could potentially be much faster. Although, assuming the optimization mentioned above is made at some point, I don't believe the difference would be all that great.