Three things

[philipturner]

[vojtamolda]

Hello @philipturner,

Thanks a lot for posting the message here! I'm glad that S4TF is being worked on again. As far as I know, work on the differentiation in the compiler itself actually never stopped. A lot of S4TF team members found a way to continue elsewhere. So to be honest, I'm not sure if I support the resurrection of the whole TensorFlow project. The XLA part of is a pretty scary aberration. Don't get me wrong, though, I'll be very happy if it happens, but I won't be able to invest any effort.

I personally think that Swift needs a multi-dimensional array type, aka the only the Tensor struct from TensorFlow. Ideally with hardware acceleration support where possible. It would serve the same role as numpy.ndarray in the Python ecosystem. Numerical libraries (like this one) could be built on top of it and would become compatible with each other. Right now people have to roll their own half-baked implementations and the situation is far from ideal.

1. Nice fix for the meantime!

2. Sorry for the broken link. The whole post needs to be updated anyway. I'll get to it eventually...

3. I don't have a problem with unarchiving the repo. Hope it helps at least little bit.

[vojtamolda]

I didn't mean the ShapeArray; I view it's purpose only as an escape hatch to bridge different Tensor-ish type implementations and convert data back and forth. I was talking about a single generally used and versatile implementation like the ndarray from Python.

The tensor-thumbnail repo was already unarchived when I posted my first comment. So I think we're done here.

[RahulBhalley]

@philipturner I doubt if Google will agree to unarchive the S4TF project. Have you had any conversations with the team upon unarchiving S4TF approval? I believe they're busy doing their own new jobs in Google and won't be able to work on this project anymore like handling PRs/issues/no design meetings and more. Also it's very very much possible (almost 100%) that the root-level access to swift-apis or any other repository will not be given.

I was talking about a single generally used and versatile implementation like the ndarray from Python.

I believe we should instead create a new differentiable Tensor library (with Metal support and later CUDA / XLA (that can be also adopted from the current state of S4TF but I think multi-TPU cores support was not made available)) using Swift development toolchain and distribute it as a package. Then build a deep learning library on top of it. This way, in future, the differentiable Tensor library can be used for building other packages too.

[vojtamolda]

Please, don't let a grumpy dude like me slow you down ;) I'm honestly really happy that you're doing what you're doing. It's just that I'm not big on social media or discussion forums. I don't feel like posting about un-archiving of a tiny library that has 0 users besides myself.

I agree with all @RahulBhalley wrote. The most useful project is to rip the Tensor type out of TensorFlow and revive it with a new Metal backend. And I don't think you need to involve Google in this.

The unique part about S4TF is differentiation of arbitrary code and structs. To me it allowed differentiable simulations which is something that was very hard to do before. If one wants to use S4TF just for machine learning there is just too many obstacles and sharp edges that in the end don't pay off. People are better off staying in the Python world for pure ML and it doesn't make sense to sail against this wind.

However, if you want to do specialized things, the differentiability built into the compiler is very unique. There's not that many other projects that allow you to do it.
So to me it makes sense to leverage this niche of differentiable numerical algorithms that isn't possible to replicate in JAX/PyTorch/TensorFlow. I believe this is where the next generation of numerical solvers gets created.

[vojtamolda]

One alternative idea to reviving the Tensor type I have is to use the ArrayFire library, wrap it in Swift and then write custom backward passes for all ops.

ArrayFire has support for multiple backends already. Maybe this avenue will yield better community support. People who use the library on macOS could benefit from the Metal backend as well.

@philipturner, What do you think about this idea?

[RahulBhalley]

The most useful project is to rip the Tensor type out of TensorFlow and revive it with a new Metal backend. And I don't think you need to involve Google in this.

This seems like a correct, simple, and viable approach to me. Google focused on Swift for TF only because the creator, Chris Lattner, of Swift was on-board actively guiding the project. Google doesn't care much about Metal tbh because their primary goal was to make S4TF CUDA & XLA-compatible and not Metal (even when M1 chip 2020 came out creating a lot of buzz). So, I think Metal cannot be a reason (it was Chris Lattner) for Google to continue S4TF project.

Given that the duration of time passed since S4TF was shutdown, JAX has taken its place. I'm not discouraging you & if this's something you really want to spend your efforts and time on the now then I'd say you should. But I think creating a new simple library will be a good task to take on.

I think there's a good chance I'll beat Torch's new Metal backend in performance, because I have much more experience with Metal than the people currently working on it.

And I think if you instead put your efforts in making LibTorch-Lite's Metal backend better, I believe people could benefit from it a lot because it's still in its beta stage and there are many bugs. One I encountered was moving tensor between Metal and CPU backend causing memory leaks. Another is that Metal support by LibTorch lacks for macOS platform. Just suggesting that this project can benefit from your Metal skills more than S4TF would.

[RahulBhalley]

Also, their Metal code unnecessarily uses Objective-C.

I wonder what would've been a better approach instead of Objective-C++ when interfacing C++ Tensor library ATen to Metal Performance Shaders?

The real problem is using Metal for training.

You can help them call backward() function on tensors in LibTorch-Lite as it doesn't work. Then allow updating parameters.