Requesting assistance

[philipturner]

From @compnerd’s comment in #1184

The project doesn't seem particularly enticing to me - the "rebuilding from the ground up for GPU acceleration through Metal" is a big component of that. If you were designing it for optimization for DCOM+ (DirectX is effective accessed through COM), that might be different. GPU environments are quite fragmented and designing around a particular one is a choice that I do not particularly agree with.

One reason I started this project is I didn’t like that S4TF ran exclusively on CUDA. I planned to bring Metal to it, but I’m open to more platforms. I started trying Swift on Windows, although I couldn't get the REPL to work (even with the workaround @compnerd recently posted). I am interesting in learning DirectX and making a backend for DirectML. I would author it in C++, then bind it to Swift through C at call sites for raw operations.

This also seems very much as other platforms are an afterthought. The overall structure of S4TF is far better from a design standpoint - it generically solves the problem in software and has abstractions over a compilation pipeline to separate the concern of optimization and device/runtime independence.

I read the research paper on S4TF, and I realized that the eager execution model of TensorFlow was much more achievable than what I had planned for DL4S. I then considered using an industry-standard interface like XLA for interfacing with both the Metal and DirectX backends. That investigation led me to greatly appreciate S4TF’s public API. I would rather reuse the code base of S4TF than overhaul DL4S. I have been changing my mind left and right over the past week, so my decision isn’t concrete.

I am considering forking tensorflow/swift-apis and adding the new backends to that project. In addition, I will add iOS/tvOS support by removing the need for a separate Swift toolchain - one of the major reasons @palle-k made DL4S. Differentiation is disabled in Swift’s release build, but I found a short-term workaround by wrapping the Swift standard library’s differentiation directory in a Swift package. Ideally, S4TF 0.14.0 would still work with tensorflow/swift-models and fastai/swiftai without any modifications to those repos. The contributions would be purely additive.

[philipturner]

From the Apache 2.0 license, I am allowed to do this as long as I properly document my modifications. There are also precedents of Apple and Microsoft making re-implementations of TensorFlow for MLCompute and DirectML. I am fully aware that I do not own the TensorFlow trademark, and will find a solution that respects Google’s right to the trademark.

Assuming I stay on track with my current plans, I would prefer to have my changes merged into the main branch and get S4TF un-archived. I re-read the last Open Design Review meeting presentation and observed a strange pattern of cancellations leading up to it. I infer that the S4TF team did not give up because they lost funding, but because they got burnt out.

However, Google has a precedent of donating projects they gave up on to an open community - Google Cardboard. A similar thing also happened with a Swift framework, Kitura (although not from Google). There are numerous software developers who would like to see S4TF revived, including @RahulBhalley.

As an alternative, I could maintain a fork of Swift for TensorFlow, which would be the community-hosted continuation. I would ask for one final PR to tensorflow/swift, adding my resurrection to the list of spin-off projects. I assume Google is unwilling to revive the main S4TF repository, so I would ask you to validate my commits to the fork instead.

Ideally, I would like to work with the former S4TF team and have their assistance. I expect to spend 1000 hours on this project, but the end product may be less professional and reach a narrower audience without your endorsement. In addition, I may accidentally break CUDA support since I can’t run the rigorous validation tests configured on tensorflow/swift-apis.

TL;DR - Please let me know your stance on me contributing to Swift for TensorFlow.

[ProfFan]

Hi Philip,

Just a few cents on my end. I definitely would like to see this project resurrected, but there are a few challenges that I can think of.

Biggest one is the toolchain. Autodiff is in Swift mainline, and there is already a lot of adoption (mainly in Linux world), but a big caveat here is that it hasn't been finialized via Swift Evolution, if I remember correctly. That means at any given time, autodiff may get removed. I am very willing to vote (and even work in weekends) for it, and I think a lot of other people do, but it has to be done. Since @rxwei is the most reliable source on this end, please forgive me if my comment above is wrong.

The last thread on evolution is https://forums.swift.org/t/differentiable-programming-for-gradient-based-machine-learning/42147/96

After the adoption of RequirementMachine, quite a few bugs have been detected in the autodiff code, which further increased the urgency of fixes and improvements. For example,

• https://bugs.swift.org/browse/SR-13263
• [AST][AutoDiff] Provide Lit test case for AutoDiff GenericTypeParamDecl AST verification failure. swiftlang/swift#38745 (comment)
• https://bugs.swift.org/browse/SR-13166
• [AutoDiff] [TF-1288] Supporting differentiable functions with multiple semantic results swiftlang/swift#38781 (comment)

These are the reasons why the PR you have seen is still pending. Actually in debug mode the unit tests already all pass, but in Release mode the compiler crashes...

I really appreciate your work in DL4S, and as you have already observed, the architecture of swift-apis is so elegant that most of the code is generic: it's almost like the only work needed for another backend is changing Tensor... sans making the compiler work.

Fan

[philipturner]

I anticipate that my work won't be completed for up to a year. Would that be enough time for the Swift Standard Library to support differentiable without importing my package?

https://github.com/philipturner/Differentiation

If the S4TF team is up for at least considering the resurrection of this project, then I'll put my time toward getting differentiation stable in the Swift standard library. It should get fixed entirely within at most a few weeks.

[ProfFan]

@philipturner I am just a fellow user (our package SwiftFusion builds on S4TF). Your package looks great! If you can make a demo of this working in iOS it may attract a lot of people in the Swift Forums.

AFAIK, I am fairly confident that most of the old team is still willing to resurrect this project, just they all now have now jobs and cannot devote time. We need to work out most of the issues ourselves so that the project gets momentum again.

[philipturner]

How about I do the most time-consuming work of hunting down the source of bugs. That means you could still work it out yourselves, sans the time cost. The reason I have time is because I’m not yet in college (a year left though, hence the time constraint). Thus, I have an unusual capacity to contribute compared to you all.

…

Sent from my iPhone

On Dec 2, 2021, at 8:54 PM, Fan Jiang ***@***.***> wrote:  @philipturner I am just a fellow user (our package SwiftFusion builds on S4TF). Your package looks great! If you can make a demo of this working in iOS it may attract a lot of people in the Swift Forums. AFAIK, I am fairly confident that most of the old team is still willing to resurrect this project, just they all now have now jobs and cannot devote time. We need to work out most of the issues ourselves so that the project gets momentum again. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[philipturner]

I might be able to pull off an ARHeadsetKit demo of differentiation by tomorrow or the day after. Would it be appropriate to post a low-resolution video or gif on Swift forums?

I may have misunderstood what you were considering I make a demo of. Differentiation or S4TF?

[ProfFan]

Oh I mean differentiation (some application that is enabled by differentiation)

[philipturner]

Epic. And would videos/gifs be appropriate for Swift forums? I think this is the perfect opportunity since it will give a visual impression and be downloadable (as an Xcode project) and interactive.

[philipturner]

With Swift 5.6, we could transfer the Colab notebooks to something similar to Apple's SwiftUI tutorials. I made numerous ARHeadsetKit tutorials in DocC and they're very effective. The content should transfer over easily, and could be hosted on a GitHub Pages website.

A discussion about this is already happening on Swift Forums: https://forums.swift.org/t/support-hosting-docc-archives-in-static-hosting-environments/53572

[philipturner]

@ProfFan the demo has been posted on Swift Forums, although it got caught in the spam filter. We will need to wait a bit before the post goes public.

• Repository: https://github.com/philipturner/differentiation-ios-demo
• Swift Forums post: https://forums.swift.org/t/swift-for-tensorflow-resurrection-differentiation-running-on-ios/53841/1

[philipturner]

The S4TF toolchain was broken by concurrency in Swift 5.5. I can't use it in Xcode 13.

Also, the following tutorial is too far off from the current state of differentiation to be salvageable in DocC:
https://github.com/tensorflow/swift/blob/main/docs/site/tutorials/custom_differentiation.ipynb

After digesting all the information in the Swift Differentiable Implementation Programming Overview, I'm going to convert it to Markdown, which was requested at the top of the document. This should help me with solving the compiler bugs you are currently facing. In addition, anyone in the public can dismiss a comment in the Google Doc, which is not good. That won't happen with GitHub comments.

I'll try as hard as I can to preserve it in its original form so we can discuss and agree on changes. Therefore, it would be better if I uploaded the Markdown file to a repository you owned, so I couldn't make official changes without your validation. @compnerd @rxwei @BradLarson would a branch of tensorflow/swift-apis work for this purpose? Preferrably main or a new branch created from it. We would then discuss modifications in a pull request to the branch.

This PR would also let me fill out the Google contribution agreement that @ProfFan did when making the PR to this repo.

[philipturner]

Would work on S4TF be okay if we just get VJP working (and not JVP, linear maps, transposition)?

[BradLarson]

It's great to hear about the interest in carrying on work around differentiable Swift and the Swift for TensorFlow APIs / models.

Focusing specifically on what it would take to build tensorflow/swift-apis once again (a first step to getting things operational again), @ProfFan's PR #1184 brings tensorflow/swift-apis up to the current state of Swift syntax. It is currently blocked by two compiler assertion failures that are being worked on. These don't yet have Jira tickets because I haven't yet arrived at a single-file reproducer for them, but they both are connected to differentiation through functions with inout values and control flow (if statements, for loops, etc.). Those will both need to be resolved to unblock that PR, and we also have local code that's impacted by these assertion failures.

A new and I believe related assertion failure just started appearing with the 2021-12-04 nightly snapshots, and that I was able to reduce down to a reproducer in SR-15566. I'm guessing that will also cause problems with the above-linked PR and will need to be resolved as well. I'm also looking into this, which is easier with the reproducer.

I think all of the autodiff issues that the new Requirement Machine has exposed have since been resolved, so we're good on those right now. All of the current issues are clustered around functions with control flow, and those involving inouts in particular.

Regarding reverse- and forward-mode differentiation, right now only reverse-mode is enabled and within the scope of the initial implementation of differentiable Swift. That should be perfectly fine for almost all of the applications you'd have in mind for frameworks leveraging accelerators and automatic differentiation, so while it would be nice to have the other capabilities I think we'll be just fine with reverse-mode for now.