Python Support for Fable

[dbrattli]

Description

This issue is a discussion about Python support for Fable. A POC has been made to show that this is possible. However, quite a lot of work is needed to:

1. Add decent support for Python
2. Make Fable more JS agnostic and support multiple target languages.

Use cases:

• F# running in Jupyter notebook with Python kernel, i.e similar to what is done with e.g Hy and Calysto Hy.
• F# scripting using Python for reduced friction
• More easily support embedded environments like the micro:bit and Rasberry PI.
• Sharing data model between F# and Python code

Things to discuss and decide:

• Should Python for F# be more Pythonic than .NET? There is already .NET support for Jupyter which we should not try to replace or compete with. Fable for Python should perhaps target Python instead of .NET (if possible) and be a more natural fit for Python developers looking to F#.
• Should we try to be compatible with Babel AST or should we diverge into our own Python AST. This is perhaps inevitable, but Babel gives us a good starting point.
• Should we for Fable (as with Peeble) or should we stay within Fable and have Fable support multiple languages. Having our own repo gives freedom and speed (by not needing to care about JS), but we will risk being left behind and obsoleted over time. My feeling is that we should become part of Fable.
• Python basic types is also different from F#. E.g int has an arbitrary length. Should we emulate .NET (machine) types to be compatible or should we expose the Python int type as an F# int?

Installing the POC

The source code of the POC currently lives here:

• https://github.com/dbrattli/Fable/tree/python (Python support for Fable 🎉 #2345)

1. Install the latest Python 3.9 from https://www.python.org or brew install python@3.9 on Mac. Note that python may be available on your system as python, python3 or both.
2. Clone both repo and switch to the python branch.
3. Build fable-library for Python: dotnet fsi build.fsx library-py
4. To run tests: dotnet fsi build.fsx test-py
5. In Fable, edit QuickTest.fs to some simple F# code. Then run dotnet fsi build.fsx quicktest-py from the top directory.

Now you can edit both Fable and Expression code and the example code will recompile when needed. Note you need an additional save to QuickTest.fs to trigger recompile after modifying Fable code. The python file will be available as quicktest.py. A nice demo is to view both F#, JS, and Python in vscode at the same time. Then you will see how the code is being transformed when you save the F# file:

You can run the generated Python code in the terminal:

$ python3 quicktest.py

Links

A few relevant links:

• https://github.com/dbrattli/Fable/tree/python
• https://github.com/dbrattli/Fable.Jupyter
• https://github.com/cognitedata/Expression/tree/fable
• https://github.com/thinkbeforecoding/peeble
• https://docs.python.org/3/library/ast.html
• https://github.com/babel/babylon/blob/master/ast/spec.md
• https://fable.io/docs/dotnet/numbers.html

Feel free to contribute with discussions, comments, ideas, and code 😍

[alfonsogarciacaro]

This is great work @dbrattli! 👏 👏 👏 My original idea with Fable was to provide a way to easily compile F# to other languages. For better or worse, Fable has evolved with JS as its main focus so it would require some work to make it more language agnostic but I would be open to that. Some comments to your questions:

• Same or different repo? I would be in favor of keeping as much common code as possible. Using different repos will likely make them diverge quite fast (it's already difficult to keep in sync the branch with quotation support) but we also need to make the code modular enough so we don't have conditions everywhere depending on the target language. An intermediate solution would be to try to extract as much code as possible to a Fable "Nucleus" (we cannot use Core 😉) and then have different repos for the language implementations.

• Babel AST: This was used because originally we were relying on Babel for printing the code. In an ideal world, now that we have our printer it should be possible to remove it. But there's a lot of work happening in the Fable-to-Babel step so it would be difficult. For the same reason, it would be preferable to reuse as much code as possible from this step, maybe making the Babel AST something more generic for non-functional languages. Some examples:

  • Move from expression-based AST to expression/statement AST. For JS, this includes moving variable declarations to the top of the function when necessary to avoid too many IIFEs. I guess it will be something similar with Python.
  • Transforming pattern matching (DecisionTree) to if/switch statements.
  • Tail-call optimization. This is done using JS labelled loops. In Python we would need to use another technique to make sure we don't break from a nested loop.
  • Converting imports into unique identifiers (this could be done directly in the Fable AST if necessary).

• Make F# more Pythonic? With Fable I always tried to find a balance where .NET semantics is sacrificed sometimes to avoid overhead. For example we just use JS number instead of having a specific type for ints (although we do that for longs), this would apply to Python int I assume. However, many contributions have been done to respect .NET semantics at specific points to avoid unexpected results, for example in integer divisions or explicit numeric conversions.
  Anyways, when you say "more pythonic" you mean code style or native APIs? Fable doesn't try to support most of .NET BCL but at the end we need to support the common function/operators because this is what devs are used to (even newcomers, because that they would find in tutorials). It's good to give the option to use the native APIs in any case (Fable.Extras is a good move in that sense). For example JS.console.log usually formats objects more nicely than printfn, but we need to support the latter because it's what devs use the most.

[PawelStadnicki]

I love this idea. Especially I'm looking forward to use Python libraries in F#. Ideally we should have the option to consume either pure Python libraries or SciSharp - they tend to have almost exact API as Python (same F# code run in Python or .NET context!)

Such F# code can be compiled with Fable into Python and then used within Notebooks kernels (Jupyter or .NET Interactive) making analytics workflow being real domain-driven facility ( I could consider it all as a "killing feature").

.NET Interactive is very extensible, running Fable inside F# Kernel is very easy ( I have working PoC).

Another thing is that I predict that MS will reveal something similar quiet soon for entire .NET (the prediction is based only on activities and goals .NET Interactive tackle so I may be wrong). Even if they do we still want Fable to have it separately in order to have complete JS/F#/Python arsenal in a functional world. MS approach would be more like a Blazor-flavour ( I guess).

[dbrattli]

Thanks, this is great feedback @alfonsogarciacaro, @PawelStadnicki . What I meant was if we want to let the programmer get the "Python" feeling by exposing more of the Python standard library instead of trying to reimplement .NET (then they should probably just use .NET instead).

I guess we could have the best from both worlds, and have the user decide what to import or not (e.g use datetime from Python or DateTime from .NET). So no need to decide, but just implement what we want the most (on-demand). But I suspect the same problem is relevant for Fable JS. You know that you are implementing a web app, and probably don't expect all of .NET to be available. You would rather have better access to the JS ecosystem.

So no need to decide up-front. Let's roll up the sleeves and get going 😄 It will be great to be part of Fable and have this issue to ask questions along the way, e.g about transformations etc. There is much to learn.

PS: Expression btw also have a tail-call decorator we can use (both sync and async). Here's an example usage in aioreactive.

[jwosty]

Makes me think of #1601.

I wonder if there's a good way to make Fable powerful enough that these could be done as third-party backend "plugins" (think LLVM but specific for F#). Endless possibilities.

[alfonsogarciacaro]

We could try to make the Babel AST more generic (and typed) so it can be easily transformed/printed into C-like languages. However, if I've learned something while developing Fable is that writing a simple F#-to-another-language compiler is (more or less) easy, but providing a good development experience and integration so the advantages of F# minus the friction with the foreign ecosystem weigh more than developing in the native language is quite hard.

[ncave]

What @alfonsogarciacaro said. Part of that challenge is to migrate more of the Fable BCL implementation to F# so it doesn't have to be rewritten for every language, while maintaining acceptable native performance for the generated code, something which is non-trivial even with JavaScript.

[jwosty]

Interesting. I didn't realize that significant portions of the Fable BCL implementation were written in (presumably) Javascript. I assumed it would be more F# already. I don't doubt there's good reason for this -- why did it turn out to be easier that way?

[ncave]

@jwosty The (not so short) answer is: for performance reasons, and to bridge the difference between F# types vs types natively supported by the browser, and for better integration of the generated code within the JavaScript ecosystem.

Also .NET BCL has a very large API surface, on top of FSharp.Core, so it's hard to maintain this effort without a bigger team (although @alfonsogarciacaro somehow manages to maintain and evolve it mainly by himself, which is amazing and commendable).

I guess what I'm trying to say is, contributions (and ideas) are very welcome and appreciated.

[jwosty]

The (not so short) answer is: for performance reasons, and to bridge the difference between F# types vs types natively supported by the browser, and for better integration of the generated code within the JavaScript ecosystem.

I can see that.

I guess what I'm trying to say is, contributions (and ideas) are very welcome and appreciated.

For sure! I understand how much effort goes into maintaining projects as big and as important as this, and I really appreciate the blood sweat and tears you guys have poured into it. Definitely am keeping my eyes open for places to contribute, as Fable has lately been very very very useful for me and I'd love to give back.

[dbrattli]

I could use a little help. I'm currently using the QuickTest project for development (i.e dotnet fsi build.fsx quicktest). How do I print the Babel AST while developing? (Update: I found ASTViewer).

My current problem is that Python do not support multiline lambda (arrow functions) so the lambda need to be transformed and extracted to a named function and call that function instead. Any tips here would be appreciated. E.g:

let fn args cont  =
    cont args

let b = fn 10 (fun a ->
    a + 1
)

Will need to be transformed to:

let fn args cont  =
    cont args

let cont a =
    a + 1
let b = fn 10 cont

How should I think when replacing one call expression with multiple statements since when transforming the function call with args I need to replace it at a higher level, or use some kind of block statement e.g if true then ... Any ideas?

[ncave]

@dbrattli See also this Printer, which prints some of the properties too.

[alfonsogarciacaro]

I didn't realize that significant portions of the Fable BCL implementation were written in (presumably) Javascript. I assumed it would be more F# already. I'm don't doubt there's good reason for this -- why did it turn out to be easier that way?

Yes, as @ncave said, the main reason was to generate more standard JS. For example, in Funscript all the FSharp.Core replacements were written in F#, but with Fable I wanted to integrate with JS standards when possible (e.g. using JS iterators instead of .NET IEnumerable) which brought many chicken-and-egg problems so it was easier to write it in JS/Typescript at the beginning. At the beginning I also had the idea of publishing fable-library as a separate package that could be used in JS projects, but I quickly discarded it.

In Fable 2 we started an effort to port some of the modules to F#. This has been very beneficial because it increases dogfooding and let us easily reuse the improvements in FSharp.Core (as with maps and sets recently). However, there are still some hacks here and there to work around the chicken-and-egg issues. And let's not talk about the big monster that is the Replacements module :)

Also as @ncave says, maintaining this library is a huge undertake, that's why I'm always reluctant to increase support for BCL to reduce the maintenance surface, but I've been very lucky that there've been multiple great contributions to the library from the very beginning.

@dbrattli About printing the AST, it's been a while I haven't used the ASTViewer tool in Fable repo. Better options are the one pointed by @ncave or the Fantomas visualizer which is what I use recently when I need to inspect the F# AST (just make sure to select Show Typed AST). Unfortunately we don't have specific printers for Fable or Bable AST yet. In the case of Fable AST, as it's just unions and records a simple printfn "%A" works, although the location info is a bit noisy.

About extracting the lambdas. It's possible but requires some work. I guess we can do something similar to imports, that is, collecting them when traversing the AST, and assign them a unique identifier. Later we can just declare the functions at the top of the file. The main challenge will likely be the captured values, I don't remember if the F# AST has information about this, but even if it has it, unfortunately we don't keep it in the AST, so I guess we'll have to find all the idents used in the lambda body which doesn't correspond to arguments (or bindings within the lambda) and convert them to extra arguments of the lambda (and I guess they must go at the beginning to avoid interfering with currying).

[dbrattli]

Thanks for great info @alfonsogarciacaro, my first PoC was just altering Babel.fs / Fable2Babel.fs, BabelPrinter.fs etc. I've now starting from scratch with a separate Python AST i.e Python.fs, PythonPrinter.fs. Then I'll will try to add Babel2Python.fs to transform from Babel AST to Python AST as that may be easier than converting from Fable AST (but also possible with Fable2Python.fs if converting from Babel ends up being difficult). This way I will not touch the existing code base too much, and have my own transformation where I will try to do the lambda rewrite.

[alfonsogarciacaro]

That makes sense. One difficulty is Babel AST is not made of DUs so it's a bit more difficult to traverse it with pattern matching. Maybe #2158 could help?