3.7 import issues with webpack

[doconix]

(btw, the new 3.7 structure rocks. It is much prettier to put the runtime in a separate file.)

I'm using the new 3.7 with webpack using py-loader. I'm modifying py-loader for the 3.7 , but there is one issue that I'm not sure how to solve.

We need to solve it "correctly" so py-loader and transcrypt work together for all users on webpack. I don't want to pretend that my setup is super important compared to others, but I would argue that making transcrypt work really well within webpack is very important to its wider adoption. Webpack makes transcrypt seamless because we can import the .py file directly (see below) instead of the transpiled .js file. All transpiling is totally transparent.

Let me see if I can explain the issue clearly. Suppose I import a transcrypt file in a JS file called main.js:

import { something } from './module.py'

When webpack sees this import, it uses py-loader to call transcrypt. This results in the following:

./main.js
./module.py
./__target__/module.js
./__target__/org.transcrypt.__runtime__.js

webpack doesn't use the compiled module.js from the disk. It instead pipes the contents of module.js from py-loader directly. The current directory is still ./, not the subdirectory. This worked fine in the old Transcrypt because everything was in one file.

However, since the runtime is in a separate file (and imported into module.js), webpack can't find the runtime file. To be clear, webpack needs this directory structure:

./main.js
./module.py
./module.js
./org.transcrypt.__runtime__.js

(the module.js file actually doesn't matter because it gets streamed).

Possible solutions I can think of:

1. Include the runtime manually in my webpack config somehow. The issue here is the current directory may not be the right place to import the runtime vars from. That line is hard coded into transcrypt.
2. Make the __target__ subdirectory name be configurable. That way I can have all the files output to the current directory (i.e. no subdir).
3. Have py-loader copy the runtime to the current directory. This is a bit of a hack but will work.

#1 above is the best solution, IMO, because it allows me to include the runtime once in my entire project and then reference the single runtime js file from all the modules. I'm not sure how to do this, though, because the modules could be in any directory of my project, and ES6 imports have to be relative.

Ideas?

[JdeH]

Thanks for your concise explanation, I think I get the essence of it.

You mention 3 possibilities, of which possibility 2 seems the most simple and effective. The only problem with it is cleaning up, especially in case of multiple applications in one directory. Since they may not be compiled simultaneously, the timestamps of the files cannot be used as deletion criterion. Moreover, with sourcemaps the sourcefiles will be copied to the target directory as well, even of "standard" modules. I'd like to keep it that way, because it's simple and robust. This means that making the target dir equal to the source dir will lead to overwriting the original source file, the editor wanting to update etc., some things I don't like. They are very hackish, also if py_loader rather than Transcrypt would do this copying. Configuring the targetdir to coincide with the sourcedir has the same problems.

So what to do?

If I think about it "backward", then I'd say py-loader + webpack want the source file to be in the same directory as the target files. But they don't care which directory that is. It may be <projectdir> but also <projectdir>/__targetdir__. In my view the real source directory is sacred. Nothing should be written there, except by the developer personally. Actually the whole idea of having __target__ is that it's a "trash can". Target code can be made by a computer, hence from a developer point of view it's low value, throwing it away is OK , no need for bulky backups etc.. Actually the same design considerations that lead to introducing Python's __pycache__ dir.

The source directory should be minimalistic, lean and mean. No machine generated garbage there, just the fruits of hard work of the developer. But of course there isn't any objection to having a duplicate of the source files somewhere else, most notably in the target directory. The only thing is Transcrypt can't put them there, because Transcrypt is the one that has to be triggered by py-loader + webpack in the first place.

So I arrive at the following solution: When working on a project, you work in a source dir and its subdirs. The files in the subdirs are submodules, so py-loader/webpack doesn't care about them, since Transcrypt will draw them into the transpilation automagically.

But the source files of the apps, one or more per webpage are of interest to py-loader/webpack. Webpack wants to find them in the same place as the generated files. This can be achieved by a simple polling application, watching the source dir for changes. Whenever a file changes (because it has been edited by the dev, no other reason generally), the polling application will copy it to the targetdir.
So py-loader/webpack will always find everything in the targetdir and have the illusion that it is the source dir as well. I propose we call this polling application ts_collect.py.

So all ts_collect does is watching files in <projectdir> and copy them to <projectdir>/__target__ if anything changes to them. Later on ts_collect.py could be made configurable, enabling it not only to do as described, but also copy anything in <projectdir>/__target__ to anywhere else, e.g. to dynamically merge projects with separate source dirs. What I like about this solution is it's non-intrusiveness, separation of concerns, flexibility and "stupid" lack of complexity.

There will be some interaction if the -b switch is used, but my intuition says that's solved relatively easily.

What do you think? (Honestly...)

[doconix]

I've been thinking about this all night. After reading your response, I very much agree that the source dir should be sacred. Only the dev can make changes there. Good call there. I do something similar on my DMP project where I have a __dmpcache__ directory.

I'm not sure I agree on the proposal though. I think you may be thinking in the traditional web sense--that we're linking to the output of transcrypt (the .js). One of the core ideas of webpack's loaders is we link to the source file (the .py). Then we forget about the transpiling part. Using webpack, we can essentially mix and match js, transcrypt, coffeescript, typescript, etc. and import these files interchangeably. It's really slick and hides the messy details. We can do this all in the same JS file:

import { something } from './somewhere.py'
import { something2 } from './somewhere2.js'
import { something3 } from './somewhere3.ts'
import { something4 } from './somewhere4.coffee'
...

Webpack interprets the JS file, sees the different imports, and transpiles everything to JS as needed. Mixing a bunch of languages is obviously a bad idea. But doing just JS and .py together seems to be the exact goal of transcrypt. It becomes seamless.

Moving on to a new proposal... With the runtime js file, we essentially have a dependency. We only need that dependency once per project. In the node/npm world, we would npm install transcrypt-runtime or something. It would download the runtime to the node_modules directory. Then it becomes available for import throughout the project -- without relative directories.

This last paragraph is really the "right" npm way. My opinion is that the npm repository is the future of JS modules. Personally, I really dislike the wild west feel of the npm repo, but it just seems like all the movement in the JS world is toward npm going forward. This point is really the driver for me rather than a technical reason -- JS devs are going to want to use npm install for the dependency.

To do this, we need some way (yet another switch? :) to tell transcrypt to use the following import:

import {TRANSCRYPT_VERSION, AssertionError, AttributeError, ...} from 'transcrypt_runtime';

The only sticky point is you'd need some way to check that a version mismatch between module and runtime doesn't exist. It would be common to install the runtime at one version and use transcrypt at another because one is in npm and one is in pip. I added the version constant above so the module can check right after import, but that's just one idea for controlling it.

It also means you'd have to publish the runtime to the npm repository. I've done this and am happy to help if you need it.

My $0.02. Cheers.

[doconix]

P.S. I'm not a fan of webpack or npm or JS in general. That's why I'm drawn to something like transcrypt. Webpack has tons of options and was difficult to wrap my head around when I started using it. Perhaps I am projecting here, but I wonder if many who come to the transcrypt project might feel the same.

I'm not webpack expert, but I'm happy to put together a tutorial on installing and using transcrypt/webpack once the 3.7 changes settle down. But whoever writes it, I think that needs to be prominently featured on your web site. IMHO, It's the future of JS. Your 3.7 changes make transcrypt ready to be a first-order language like typescript and coffeescript because it can fully plug in to this "modern" way of doing JS.

[JdeH]

Hi, I'd like to exchange some ideas with you on this.
I've sent you my skype ID.
If you have the opportunity, could you contact me?

Kind regards
Jacques

[doconix]

Will continue on PM.

[doconix]

Jacques --

I've been playing with the compiler quite a bit, but I keep crashing it because of ripple effects. I'm wondering if you can help or at least provide some direction. Essentially, I need this:

    def revisit_Import (self, node):  # Import ... can only import modules
        for index, alias in enumerate(node.names):
            if alias.asname:
                self.emit ('import * as {} from \'{}\';\n', self.filterId (alias.asname), alias.name)
            else:
                self.emit ('import * as {} from \'{}\';\n', alias.name, alias.name)
            if index < len (names) - 1:
                self.emit (';\n')

When we revisit the import statement, I need it to simply emit a new import statement and not recurse and compile the imported module (webpack will do this). self.useModule is what actually does the compiling (so I'm skipping it), but it also modifies the data structures. Since the above code isn't doing that, it crashes.

The same thing needs to be done with revisit_ImportFrom, but I can look at that later.

If you are able, would you mind creating a branch (off of modules) and helping me see what needs to be done for the above code? It obviously breaks the current way of doing things, but once it's working the way I need, we can figure out how to make both work.

If you are swamped with other features/bugs, any direction you could provide would be great

Thanks.

[JdeH]

Hi Conan, unfortunately I am not able to assist you rightnow. I’ll try to get back to you in about 2 weeks. KR Jacques

…

On 28 Apr 2018, at 03:09, Conan Albrecht ***@***.***> wrote: Jacques -- I've been playing with the compiler quite a bit, but I keep crashing it because of ripple effects. I'm wondering if you can help or at least provide some direction. Essentially, I need this: def revisit_Import (self, node): # Import ... can only import modules for index, alias in enumerate(node.names): if alias.asname: self.emit ('import * as {} from \'{}\';\n', self.filterId (alias.asname), alias.name) else: self.emit ('import * as {} from \'{}\';\n', alias.name, alias.name) if index < len (names) - 1: self.emit (';\n') When we revisit the import statement, I need it to simply emit a new import statement and not recurse and compile the imported module (webpack will do this). self.useModule is what actually does the compiling (so I'm skipping it), but it also modifies the data structures. Since the above code isn't doing that, it crashes. The same thing needs to be done with revisit_ImportFrom, but I can look at that later. If you are able, would you mind creating a branch (off of modules) and helping me see what needs to be done for the above code? It obviously breaks the current way of doing things, but once it's working the way I need, we can figure out how to make both work. If you are swamped with other features/bugs, any direction you could provide would be great Thanks. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.

[ly-cultureiq]

Another way of solving this seems to be if transcrypt would accept input from stdin and output the results to stdout. In that way you can set the compiler streaming to true in py-loader and have no built js files lying around.
I looked around for such an option and couldn't find it

[jpic]

@JdeH would you approve the above suggestion by @ly-cultureiq ?

Note: reproduced in a minimal repo here: https://github.com/jpic/test-transcrypt-webpack