file_packager: Add suport for glob or loading file list from file

[KivApple]

Sometimes you might not want to pack all files from a specific folder, but only some. In this case glob pattern (there is a function in the Python standard library for doing this) would be useful like assets/**/*.png, but it is not supported by file_packager. There is a workaround - you can specify all files explicitly by hand or generate a command line using another script. But in this case you may hit command line length limit on Windows if you have too many files. In such case it would be useful to be able to read file list from text file (e.g. compilers allow you to pass options using text files). But such feature is also missing from file_packager.

[sbc100]

Both those features sound reasonable to me.

[debadree25]

If i understand correctly as per this issue we want to be able to do something like emcc file.cpp -o file.html --preload-file asset_dir/**/*.png?

[debadree25]

Another question I had at

emscripten/tools/file_packager.py

Line 431 in 03c61d5

srcpath = dstpath = arg.replace('@@', '@')

we are setting source and destination path from the arg but when we use globs what should be the dest and source path? like in case of asset_dir/**/*.png should asset_dir maybe be the dest path? @sbc100

[sbc100]

I think if we you want a different source and dest path then you use @ symbol in the argument so uses_at_notation would be set in that case.

I agree we would have to come up with a mapping from source the dest that might be a little more complex. Its not clear how it would work, but I think the @ sign should continue to be the signal that you want a different source and dest.

[debadree25]

Yeah, there's no issue when the @ character is there then no issue its when we don't have it, any ideas or inspirations you can think of in this case? I have no experience here so asking 😅

[sbc100]

When the @ char is not there, the user is saying that source and dest pathnames are the same. So shouldn't we just map every file in the glob that create corresponding file with the same name in the dest root.

[debadree25]

Ahh I think then if src is glob like we can recursively find all the files and append them as DataFile that can work i think

[sbc100]

How else would the glob work, if not by recursively finding all the file? You can use python glob or fnmatch modules to do the finding, right?

[debadree25]

yes yes was a little confused at first 😅😅, will attempt this

[KivApple]

I think @ can replace file path beggining until the first glob pattern part *.

E. g. asset_dir/**/*.png@dir would put all images under asset_dir into dir preserving nested directory structure.

If user needs more precise control over this mapping, he would specify multiple glob patterns.

[ChinYikMing]

Preload or embed necessary files inside a directory can reduce the size of WASM module substantially if there are multiple unnecessary files inside the same directory. In my case, I can reduce from size 62M to 23M.

I think I can help to solve this issue. May I be the assignee?

[sbc100]

By all means, feel free to submit PR for this.

[sbc100]

Having said that, cant these use cases also be solves using a 2-phase deployment setup:

1. Copy of the files you want to deploy into a "dist" directory.
2. Pass this "dist" directory to emscripten/file_packager.

[ChinYikMing]

Having said that, cant these use cases also be solves using a 2-phase deployment setup:

1. Copy of the files you want to deploy into a "dist" directory.
2. Pass this "dist" directory to emscripten/file_packager.

Seems workable! However, I think providing a general mapping option might easier for user (at least for me).

[sbc100]

I guess it depends on how much complexity this would add to file packager, vs how it is for users to do this copying/preparation step.

In the world of open source software its very common to have a make install type step that stages all the files necessary for a program. My current inclination is that this is reasonable thing for toolchain users to do themselves and that we adding more complexity to file packager should be avoided if we can.

[ChinYikMing]

In the world of open source software its very common to have a make install type step that stages all the files necessary for a program. My current inclination is that this is reasonable thing for toolchain users to do themselves and that we adding more complexity to file packager should be avoided if we can.

Agree.

How about mention 2-stages deployment to the document and close this issue? Otherwise, I think this issue will keep opening.