[doc] Setting up PerlIO callbacks when embedding a Perl interpreter using C (before any `*.pm` modules were ever loaded or any Perl code executed)

[vadimkantorov]

Hi!

I managed to do a fully hermetic single-file static build of perl via building all modules statically (followed https://perldoc.perl.org/perlembed) and providing my own implementations of open/fopen/read/seek to serve *.pm system files from memory.

Is there a way to hook up to the Perl's own PerlIO layers system to make sure that Perl only calls these functions (including for module/*.pm discovery and loading) and never goes to libc's IO functions or does libc IO function calls / IO syscalls? This would be much cleaner and a more robust solution.

It would be nice if setting up PerlIO in perlembed scenario was covered in docs.

I also wonder how diamond operator is implemented in the code and which functions from https://github.com/Perl/perl5/blob/blead/perlio.c it calls and in what sequence (e.g. for perl -e 'open(f,"<","my.txt");print(<f>);' and for perl -e 'open(f,"<","my.txt");$line=<f>;print($line);')

Thanks!

---

If anyone's curious to see what my hack looks like - https://github.com/vadimkantorov/perlpack, but it's very much a WIP

My current problem is that overriding open /close / read / stat / lseek / access / fopen / fileno was sufficient for perl -e 'use Cwd;print(Cwd::cwd(),"\n");', so it can successfully discover and load the Cwd.pm file from my virtual read-only FS, but doing perl -e 'open(F,"<","/mnt/perlpack/.../Cwd.pm");print(<F>);' does not work - probably because Perl is trying to do fcntl/ioctl/some other version of stat call and I am not implementing these. In any case, it is currently not invoking the read function for some reason when I'm using the diamond operator because of some failures on the way. Which IO/stdio calls are used by Perl in a typical opening/reading a file? strace shows open -> fcntl -> ioctl -> lseek -> fstat -> mmap -> read, but these are raw syscalls, so I'm wondering what are the concrete libc/stdio IO functions (I imagine this is somewhere in perlio.c or do_io.c but there are quite a few of indirection layers - so hard to parse through by a novice in the perl's codebase) are used by Perl in a typical opening/reading a file (e.g. stat has many variants) - so that I can override them.

[tonycoz]

There is PERL_IMPLICIT_SYS, but that replaces all I/O (not just module loading) and only has a host implementation on Windows.

If you just want modules to be loaded from memory you can add a hook to @INC that checks for a known name and loads that module from memory, see perldoc -f require.

[vadimkantorov]

I'll check out PERL_IMPLICIT_SYS - replacing all I/O is fine for my usecase, as my custom I/O functions only serve from in-memory for some special prefixes like /mnt/perl. Is anywhere any docs / examples of using PERL_IMPLICIT_SYS to override? (and what functions need to be overridden for ensuring both module loading and for perl -e 'open(f,"<","my.txt");print(<f>);'. I'm only concerned for compiling/running on Linux for now.

If you just want modules to be loaded from memory you can add a hook to @INC that checks for a known name and loads that module from memory, see perldoc -f require.

Actually, interested both for modules and for regular, basic file reads. For modules, can such INC-hook be added via C perlembed interface (without executing Perl code)?

and only has a host implementation on Windows.

And regarding PerlIO infra, is it relevant for my usecase (module / *.pm loads and regular basic file reads)? Can it be configured via C perlembed interface? Or would you recommend using PERL_IMPLICIT_SYS? Or is using PERL_IMPLICIT_SYS on Linux impossible?

Thank you!

[Leont]

I'll check out PERL_IMPLICIT_SYS - replacing all I/O is fine for my usecase, as my custom I/O functions only serve from in-memory for some special prefixes like /mnt/perl. Is anywhere any docs / examples of using PERL_IMPLICIT_SYS to override? (and what functions need to be overridden for ensuring both module loading and for perl -e 'open(f,"<","my.txt");print();'. I'm only concerned for compiling/running on Linux for now.

It's only ever been done before for Windows but there's no reason it would be impossible on Linux. See perlhost.h, win32.c and perllib.c in win32/ for prior art.

[vadimkantorov]

Thanks for the pointers! I'll look into what entails using PERL_IMPLICIT_SYS on Linux.

And maybe the last question, if you would know if PerlIO can also be used for this I/O override goal? And if so, can it be configured via a C API before any Perl code gets executed?

[tonycoz]

For modules, can such INC-hook be added via C perlembed interface (without executing Perl code)?

After perl_construct() something like:

CV *hook = newXS("MyPackage::my_hook", \&xs_my_hook_xs, __FILE__);
AV *inc = get_av("INC");
av_unshift(inc, 1);
av_store(inc, 0, newRV_noinc(hook));

You could also define the hook sub in perl with eval_pv()/eval_sv().

And maybe the last question, if you would know if PerlIO can also be used for this I/O override goal? And if so, can it be configured via a C API before any Perl code gets executed?

You might be able to do it by modifying PL_def_layerlist or via PERLIO in the environment, but I've never tried it.

It also won't allow you to hook operations like stat() and fcntl().

[vadimkantorov]

You might be able to do it by modifying PL_def_layerlist or via PERLIO in the environment, but I've never tried it.

This would be interesting if this worked, and it could do for just *.pm embeds into the binary! It seems that stat() is not used on the *.pm search / loading path...

Basically, I was looking for a portable way to have hooks to embed and have loaded *.pm in fully static https://perldoc.perl.org/perlembed setup... I achieved this using static musl libc symbol renaming (https://github.com/vadimkantorov/perlpack), but having this using PerlIO hooks would be a nicer and a more portable way.

[vadimkantorov]

CV *hook = newXS("MyPackage::my_hook", \&xs_my_hook_xs, __FILE__);

Would be grateful if anyone could suggest an example how this void xs_my_hook_xs(pTHX_ CV* cv) {} C-land INC hook should look like. If I understand it should pop from cv the reference to itself and the requested file name an then somehow push a return value?

From https://perldoc.perl.org/functions/require:

INC hook will be called with two parameters, the first a reference to itself, and the second the name of the file to be included (e.g., Foo/Bar.pm). The subroutine should return either nothing or else a list of up to four values in the following order:

A reference to a scalar, containing any initial source code to prepend to the file or generator output.

A filehandle, from which the file will be read.

A reference to a subroutine. If there is no filehandle (previous item), then this subroutine is expected to generate one line of source code per call, writing the line into [$_](https://perldoc.perl.org/perlvar#%24_) and returning 1, then finally at end of file returning 0. If there is a filehandle, then the subroutine will be called to act as a simple source filter, with the line as read in [$_](https://perldoc.perl.org/perlvar#%24_). Again, return 1 for each valid line, and 0 after all lines have been returned. For historical reasons the subroutine will receive a meaningless argument (in fact always the numeric value zero) as $_[0].

Optional state for the subroutine. The state is passed in as $_[1].

[Leont]

I don't think overriding PerlIO is the way to go here. That is, unless you want the modules to be open()able as if they really existed.

Pushing your own handler (either a coderef or an object) into @INC is probably the best approach, in either case you need to add some code to core to handle that. An alternative approach might be to override the require opcode.

[vadimkantorov]

Yeah, I think figuring out both the avenue of installing a INC-hook from C land (for embeding modueles' pm/pl files) and the PerlIO avenue (for embedding data-files installed into the Perl prefix) would be very useful nice for making basic embedded, static Perls more practical, simple and transparent.

It also would allow more simple compilation of Perl to WASI target (which might not have the built-in virtual FS feature as opposed to Emscripten)

[Leont]

It also would allow more simple compilation of Perl to WASI target (which might not have the built-in virtual FS feature as opposed to Emscripten)

That really sounds like you should want to go the implicit sys route. That way you can virtualize all system/io interactions.

[Leont]

It probably would help a lot if we'd first write a generic implicit sys implementation, because you probably don't want to redefine all of it.