Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add LoadKernelExtension, IsKernelExtensionAvailable #4231

Merged

Conversation

fingolfin
Copy link
Member

@fingolfin fingolfin commented Jan 27, 2021

The idea is to provide "better" (?) APIs for packages to load kernel extensions. With IsKernelExtensionAvailable (and an upcoming kernel function to make it better), packages can simplify their TestAvailability functions, and at the same time make them more robust. Likewise, LoadKernelExtension simplifies init.g code.

The big win is that it'll then make it much easier for us to change how to locate and load kernel extensions; e.g. to better support make install of packages, or also better Julia integration.

Open questions:

  1. perhaps I should rename these functions to LoadKernelModule and IsKernelModuleAvailable? Background: in the manual and various other places, we talk about "kernel modules". I (and some other places/people) use "kernel extension" (possibly a short hand for "kernel extension module") to distinguish it from built-in kernel modules; I guess one could also refer to it as "external kernel module" (but that would not be quite right for "static" kernel extension modules...).
  2. In case of failure, perhaps IsKernelExtensionAvailable resp. IS_LOADABLE_DYN should return proper error codes (instead of just false) to indicate what exactly went wrong (file missing / file not a loadable module / file not a GAP kernel extension / kernel extension but built for wrong GAP version / ....). It could even be just a string; in any case, something that helps produce better error messages for AvailabilityTest in PackageInfo.g
  3. Perhaps add an optional field KernelModule to the package info record, which when present would prompt GAP to automatically invoke IsKernelExtensionAvailable when checking the package availability; and LoadKernelExtension when loading the package (possible values: KernelModule := true = use package name, KernelModule := "modname"; or even KernelModule := ["mod1", "mod2"] if someone wants to support multiple kernel modules in one package. Alas, that seems very fringe, so I'd probably not do it)

@fingolfin fingolfin added do not merge PRs which are not yet ready to be merged (e.g. submitted for discussion, or test results) topic: kernel labels Jan 27, 2021
lib/files.gd Outdated Show resolved Hide resolved
lib/files.gd Outdated Show resolved Hide resolved
Copy link
Contributor

@ThomasBreuer ThomasBreuer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice idea. See my comments.

lib/package.gi Outdated

# also add a new mechanism to integrate kernel extensions into a package:
# let the package declare in its PackageInfo.g that it has a kernel extension;
# how to build it; what's its name; then GAP can take care of that dynamically
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can the intended new entries in PackageInfo.g also replace the prerequisites.sh files used byBuildPackages.sh?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hopefully / perhaps -- it depends a bit on what the package does in there...

@fingolfin fingolfin force-pushed the mh/LoadPackageKernelExtension branch 2 times, most recently from ea7194d to a0ac0f1 Compare January 27, 2021 23:52
@fingolfin fingolfin added the kind: discussion discussions, questions, requests for comments, and so on label Feb 17, 2021
@fingolfin fingolfin force-pushed the mh/LoadPackageKernelExtension branch from a0ac0f1 to 5fff8c2 Compare May 19, 2021 13:35
@codecov
Copy link

codecov bot commented May 19, 2021

Codecov Report

Merging #4231 (89e4961) into master (1dc1f81) will decrease coverage by 82.30%.
The diff coverage is n/a.

❗ Current head 89e4961 differs from pull request most recent head 5fff8c2. Consider uploading reports for the commit 5fff8c2 to get more accurate results

@@            Coverage Diff             @@
##           master   #4231       +/-   ##
==========================================
- Coverage   82.30%       0   -82.31%     
==========================================
  Files         678       0      -678     
  Lines      287790       0   -287790     
==========================================
- Hits       236876       0   -236876     
+ Misses      50914       0    -50914     
Impacted Files Coverage Δ
src/modules.h
lib/cyclotom.gd
lib/helpdef.gi
lib/clas.gd
lib/cmdledit.g
lib/grpffmat.gd
lib/grppcfp.gi
lib/stbcbckt.gd
lib/ieee754.g
src/debug.c
... and 665 more

@fingolfin fingolfin changed the title WIP LoadPackageKernelExtension, IsPackageKernelExtensionAvailable LoadKernelExtension, IsKernelExtensionAvailable May 19, 2021
@fingolfin fingolfin added the topic: packages issues or PRs related to package handling, or specific to a package (for packages w/o issue tracker) label May 19, 2021
@fingolfin fingolfin marked this pull request as ready for review May 19, 2021 14:32
@fingolfin
Copy link
Member Author

I've overhauled this now, and IMHO it is mostly ready, except for the lack of documentation.

Going beyond this, I wonder if we should add an optional field KernelExtension to the package info record. If that is present, then GAP could automatically invoke IsKernelExtensionAvailable when checking the package availability; and LoadKernelExtension when loading it. That said, perhaps this is too fringe?

@fingolfin fingolfin force-pushed the mh/LoadPackageKernelExtension branch from 5fff8c2 to d69a1f9 Compare May 19, 2021 15:33
@fingolfin fingolfin marked this pull request as draft May 19, 2021 15:33
@ChrisJefferson
Copy link
Contributor

I generally like this PR. I have one (larger) comment -- I wonder if we should separate static and dynamic modules more?

Is there any evidence we can still (or demand for?) loading extensions as GAP static modules would work? The built-in ones (where we compile some library files with gac) are not handled in the "standard way", READ_GAP_ROOT calls LookupStaticModule. Therefore we could, perhaps, simplify this code some of the general loading code by removing mentions of static modules, and (even) rename 'static' to 'library' or something.

Of course, if people want the option in future of doing a big static build, where we static compile extensions, that wouldn't work.

@fingolfin
Copy link
Member Author

@ChrisJefferson I am not sure what to make of your comment. It seems orthogonal to the content of this PR? If at all, this PR would help with "getting rid and/or renaming static modules": it removes the need to explicitly call into the APIs for static modules, by providing a fresh abstraction layer. So once we have this PR in, future packages could switch to using LoadKernelExtension and IsKernelExtensionAvailable. Then the package authors would not have to think about "static modules" anymore, even if they still wanted to support those.

The sooner we have this in, the sooner packages could start using it. Conversely, even if we get this into 4.12, it'll take time for this to be adopted by packages.

@fingolfin
Copy link
Member Author

I notice that in the manual and various other places, we talk about "kernel modules". I (and some other places/people) use "kernel extension" (possibly a short hand for "kernel extension module") to distinguish it from built-in kernel modules; I guess one could also refer to it as "external kernel module" (but that would not be quite right for "static" kernel extension modules...).

Aaaanyway: perhaps I should rename these functions then to LoadKernelModule and IsKernelModuleAvailable ?

@fingolfin
Copy link
Member Author

I also wonder if IsKernelExtensionAvailable resp. IS_LOADABLE_DYN should perhaps produce a proper error code (instead of just returning false) to indicate what exactly went wrong (file missing / file not a loadable module / file not a GAP kernel extension / kernel extension but built for wrong GAP version / ....). It could even be just a string; in any case, something that helps produce better error messages for AvailabilityTest in PackageInfo.g

@fingolfin fingolfin force-pushed the mh/LoadPackageKernelExtension branch from d69a1f9 to 7f2573f Compare May 26, 2021 13:59
@fingolfin fingolfin marked this pull request as ready for review January 19, 2022 21:41
@fingolfin fingolfin force-pushed the mh/LoadPackageKernelExtension branch from 7f2573f to 0682df8 Compare January 19, 2022 21:42
@fingolfin
Copy link
Member Author

Perhaps we should just merge this? There are plenty more things one could do, but this is already an improvement...

@fingolfin fingolfin added this to the GAP 4.12.0 milestone Jan 19, 2022
@fingolfin fingolfin removed the do not merge PRs which are not yet ready to be merged (e.g. submitted for discussion, or test results) label Jan 19, 2022
@fingolfin fingolfin added the release notes: to be added PRs introducing changes that should be (but have not yet been) mentioned in the release notes label Jan 20, 2022
@fingolfin fingolfin merged commit 28cd96c into gap-system:master Jan 22, 2022
@fingolfin fingolfin deleted the mh/LoadPackageKernelExtension branch January 22, 2022 23:05
@fingolfin fingolfin changed the title LoadKernelExtension, IsKernelExtensionAvailable Add LoadKernelExtension, IsKernelExtensionAvailable Aug 17, 2022
@fingolfin fingolfin added release notes: use title For PRs: the title of this PR is suitable for direct use in the release notes kind: enhancement Label for issues suggesting enhancements; and for pull requests implementing enhancements and removed release notes: to be added PRs introducing changes that should be (but have not yet been) mentioned in the release notes labels Aug 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind: discussion discussions, questions, requests for comments, and so on kind: enhancement Label for issues suggesting enhancements; and for pull requests implementing enhancements release notes: use title For PRs: the title of this PR is suitable for direct use in the release notes topic: kernel topic: packages issues or PRs related to package handling, or specific to a package (for packages w/o issue tracker)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants