Proposed addition to libraryPath searching to support NPM

[flibbles]

Edit: Content removed. Look to my new proposal a few comments down will. Read that instead.

[netlify]

✅ Deploy Preview for tiddlywiki-previews ready!

Name	Link
🔨 Latest commit	0f35b8e
🔍 Latest deploy log	https://app.netlify.com/projects/tiddlywiki-previews/deploys/696c4fcf5e9c1100082498b3
😎 Deploy Preview	https://deploy-preview-9549--tiddlywiki-previews.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[github-actions]

Confirmed: flibbles has already signed the Contributor License Agreement (see contributing.md)

[github-actions]

📊 Build Size Comparison: empty.html

Branch	Size
Base (master)	2451.4 KB
PR	2451.5 KB

Diff: ⬆️ Increase: +0.2 KB

---

✅ Change Note Status

All change notes are properly formatted and validated!

📝 $:/changenotes/5.4.0/#9549

Type: enhancement | Category: plugin
Release: 5.4.0

Support for installing TiddlyWiki plugins through NPM

🔗 #9549

👥 Contributors: Flibbles

---

📖 Change Note Guidelines

Change notes help track and communicate changes effectively. See the full documentation for details.

[flibbles]

Hmm. The problem with windows that @CrossEye does appear to be the case. This solution for NPM does not work on Windows.

Back to the drawing board for the moment.

[flibbles]

New Proposal! I tooled around with a bunch of other possibilities and landed on this!

We have a dedicated tiddlywiki section in package.json which maps plugin names to directories. See Relink's package.json file.

Then we have function in boot/boot.js which will search npm module packages if it can't find a plugin the old fashion way. It builds a map by collecting together those tiddlywiki sections.

This works. It's clean. The tiddlywiki section is very straight forward for our plugin developers. And it does result in seamless plugin installation with npm for end users. A few points to discuss.

$tw.npmMap

Because searching through package.json files requires looking at a TON of directories, I make sure we only do this if any plugins can't otherwise be sourced, so no slowdown for most users. I also cache it in $tw.npmMap for now, which probably isn't appropriate. Anyone know where it is?

Speed

Startup is still pretty quick, but if we want to optimize this, we can prioritize searching through a project's packages-lock.json file for relevant packages first. That way, we don't trawl through all the node packages, of which there are a lot.

This is robust

At least it seems that way to me. This should handle both locally and globally installed packages.

Anyway, what are everyone's thoughts for this approach?

I can't see it being a problem on Windows, but I'll test that also before clearing this PR.

(Also: It's really ****ing annoying that ESLint rushes to tell me my PR is bad because one one small mistake, but when I run lint locally, I get 1500 mistakes.)

[flibbles]

I don't know why I'm getting dozens of emails about this PR failing. what the hell is this "validate-changenotes"? Why does it not exist in the repository? And how do I make it shut the hell up?

[flibbles]

Okay. I am done trying to appease this garbage build process that's been introduced. I have tried building the release notes. It's still flooding my inbox with emails calling me a failure.

It is not telling me what I'm doing wrong.

[flibbles]

Nope. Rebased to the broken master branch, and it's still fucking with me.

[flibbles]

Now Lint is failing again.

Sorry @pmario. You're going to get a lot of emails, because when I run lint on my end, I get 4253 errors. So I guess I've got to do a thousand little pushes.

[CrossEye]

@flibbles:

Now Lint is failing again.

I see that far too often here. It's making contributions far too difficult. I haven't had time to dig into why this happens. But it really must be fixed.

[flibbles]

Finally.

A few remarks I'd like to make.

1. Your build process failed me because I used '', instead of "". Are you all not aware that double quotes are the LEAST efficient form of quotation tiddlywiki could be using?
2. It wouldn't allow } catch {, but it also wouldn't allow } catch(e) { unless I put a vacuous e; in the catch block. But at least Lint is happy! Thank God for that.
3. It rode my ass for lines of code I never even wrote. They just happened to be near my PR
4. My blood pressure is through the roof right now
5. If this is what pushing PRs to TiddlyWiki is like now, I might be doing it a hell of a lot less from here on.

[yaisog]

it also wouldn't allow } catch(e) { unless I put a vacuous e; in the catch block

For me that only triggers a warning, not an error. Both locally and here. I use the ESLint config eslint.config.mjs from the repository also locally.

[pmario]

I think focusing on NPM for the start is OK. But

There are several runtime systems, which can run tiddlywiki. Node in combination with NPM is just one of them. There is Deno.com, Bun.com which basically are drop-in replacements. But we still need to test them.

If we add "library" searching code, we need to make sure that Deno, Bun and probably others also work with our new code. Otherwise we will get in trouble.

So imo function names like npmMap() are to specific.

Deno is runtime and npm library manager, but I think it stores the libraries loaded form npm in a different directory structure. But I am not sue about this one, since I did never analyse it that deep, because it was not necessary for TW.

IMO now it is necessary. ... Just some thoughts.

Also we need to make sure that Windows support is first class and not an oversight.

[Leilei332]

Now Lint is failing again.

Sorry @pmario. You're going to get a lot of emails, because when I run lint on my end, I get 4253 errors. So I guess I've got to do a thousand little pushes.

It is suggested to install eslint plugins if the code editor supports it. I also think it is annoying to receice "PR run failed" emails. Perhaps we can include eslint error in the comment rather than making pipeline fail.

[flibbles]

So imo function names like npmMap() are to specific.

@pmario, it may be possible to move the NPM specific stuff into the tiddlywiki.js file off the root directory. I believe (but I'm not sure) that file is only ever called by NPM implementations. The package.json hunting could happen there, and it could leave the plugin map somewhere that boot.js could find it.

Or something like that?

[flibbles]

Hey @pmario,

Since we're on the topic of overly specific code in the boot.js file, is there a reason we've never moved all the Node.JS specific boot code into a file like boot/boot.node.js, which boot.js could conditionally include? It would

• Shave about 500 lines that we'd never have to send to clients, or pack into standalone files.
• Break up the massive boot.js file a little in a meaningful and intuitive way.
• Allow for more modern javascript in the boot.node.js file, since we know it will never need to run on browsers.

If there's any merit to this idea, (Maybe by thumbs-upping it) I can create a separate issue to discuss it.

[pmario]

@flibbles -- There is a core-server directory in the tiddlywiki5 root directory now, which contains Node.js specific code. So there is some work going on to split stuff, that should not be needed by browser clients.

I am pretty sure, that Jeremy will be OK, if we can split stuff out of boot.js. Since he several times already mentioned, that this file is way too big and complex. boot.js has about 2700 lines including whitespace and comments. If we can make it shorter it will be a good move.

So I think you should open a new issue or add to an existing one, if there is one.

[flibbles]

@pmario, done. (#9557) But in the meantime. Would you say that you'd accept a change like this if it were sandboxed to a node specific boot file like bootnode.js? Or do you think this library searching is best suited in tiddlywiki.js.

[flibbles]

I have put in the following key changes:

• I've moved the library path loading into a boot/boot.node.js file so it doesn't bog down the browser version of TW.
• NPM loading depends on whether the npm_lifecycle_event is present, so it shouldn't disrupt any existing deployment environments like Deno or something.

Bear in mind that this PR does not have to be perfect.

All that matters is that it doesn't introduce bugs or backward incompatibility issues, nor does it introduce any code we'll need to commit to. It doesn't. Everything it introduces is private within a scope, or it's in a boot.node.js file that tells you that you shouldn't use it.

All that matters is that we agree on the design for how TW becomes NPM available. In this case, that packages will have a tiddlywiki field in their package.json file which maps plugin names to plugin directories. If THAT is agreeable, then I'd say merge this. Since I've kept everything private, we can elaborate our NPM support later, as we figure out how it should be done, and as people begin to adopt it.

[flibbles]

I have no idea why some of these build steps still haven't completed after 5 days, but I'm pretty sure it has nothing to do with my code.

I still posit that this PR is good to go as is, and that the finer details of NPM node_module adoption can be ironed out in later v5.4.x releases.

[flibbles]

Pinging @Jermolene on this.

NPM support doesn't need to be perfect for initial implementation. In fact, more basic is better, so we're committing ourself to very little. This only commits us to the idea of having tiddlywiki sections in package.json files. That's the only thing we need to be sure of with this PR.

[pmario]

@flibbles ... This seems to happen sometimes. Create a new commit and push it. So the process should start again.

[Jermolene]

Thanks @flibbles. On reflection I feel that there is not enough time in the v5.4.0 cycle for adequate review by other plugin authors who might be impacted. So I've added it to the plan for v5.5.0

[flibbles]

I understand, but I can't say I'm not disappointed.

One common complaint I see about Relink is people can't figure out how to install it on Node.JS. It involves directory copying or symlinks. Or one has to drop big json files into the correct directories.

While Uglify can be used to compress individual tiddlywiki files, it's real strength is that it can be added to Node.JS servers or plugin projects to compress output. But I don't think anyone does that because there's no easy way to integrate it alongside other build processes like gulp or sass. Not without checking all of Uglify into your own repository.

TW5-Graph integrates with other independent plugins, like Mermaid, and soon ECharts, but the only way to make it part of the site presentation, or integrate it into testing, would be to wholesale copy those plugins into my projects, which isn't good git hygiene.

And I already know of several other plugins, some which I'm not even the author of, which could benefit from being installable by NPM.

So while I understand, I really do hope this support comes in soon. I do believe that while TiddlyWiki is making strides in making plugins more easily accessible through the browser, its means of using plugins for Node.JS is lacking. Something that NPM has long figure out.

I will ping this thread occasionally. I won't let this fall into TiddlyWiki's dusty PR closet.