Official userland modules and simplifying core

[Qard]

A couple times now, I've seen people asking for things currently available in userland to be adopted into core. Some notable examples are:

• Ship websockets in core Ship websockets in core #1010
• Feature Request: HTTP2 Feature Request: HTTP2 #4

However, we've always had a policy of keeping core small and light. If anything, we wish we could remove things from core.

How about a compromise between the two: official userland modules. We'd bring a few things that a lot of people use into the iojs organization, form working groups for them, and have some coordination with core to make sure they remain stable.

The second part of this idea is to pull some of the JS parts out of core and include them in a new, global node_modules location, similar to how it used to work before making node only recognize local node_modules folders and npm changed to globally installed modules only exposing binaries, except npm wouldn't be able to change it. The existing core JS modules would come pre-installed in this location, but newer versions could be depended on locally.

This allows versioning the official JS modules independently of core, and allows a good transition path that we can encourage people to explicit depend on the modules in their package.json, but not break the world that doesn't do that currently.

[Fishrock123]

Ship websockets in core #1010
Feature Request: HTTP2 #4

I disagree that these are similar. One is a new module, and the other should be bringing an existing module up-to-spec.

[tellnes]

I'm +1 on official userland modules, but I don't think they should behave any different than normal userland modules. So I'm -1 on creating a new, global node_modules location.

You always wants to depend on some specific versions of a module, so you will never depend on the global installed module.

Also, if we where to implement this, people will start to put their own modules in there. It regularly comes up feature request to support requiring global modules, but it is a reason they are not accepted.

[jonathanong]

most of core should be "official userland modules" imo.

[Qard]

@Fishrock123 I'm not making any statement on their validity in being core modules, just that both exist in userland and people are asking that they be brought into core. As you say, HTTP/2 is more obviously a good candidate for core inclusion.

@tellnes I'm not suggesting this location is publicized. That suggestion is just a (hopefully temporary) migration path to pulling stuff out of core. We can't just pull these things out of core without providing an upgrade path or we'll break a massive number of things in userland.

[Qard]

Actually, it occurs to me that this could probably also still be built into the binary like it currently works, to prevent people tampering with it, as @tellnes commented. All we'd really have to do is reverse the logic that prevents core module names being overridden by userland modules of the same name, and then just claim those module names on npm.

[chrisdickinson]

The second part of this idea is to pull some of the JS parts out of core and include them in a new, global node_modules location, similar to how it used to work before making node only recognize local node_modules folders and npm changed to globally installed modules only exposing binaries, except npm wouldn't be able to change it. The existing core JS modules would come pre-installed in this location, but newer versions could be depended on locally.

most of core should be "official userland modules" imo.

I see this sentiment often, but the problem is that the core modules are a wad of peerDependencies, which, as we've seen play out in the npm ecosystem, don't play nicely with our expectations of how the module system should work. Everything in core depends (very nearly) on everything else in core; and expects to have common, shared data structures with everything else in core (event emitters, streams, buffers, et al). Breaking it up into individual modules will not increase the modularity/agility of core.

How about a compromise between the two: official userland modules. We'd bring a few things that a lot of people use into the iojs organization, form working groups for them, and have some coordination with core to make sure they remain stable.

Core has always tried to avoid "kingmaking"; the thought being that blessing a particular package for a particular purpose stifles innovation in that problem domain. It is a balance, though – new users don't (and shouldn't) care that they have 99 options for X, they just want the best X out there. npm is currently trying to solve through their website, and it'd be cool for our documentation to link into their website to appropriate topics / ecosystems of packages. That way we give new users a direction without stifling folks that want to try their hand at a faster HTTP2 parser.

[Qard]

The problem with avoid "kingmaking" is that some very popular packages suffer greatly in development support because there's little visibility of their development state. There are several modules in npm with 100k+ downloads and very few, or no active maintainers.

I don't see kingmaking as a problem anyway. People will always make alternatives of varying granularity anyway. Express was the undisputed king of routing frameworks for a long time, but that didn't stop many other people making their own version that fit their needs better.

You can also look at Go for comparison. They have their own routing framework built in, yet there's still a multitude of userland alternatives.

As for the cross-dependency issue: it is mostly the case that data structure interfaces are depended on by many things, but things like http/https are not depended on by anything else. Some things, like crypto, would make more sense as a regular dependency than a peer dependency. I'm not suggesting pulling all the JS out of core. Just that there are some things which maybe could be pulled out, but we don't even consider it because we don't have a clear upgrade path in place.

[mscdex]

@Qard It's really hard to extract any useful information from the npm downloads statistics because there are lot of npm mirrors and such out there. Add to that the problem of knowing whether a module is being downloaded because it's a dependency of another package or if it's being installed to be used directly. So who knows how many "real" downloads there are for any given module.

There used to be a short "notable mentions"-type list at the bottom of nodejs.org some time ago. However I pretty much agree with @chrisdickinson here, I think this discoverability problem is something npmjs.org and other sites (e.g. node-modules.com) should be solving, not node/io.js core.

[yosuke-furukawa]

I am +1 on official userland modules. golang team also chooses the similar strategy. They develops the Sub-Standard modules in userland (the modules is not built-in, we use go get command like npm install.).

And I also like the idea that some other core modules should put on official userland modules like readable-stream. But I think the changes occurs lots of discussion, so the changes would be better to discuss in NG working group.

[Qard]

@mscdex I disagree. Discoverability for the purpose of installing a module and discoverability for the purpose of finding things that need maintenance help are too completely different things, and the second is not something npm should have to concern itself with.

Also, whether or not a module is depended directly is irrelevant. If anything, it makes it more critical to make sure they are properly supported. Modules like ws, which are low-level dependencies that most people aren't even aware they are depending on should be made extremely stable to ensure they aren't going to cause problems for people in the future.

We may want to form a WG for official userland modules.

[chrisdickinson]

discoverability for the purpose of finding things that need maintenance help are too completely different things

Even if io.js forked these projects under its org, and built WGs to maintain them, we'd still have to convince everyone using the original version to switch, or get the original maintainers to give us rights to take over the project. If these projects are in need of help, I think it would be best for us (and for folks using those modules) to go help them where they are located at now, versus taking them over ourselves. I'm not opposed to calling attention to the need for more maintainers through the iojs evangelism WG.

[chrisdickinson]

cc'ing @rockbot and @zeke into the conversation, to get a better idea of where npmjs.org stands on the idea of directing folks to popular packages as well as packages in need of maintainership.