Support package.json auto-discovery and bare specifiers

[dsherret]

Moved out from #17475

• Auto-discover package.json (only check cwd to ensure it's fast)
• bare specifiers work (eg. import express from "express")
• Import map and deno.json take precedence over package.json (bare specifiers first are resolved in deno.json and then fall back to package.json)

These changes will allow most projects written for Node.js to work out-of-the-box in Deno.

Future support for { "type": "commonjs" } for running legacy code.

Example. Suppose you have a package.json file like this:

{
  "name": "01-deno-app",
  "type": "module",
  "scripts": {
    "main": "deno run -A main.ts"
  },
  "dependencies": {
    "cowsay": "^1.0",
    "chalk": "^2.0"
  }
}

Then main.ts file that looks like this, will work out of the box:

import chalk from "chalk";
import cowsay from "cowsay";

const message = cowsay.getMessage("Hello");
console.log(chalk.green(message));

Side note: Maybe in the future: deno from-node subcommands, creates deno.json from package.json (doesn’t necessarily need to be built-in, e.g. deno run deno:from-node) writes a deno.json, also fixes your node global imports, and node build-in imports.

[aapoalas]

• Auto-discover package.json (only check cwd to ensure it's fast)

I expect the cwd-only discovery will need to be revisited, as enterprising Node users will have grown used to the recursive search and will expect that to work the same in Deno as well.

Future support for { "type": "commonjs" } for running legacy code.

Won't this bring sloppy mode support into Deno as well? A CJS Node project without "use strict"; declared may expect itself to be run in sloppy mode and may rely on some features of it as well.

"chalk": "npm:chalk@^1.0"

This is of course just an example but why would something like this exist?

Side note: Maybe in the future: deno from-node subcommands, creates deno.json from package.json (doesn’t necessarily need to be built-in, e.g. deno run deno:from-node) writes a deno.json, also fixes your node global imports, and node build-in imports.

Definitely support this being an external script. It's a one-time-only script so its performance is not super critical, so whether this is written in well-optimised TypeScript or well-optimised Rust should make no practical difference.

[KnorpelSenf]

Can you outline the reasoning behind this suggestion? It seems to me like migrating a Node project to Deno already is fairly easy since we got npm: support. If the compatibility is taken too far, we'd start to sacrifice some of the good things about Deno. For example, not having to probe the file system for the existence of lots of config files, that's one thing I liked about Deno. How did you assess how much demand there is for this change?

[dsherret]

• Auto-discover package.json (only check cwd to ensure it's fast)

I expect the cwd-only discovery will need to be revisited, as enterprising Node users will have grown used to the recursive search and will expect that to work the same in Deno as well.

@aapoalas Most likely. I personally am thinking we shouldn't auto-discover package.json recursively because it would probably add too much of a performance hit in cases that one wasn't used. I think we might want to only support a package.json if it's explicitly specified (ex. specified in a deno.json or explicitly provided on the command line).

Future support for { "type": "commonjs" } for running legacy code.

Won't this bring sloppy mode support into Deno as well? A CJS Node project without "use strict"; declared may expect itself to be run in sloppy mode and may rely on some features of it as well.

There's certain features we just won't support and we don't expect everything to work. IMO, code relying on sloppy mode would be an example code that wouldn't work in Deno.

"chalk": "npm:chalk@^1.0"

This is of course just an example but why would something like this exist?

This already works in package.json files (npm package aliasing). I'll remove that from the example because it's confusing and not many people know about that feature. It was added to show off something else that didn't make the cut.

Side note: Maybe in the future: deno from-node subcommands, creates deno.json from package.json (doesn’t necessarily need to be built-in, e.g. deno run deno:from-node) writes a deno.json, also fixes your node global imports, and node build-in imports.

Definitely support this being an external script. It's a one-time-only script so its performance is not super critical, so whether this is written in well-optimised TypeScript or well-optimised Rust should make no practical difference.

I also think it should be an external script. It would be easier to maintain it and add a bunch of "nice to have" features that way, which would otherwise be hard to get merged into the CLI.

Can you outline the reasoning behind this suggestion? It seems to me like migrating a Node project to Deno already is fairly easy since we got npm: support. If the compatibility is taken too far, we'd start to sacrifice some of the good things about Deno. For example, not having to probe the file system for the existence of lots of config files, that's one thing I liked about Deno. How did you assess how much demand there is for this change?

@KnorpelSenf Bartek will be able to explain better than me. cc @bartlomieju

[bartlomieju]

@KnorpelSenf

Can you outline the reasoning behind this suggestion? It seems to me like migrating a Node project to Deno already is fairly easy since we got npm: support. If the compatibility is taken too far, we'd start to sacrifice some of the good things about Deno. For example, not having to probe the file system for the existence of lots of config files, that's one thing I liked about Deno. How did you assess how much demand there is for this change?

That's partially true - eg. for Vite projects currently you need to put imports for all your dependencies in vite.config.js and even then in some cases in doesn't work (I couldn't get font to load properly with this approach). We tries writing a plugin for Vite that would handle npm: specifiers but that's currently not really possible due to a fact that Vite has some of the resolution logic hard coded to look for local node_modules/ directory. Sure, Vite might (and probably will fix this) in a future release, but currently it's a blocker. I'm sure there are more projects/frameworks that work on the same assumptions. Which means that Deno users are at disadvantage in using them. Additionally this requires to maintain separate project templates just for Deno and handling some quirks like requiring .mjs or .mts extension for config file to be treated as ES module. It's much easier for users if Deno could respect package.json in such projects.

Additionally copy-pasting another answer from other thread:

Bare specifiers via package.json and import map.
Future support for { "type": "commonjs" } for running legacy code.
One of the key points why my company had switched to Deno was the fact that Deno didn't support CJS. In the context of npm:-specifiers and Node compatibility, this point of the roadmap is OK, I just find the above-noted fact somewhat funny.

Some frameworks/build tools from npm still produce CommonJS output (eg. NextJS). We are still evaluating if this is needed, but with our current knowledge it is - it would be silly to support developing projects using these frameworks but not allowing to actually deploy/run the project. No one's happy about it, but it's an escape hatch (we're consider adding a warning about running CommonJS modules).

[KnorpelSenf]

I understand. How about limiting the auto-discovery of package.json files to modules that perform imports via npm:? In other words, we do not probe the file system when running native Deno modules, but only when the interop is necessary?

[aapoalas]

@aapoalas Most likely. I personally am thinking we shouldn't auto-discover package.json recursively because it would probably add too much of a performance hit in cases that one wasn't used. I think we might want to only support a package.json if it's explicitly specified (ex. specified in a deno.json or explicitly provided on the command line).

Specifying via deno.json would make sense since IIRC it was specified that deno.json would be checked for first. I guess this would still bring some unfortunate complexity from Deno having to essentially compose the import map and dependencies.

[aapoalas]

One thing that came to mind: What's the long term view of this feature related to authoring libraries/ modules, NPM and other registries, and Deno as that authoring tool? And the Deno ecosystem in general?

Currently dnt offers a way to author modules bound for NPM using Deno, but that is obviously not good enough since this issue is raised. Deno's NPM support makes an NPM first stance possible, in which case supporting package.json near fully makes sense. If the aim is to only enable transitioning from Node authoring to Deno authoring easily then a conversion script makes sense, which would then relegate package.json to only an NPM metadata file.

A hybrid strategy where package.json is near fully supported (dependencies, scripts, scope?, ...) AND a conversion script is provided AND /x/ registry tries to take on a more NPM-like form isn't really pushing for projects to move into the Deno ecosystem, and sort of reduces reasons to use the Deno ecosystem since it is becoming more like a small NPM.

As a hardliner I would of course prefer that a conversion script would be provided but no auto-discovery would be implemented. Then again, it may be a better idea in the long term to simply go quite all-in on NPM and adopt package.json completely.

[bartlomieju]

With #17864 landed, this is now done.