[Case Study] Perfect TypeScript Integration

[arcanis]

What package is covered by this investigations?

TypeScript

Describe the goal of the investigation

We currently support TypeScript through PnPify, but this isn't as good as a real integration - we need to emulate a whole filesystem with all its intricacies, it changes the command line (yarn pnpify tsc instead of tsc), disables some of PnP's features (such as semantic errors)...

This investigation's goal is to figure out all the blockers that prevent TypeScript from working with Yarn 2 so that we can provide a clear answer when asked what we would be needing - and potentially open issues and PRs against TypeScript to help this effort.

Investigation report

---

resolveModuleName / resolveTypeReferenceDirective

Those two passes already include hooks that allow to override their typical mechanism - which is what we've actually done in ts-pnp.

My main wish would be for those hooks to be builtin to TypeScript, as they aren't particularly big. To give you an idea, ts-pnp is 80 lines big with a redundant logic. The main issue is that it would involve loading the .pnp.js data in some capacity, and the TS team has historically been reluctant. I think an easy way to solve this is to simply call require('pnpapi') instead of directly requiring the .pnp.js file.

I believe this solution would work because since the pnpapi module only exists when PnP is loaded (so not when running tsc directly, and neither when running the TS that's embed within VSCode), it means that it'll be up to the user to explicitly request to use a modified environment. In the case of VSCode it means they'll have to explicitly switch to the Workspace version:

---

fileExists / directoryExists / readFile / realpath

Now that Yarn 2 is keeping the files within zip archives, accessing them from within TS requires some extra work as we would need TS to understand that accessing /foo/bar.zip/index.js would actually mean accessing /index.js within the archive /foo/bar.zip.

This part is likely the one most likely to receive pushback from the TS team, as its implementation can potentially involve adding a large logic to the TS core that may not be useful to all their users. That being said, I think a similar approach to the one detailed above could work as well: since the PnP file generated by Yarn is able to understand those zip paths, it could easily also provide an interface that could be used to implement the functions detailed above. After that, TS could implement fileExists using something similar to:

ts.effectiveFileExists = p => {
  if (!process.versions.pnp)
    return ts.fileExists(p);

  const pnp = require(`pnpapi`);
  if (!pnp.fsExtension)
    return ts.fileExists(p);

  return pnp.fsExtension.existsSync(p);
};

Similar functions might be readDirectory / getFileSize / getModifiedTime, but since they aren't in ModuleResolutionHost I'm not 100% sure that they really matter in this context.

---

getAutomaticTypeDirectiveNames

This function is called by createProgram in order to figure out all the @types packages that should be loaded. The problem is that getAutomaticTypeDirectiveNames calls getEffectiveTypeRoots, which in turns calls getDefaultTypeRoots, which in turns hardcodes an access to <project>/node_modules/@types. We never get the chance to change this behaviour to indicate to TS that the @types are stored in a separate location (or, in our case, multiple separate locations).

A proposal would be to add a getAutomaticTypeRoots(currentDirectory, host) to the CompilerHost interface which would be called by getEffectiveTypeRoots. By default it would just call getDefaultTypeRoots, but an environment like PnP could override it to instead do something similar to this - eliminating I/O calls and making TS faster in the process:

const pnp = require(`pnpapi`);
const locator = pnp.findPackageLocator(`${currentDirectory}/`);
const {packageDependencies} = pnp.getPackageInformation(locator);

const typeRoots = [];
for (const [name, referencish] of packageDependencies.entries()) {
  if (name.startsWith(`@types/`) && referencish !== null) {
    const dependencyLocator = pnp.toLocator(name, referencish);
    const {packageLocation} = pnp.getPackageInformation(dependencyLocator);

    typeRoots.push(packageLocation);
  }
}

getExtendsConfigPath

This function makes an hardcoded call to nodeModuleNameResolver, which prevents the host compiler from kicking in (the rational probably being that at this point the project options haven't been decided yet, and since the resolver takes them in parameter it's unclear how that should work).

I'm not sure what the TS teams would prefer - maybe a compiler host just for this purpose? That would avoid having to specify the project options as nullable, breaking existing resolveModuleName implementations.

[arcanis]

I went ahead and started implementing it inside a fork of TypeScript, to see in more practical terms what it would entails. Some additional findings:

• The fileExists, directoryExists, readFile, realpath methods don't actually matter! Since we operate under the assumption that PnP support is only available when TypeScript itself is running within a PnP runtime, then we also have the guarantee that fs module will see zip archives as traditional folders. Consequently, the fileExists & co functions will also see them as folders.

• The whole implementation (including support for getExtendsConfigPath and getAutomaticTypeDirectiveNames, on top of the regular resolveModuleName) takes literally less than a hundred lines.

My current branch is here, for reference:
arcanis/TypeScript@master...arcanis:mael/pnp

[accidentaldeveloper]

It's late and I'm a bit sleepy, so this may be a ridiculous idea or have lots of bad consequences.

If the implementation of PnP support in TS is so minimal, would it make any sense for the Yarn team to maintain and publish a fork of TS with native PnP support? This could just be a bridge to the time when TS implements it in the official package.

[arcanis]

It's not ridiculous; to be honest, I've been wondering that as well and I don't have a full answer yet. There are various parameters:

• I don't know how the TS team would feel about this - I know they have good intents in the way they want to handle this, so I don't want to offend them by going behind their back.

• We would have to track every TS release. They don't make them that often, so it might not be that much of a big deal - it could probably be automated in a large measure (we could use dependabot to trigger a rebase workflow each time a new one is published).

• There's a slippery slope argument to be made in that if everyone was doing this, then users would have to use N forks to get all the features they want - which is obviously impossible.

  • In particular, there's the ttypescript fork which adds support for third-party transformers.

• We would have to make it clear that we will autoclose any TypeScript-related issue.

Ping @orta, does your team have some thoughts about this?

[orta]

Yeah, personally, I don't see the rest of the team getting on-board with these changes until compiler plugin API makes it feasible to happen outside of the compiler itself. This is probably the right call for the short term.

[Skillz4Killz]

Hello 👋

Is it be possible to support Github repositories that are developed in Typescript? One of the issues I've recently hit is that when a library is developed with Typescript, you can not currently install it directly from the Github repository because the code first needs to be compiled. I think a perfect integration for Typescript should also compile Typescript code when installing the repo.

Example:
Suppose I create a library in Typescript, and host it on Github and publish the package as well. At some point, my library has some sort of an issue that I can push that fix to a branch on the repo which anyone can install. However, I don't want to push this fix to stable until such a time that the fix has been thoroughly tested.

At first, I thought this was a small problem I encountered but I just saw on Prettier repository today that they decided to not have Typescript because it would prevent users from installing from their Github directly.

prettier/prettier#6286

Let me know if this should be a separate issue.

[arcanis]

Github repository are packed using yarn pack before being used. If they list a prepack script, Yarn will first run yarn install followed by yarn run prepack. It's during this second step that repositories have the chance to specify build scripts to run before packing the project for consumption.

The other side of the coin is that it's significantly slower on a cold cache, unfortunately.

[hassankhan]

Hi all, would just like say that in a Yarn-Typescript integration, I'd love to see some support for Project References if Yarn Workspaces is enabled.

The plugin would, for each package, need to check dependencies and devDependencies to see if any of them can be locally resolved, then add the relevant entries into the references section of the tsconfig.json.

[arcanis]

Update: Yarn 2 now has transparent TypeScript support. Just run yarn add typescript and everything will work out-of-the-box thanks to the compat plugin.

[accidentaldeveloper]

Wow @arcanis , impressive work. Well done!