Advice for library authors who use polyfills?

[nolanlawson]

I loved reading this document! One thing I found missing, though, was advice for library authors who use polyfills.

For instance, consider PouchDB. We ship various polyfills as part of the library: Promise, Map, Set, Object.assign, etc. In each case we do the "right thing" – we prefer the native implementation, and we ship the polyfill in such a way that it doesn't overwrite any globals (i.e. a "ponyfill").

But imagine that someone is using PouchDB, as well as Library B, Library C, and Library D, each of which does the same thing and ships their own polyfills for Promise, Map, Set… We could avoid a lot of this bloat if each library merely shipped code using the bare Promise, Map, Set, etc., but it's difficult to do so in such a way that doesn't aggravate users who want something that "just works."

So most library authors seem to take the path of least resistance, and just ship their own polyfills to avoid any burden on their users. Hence: library bloat.

Now, I'm sure PouchDB could ship two versions of its codebase using some Babel/Rollup/whatever transform to inject polyfills for one version but not the other, and we could advise our advanced users to use the bare version and then Bring Your Own Polyfill, but this is a lot of extra effort and I fear nobody would do it unless 1) there were clear instructions for the various module bundlers – Browserify/Webpack/Rollup/Closure/RequireJS/etc. – on how to do this correctly, and 2) a large number of library authors actually got on board and did it. It may also require some kind of ad-hoc standard, e.g. a "polyfills" field inside of package.json laying out exactly which globals are required. I dunno.

Without asking folks who are more familiar with bundlers/transpilers than I am, I'm not sure what advice exactly to give. But it may be worthwhile to at least acknowledge the problem in the document. (Unless you have some ideas for how to crack this nut, which I don't. 😃)

/cc @TheLarkInn @hzoo @Rich-Harris who have probably thought about this a lot more than me.

[triblondon]

Hey Nolan, thanks for this really thoughtful feedback. We'll take this to the next face to face meeting but just one clarification: what instructions would be needed for module bundlers? My assumption is that producing two versions of your library (one with polyfills, one without) is not difficult, and developers wishing to use the 'bare' version would load the necessary polyfills globally, so would not need to recompile your lib. Did I miss something?

[hzoo]

I haven't really thought about as much from a library point of view @nolanlawson but you make a lot of good points.

Oh are a lot of a library authors shipping polyfills? I guess that makes sense but to me it seemed like the general recommendation was that applications should be providing the polyfills (but of course like you said its least resistance for users then since then don't need to know that).

1. Bundle polyfills into library
2. Make 2 versions, one with polyfills included, one without
3. You can use something similar to http://babeljs.io/docs/plugins/transform-runtime (only works on non instance methods though) which doesn't change globals either
4. ?

The instance where you don't include polyfills seems best to me but then the question is how do you signal to the user that it's required (readme, easy of finding one and including it, etc)

---

From the app side, my purpose with useBuiltIns in babel-preset-env helps with auto doing this (similar to https://qa.polyfill.io/v2/docs/). babel/babel-preset-env#84 is still in discussion

[Rich-Harris]

I tend to be of the view that library authors should include any polyfills they're using (albeit in an encapsulated, non-polluting way per this repo's recommendations) that are required to work in the environments that the library supports, just like libraries should be distributed in a bundled and transpiled form. If I have to jump through hoops like rigging up an opinionated build step just to use your library, then your library is broken.

Having an app that works is the priority, deduping a few kb of shared polyfills is low-hanging optimisation fruit for those with the necessary expertise.

(This is consistent with the view that libraries in many cases shouldn't include polyfills by the way — it's totally fine for a library to say 'we only support environments that define Promise' or whatever.)

I think the real difficulty is in communicating to consumers how to choose between two different versions of a library, since you basically have to provide examples for all the different tooling. And it gets exponentially harder for packages that have multiple modules rather than exposing a single entry point. I'm not sure there's a universal solution! But again, as long as it works by default, you're meeting the needs of your userbase while pushing the burden of extra configuration and optimisation onto the expert users who can bear it.

[hzoo]

This is consistent with the view that libraries in many cases shouldn't include polyfills by the way
I think the real difficulty is in communicating to consumers

@Rich-Harris that sounds good too!

[Jessidhia]

@hzoo transform-runtime actually results in even more bloat for the user, should the user already provide their own polyfills. Say, if the final user already loads babel-polyfill (includes core-js in shim mode) and a random dependency was built with transform-runtime, you now get a second copy of core-js bundled.

[nolanlawson]

Thanks for the feedback, everyone! To answer your questions, I'd mostly like to expand on what @Rich-Harris said because I think he nailed it.

The problem is not so much technical (Babel/Rollup/Webpack already offer many solutions), but more a problem of documentation/standardization/ecosystem. E.g.:

• Library A says "for the polyfilled version, do require('library-a/with-polyfills') "
• Library B says "for the non-polyfilled version, do require('library-b/no-polyfills') "
• Library C says "do import { polyfilled } from 'library-c' "
• insert 500 more standards here

Result: pandemonium, nobody knows how to configure their app to handle everybody's format.

it's totally fine for a library to say 'we only support environments that define Promise' or whatever

Except when every library has a different list. :) E.g. one requires fetch, another requires Object.assign, etc. Admittedly this is already the case, but I already don't like it. It's a lot of mental overhead.

I do tend to agree that the default should be "polyfill all the things" (better from UX standpoint), and the alternative should be the "no polyfills" version for those who want to squeeze out more KBs. Question is if there's a way we can standardize around that, or at least promote good patterns for the ecosystem.

transform-runtime actually results in even more bloat for the user, should the user already provide their own polyfills

Another reason it'd be nice to crack this nut. 😃 One set of polyfills for the entire app codebase is ideal, but this requires coordination between app authors and library authors.

[palmerj3]

I believe library authors should definitely ship polyfills.

You specify (or should anyway) what versions of node/npm you support via package.json engines field and ultimately the lib should ship with everything needed in order to work in that version range.

While it would lessen the bloat if you didn't include polyfills it would put a serious burden on you and other library authors that go that path. You would have to define these polyfills as peerDependencies because that is exactly what they would be. peerDependencies are rarely a good idea..

So I think library authors should definitely ship polyfills and then application authors now have the burden of creating the smallest bundles possible. Tree shaking and other methods could be used to help reduce the bloat from an application.

[hzoo]

I do tend to agree that the default should be "polyfill all the things" (better from UX standpoint), and the alternative should be the "no polyfills" version for those who want to squeeze out more KBs.

Right that's a good point that we need both options (simple default, and ability to improve): users can have a hard time with Babel when dealing with polyfills or generators (via regenerator). We don't auto include them by default (or anything really atm) so it causes some pain.

They will ask why "doesn't array.prototype.includes work (or get transpiled)".

So we can point them to babel-polyfill (the whole thing) or transform-runtime, or the new useBuiltIn option for preset-env but there's a lot of work involved. It's also the use case of making a prod app or just testing out the library.

[hzoo]

But yeah if we had a tool to remove the libraries bundled polyfills automatically (sounds crazy) that would be pretty amazing too? And maybe in the future https://github.com/babel/babili or bundler like webpack could be a part of it

[fregante]

Throughout the development of a library, its developer will make the decision to support some browsers or not.

Should I include a Promise polyfill? How about a querySelectorAll polyfill?

You, as a library user, would you prefer a 1KB module or a 10K module that's compatible with some old browser you don't care about?

In short: it's up to the developer to include them ponyfill-style or not at all. They definitely should not clutter the global space.

[Rich-Harris]

The problem is not so much technical (Babel/Rollup/Webpack already offer many solutions), but more a problem of documentation/standardization/ecosystem. E.g.:

• Library A says "for the polyfilled version, do require('library-a/with-polyfills') "
• Library B says "for the non-polyfilled version, do require('library-b/no-polyfills') "
• Library C says "do import { polyfilled } from 'library-c' "
• insert 500 more standards here

...

Question is if there's a way we can standardize around that, or at least promote good patterns for the ecosystem.

Maybe not standardisation in the strong sense, but I certainly think we can reach a community consensus. It might be a slightly painful process, and the bikeshed may get a few layers of paint before we find a colour we all like, but it's doable (consider things like pkg.browser or jsnext:main and module, which basically mean the same thing across tools and libraries).

My vote would be for a pattern based on filenames, for the following reasons:

• It's discoverable, in that you can figure out what your options for consuming a library are by inspecting the package (a tip that I like to share with people — you can easily inspect a package's innards by appending / to an unpkg.com URL, e.g. https://unpkg.com/pouchdb/)
• Tooling already supports it, in the sense that Webpack, Browserify and Rollup (with node-resolve) will already resolve module identifiers like somepkg/src/foo.js correctly, and it's possible in many cases to alias module IDs appropriately
• Tooling support could be enhanced in future (auto-detecting according to filename patterns, and so on)
• It doesn't pollute package.json, which is a pretty crowded place

I don't think this is just about polyfills, either. I'd be interested to see libraries shipping untranspiled, unbundled source code (in addition to bundled, transpiled and polyfilled distributables) to give power users the option to customise their builds.

You, as a library user, would you prefer a 1KB module or a 10K module that's compatible with some old browser you don't care about?

This is where pragmatism > ideology. In that situation I'd choose the first. Realistically though, the choice is probably more often between, say, a 20kb unpolyfilled lib versus a 25kb polyfilled one, in which case I would probably choose the latter.

[hzoo]

I don't think this is just about polyfills, either. I'd be interested to see libraries shipping untranspiled, unbundled source code (in addition to bundled, transpiled and polyfilled distributables) to give power users the option to customise their builds.

Yeah really good point as well - especially as browser support for the latest features increases (right now the base is ES5, when do we move forward or always provide everything, etc). Maybe you'd want an ES5 base, or compiled down to latest spec version as well. There's also the idea of transpiling libraries/node_modules from the application itself but not really sure it's a good idea (yet)?

[TheLarkInn]

But yeah if we had a tool to remove the libraries bundled polyfills automatically (sounds crazy) that would be pretty amazing too?

Mulling all of what you guys have mentioned leads me to feel some sort of some sort of package.json level indication (polyfills field) could be a pretty neat idea. webpack already uses this similar concept when dealing with node built-in to provide api mocks on demand (but configured). I feel we could have a package of the most common 40 builtins and only the ones that are needed via a "polyfills" field. That's just an idea in my stream of consciousness atm.

[TheLarkInn]

@Rich-Harris makes a point that it could be more about having an alternate dist that is sans-polyfill, just like lib authors ship builds sans-transpiled.

[nolanlawson]

I tend to agree with all points. So to recap:

1. There should be a standard
2. "Polyfilled" should be the default case, "no polyfills" the special case
3. package.json is too crowded (although maybe it's needed for "instance" polyfills, e.g. Array.prototype.includes?)
4. this is about more than polyfills; it's also about shipping untranspiled source

I feel 4 may be too ambitious, because I find that in many of my codebases I'm using early-stage language features, or idiosyncratic ad-hoc stuff that I modify during the build process. Separating that from "untranspiled" may be tricky.

That said, I would love to bikeshed on what the shape of this could look like. 😃 Maybe something like:

// unpolyfilled
var library = require('library/bare');
import library from 'library/bare'; // es modules

// untranspiled, unpolyfilled
var library = require('library/raw');
import library from 'library/raw'; // es modules

// kitchen sink
var library = require('library');
import library from 'library'; // es modules

bare and raw are short and crisp, although maybe they don't communicate as much as unpolyfilled or untranspiled.

[fregante]

@Rich-Harris This is where pragmatism > ideology

Because of "pragmatism > ideology" you end up with 2MB of JavaScript on a website. Those 5KB per module pile up.

promise-polyfill alone is 4 times most of my Promise-based modules. And that's cruft for most users. I'm particularly aware of this issue because every time I find a nice small module it has some unnecessary dependency 5 times its size (e.g. bluebird)

// untranspiled, unpolyfilled
var library = require('library/raw');

I'd go for library/src, which is where the source might already be for many libraries.

[reconbot]

I once shipped a version serialport that accidentally polyfilled Promise. Users with their own polyfills ended up with mine by accident due to load order and it actually broke their app. That violated the principal of least surprise.

So I'm inclined to agree to not polyfill. And have shipped "bloated" libraries that might have used object.assign and promises but didn't share them and polute the global objects.

[Draggha]

I tend to not include polyfills. How should bundlers optimize different versions of the same polyfills?

What if you always shipped your code without polyfills and provided a seperate polyfills.js file (named like that by convention) with all needed polyfills. That way your code stays lean and accessible while providing a clear path for users.

My perceived main use cases are:

1. User 1 wants to just test/use your library/framework as is. She/He includes polyfills.js and after that your code. /Profit
2. User 2 wants to only support environments that provide the APIs you polyfilled. She/He includes your code. /profit
3. User 3 provides her/his own polyfills. She/He includes your code. /profit

Bundlers could find this polyfill.js file via various means (package.json property, config file, etc.) and optimize for use cases 1 to 3.
They could also much more easily provide the polyfills when you author your code. Maybe a fast path to generate a seperate polyfill file helps.

[nolanlawson]

@bfred-it

I'd go for library/src

Frequently the /src is Typescript/JSX/Coffeescript/futuristic JS with stage-0 features/etc. I feel /src might already be effectively squatted.

@reconbot

I once shipped a version serialport that accidentally polyfilled Promise. Users with their own polyfills ended up with mine by accident

I think everyone agrees polluting globals is bad; what I'm talking about is "ponyfills," i.e. polyfills that don't pollute the global scope. :)

@Draggha

What if you always shipped your code without polyfills and provided a seperate polyfills.js file (named like that by convention) with all needed polyfills.

Unfortunately this pollutes the global scope, leading to problems like @reconbot describes.

Bundlers could find this polyfill.js file via various means

The current method for inserting polyfills (e.g. used by babel-plugin-transform-runtime) is to identify global variables like Map and Promise using AST parsing, which is nice because it's more foolproof than a separate file (which may get out-of-date or be inaccurate).

[tomccabe]

Webpack 2 has moved to a module key in package.json which seems like it's being accepted by the community (unlike js:next). The end user can use resolve.mainFields to bring in the raw es6/7/8/15 source and transpile/polyfill themselves.

This seems like the sanest course to me; polyfill requirements will not be the same between users so trying to blanket solve the problem on the module side seems like a fool's errand.

[Draggha]

@nolanlawson & @reconbot yeah, you both convinced me that my proposal isn't helpfull. I actually never ran into a case of multiple totally different polyfills myself, but since I used both Q and bluebird in projects and learned some of their differences I can certainly relate to that.

@tomccabe so what you propose is that users that don't use a module bundler like webpack get all polyfills in the generated bundle and those that use bundlers build libs from source and thus provide all polyfills themselves, right?

What about Typescript (or any other compiled language)? The build process gets more complicated when you build node_modules yourself. Do I also describe this compiled language in the module field in package.json?
Should bundlers care about the dialect a library is being authored in? Does this choice belong in userland?
Do these compiled languages have to manage their polyfills? Does the bundler have to post-process these sources to treat all polyfilled sources the same way?

Sorry if I'm not being helpfull. I'm still trying to sort this whole topic and somehow always end up running in circles. Provide polyfills or document which ones are needed? In the beginning I did bundle them. Nowadays I prefer the latter, but maybe I need to reconsider again?

[fregante]

@nolanlawson but that's the point. That src/index.ts IS the untranspiled source you were talking about.

Whether that's compatible with the user's transpiler is the user's problem; if there's a JS file with async/await in there and my transpiler is not new enough to support it, my build is still gonna fail.

[georgezzhang]

@nolanlawson I like your proposal with regards to providing users a choice between "raw" code and transpiled and polyfilled, "kitchen sink" code. Two questions I have though:

1. Is there a need for both "raw" (no polyfills, no transpiling) and "bare" (no polyfills)? It seems to me like support for language features vs. APIs part of the same calculation, and if we want to leave the decision of what to support up to the user, then we should just ship a "raw" version that follows the language spec and a complete version that works out of the box, which leads me to my second question...

2. What goes into the complete, "kitchen sink" version? Is this just up to the whim of the library's developer? If I decide that I don't care about supporting IE9 and omit my classList polyfill, but some other library that my user installs is overzealous and includes a polyfill for querySelector or something, doesn't that get kind of confusing and messy to track for my user? Do we then need some kind of field in package.json that doesn't indicate what may need to be polyfilled, but rather what browsers or Node versions the default (non-"raw") version of the library supports?

[sokra]

Just want to give my 2 cents. I think polyfills should be not part of the library. It's the job of the application to include polyfills.

• The application developer choose which platform (i. e. browser version) he/she targets. Maybe the application will only run on modern browsers and so doesn't need polyfills at all (Example: WebGL game, cooperate app with fixed browser set, or dev dislike old browsers). Only the app developer knows which polyfills are (really) required.

• Application is updated more often than libraries, outdated polyfills can be removed earlier. Not issue using "old" libraries.

• Polyfills should only be included once. On application level this can be easily archived. On library level this gets out of control.

• A library is more difficult to use without polyfills, but performance > usability! So live with it.

[Rich-Harris]

I think the reality is that there isn't a one-size-fits-all answer — whether or not a particular library should include particular polyfills is always going to be a matter of judgment, and all we can really do is develop some conventions, and perhaps some shared guidelines about when and why to use them.

For example, does it make sense for libraries to bundle their own Promise polyfill? Probably not, since it's already very well supported, spec-compliant polyfills are quite bulky, and spec compliance is important since promises are often passed around between different libraries.

But what about something like Object.assign? A good-enough-for-my-library ponyfill weighs almost nothing — is leaving it out really worth the extra hassle for the library consumer? (Granted, Object.assign is also well-supported, but that's a double-edged sword — the app developer could very easily not realise they even need a polyfill because all the environments they test in have the feature.)

Then there's things like fetch. Lets say my library has a getJSON function that is implemented like this:

function getJSON ( url ) {
  return fetch( url ).then( r => r.json() );
}

I could add some documentation insisting that the app developer include the fetch polyfill, which has a lot of stuff I'm not using, and which doesn't even provide a lot of the good stuff (since fetch is lower-level than XHR), and hope that they read it (since otherwise I'm going to spend a bunch of time closing unnecessary issues). Or I could write an XHR-based helper that takes a dozen lines and yields the exact same result. The key point here is that very often, the library author is in a position to make smarter decisions about this stuff than the app developer.

Application is updated more often than libraries

That can be true, though a library might have many more contributors, and updates benefit all their consumers. Also, we have good mechanisms for updating libraries (npm-check-updates, greenkeeper.io etc) whereas removing no-longer-necessary polyfills is the kind of maintenance that is often given a low priority.

A library is more difficult to use without polyfills, but performance > usability! So live with it.

Something I often observe in these sorts of discussions is that people like us, who spend our free time contributing to threads like this, are the kinds of experts who frequently underestimate what a colossal burden this stuff really is for a lot of developers, especially novices. Because those people are unlikely to take parts in these conversations, we have to consider their needs on their behalf. I think that it's an important principle, especially in web development, that the burden of extra configuration etc should fall on those most able to bear it.

[nolanlawson]

@bfred-it Hmm you may be right; a simple src or raw directory that means different things in different scenarios may make the most sense. E.g. it could contain JSX, TypeScript, whatever, but the message is: "you are now in Expert Mode, please read the docs carefully to figure out how to transpile this." I just prefer raw over src because 1) I use idiosyncratic stuff in my own src, and 2) it's already squatted; some libraries may accidentally ship src but it's effectively unusable.

@georgezzhang I think you're right, bare vs raw is a confusing distinction.

@Rich-Harris

people like us, who spend our free time contributing to threads like this, are the kinds of experts who frequently underestimate what a colossal burden this stuff really is for a lot of developers, especially novices

👏 👏 👏

This is why I don't think "raw by default" is feasible for most library authors. When you're inundated with bug reports from confused users, the path of least resistance is to just ship the polyfills and be done with it. Also, libraries that "work out of the box" tend to rise to the top because not every user will go the extra mile to understand a library with a long, complicated README.

I still think the most interesting solution is a standard directory for "raw" code, so that bundlers/transpilers are aware of each library's unpolyfilled/untranspiled version. As @sokra says, if polyfilling is an app concern, then app bundlers could help users understand which global polyfills they would need in order to trim down their bundle size.

E.g. imagine an interactive console in Webpack like:

We detected that 3 of your dependencies have "raw" directories.
Would you like to switch to Raw Mode?

(Webpack then parses dependencies, finds async keyword, global fetch, and .jsx files)

We analyzed your dependencies, and you would need the following to switch to Raw Mode:
- babel-plugin-syntax-async-functions
- whatwg-fetch
- babel-plugin-jsx
Install them?

Of course I'm totally making this up, and it would require a huge coordination between Webpack, Babel, and library authors to work. But maybe it's possible?

[hzoo]

bare and raw are short and crisp, although maybe they don't communicate as much as unpolyfilled or untranspiled.

haha @nolanlawson had a hard time understanding the shorter names

imagine an interactive console in Webpack like:

so in this scenario

• all the libraries would include polyfills by default and in another "raw" folder it wouldn't.
• and with the same libraries they would transpile down to some base by default and in another "raw" folder it wouldn't

webpack reads the the dependencies/node_modules (via some method) to determine the libraries/code that support this functionality and then uses those to compile. Sounds good?

Basically either way we need to have multiple targets in npm packages (one for "raw" code which includes polyfills/transpiled) and one that includes everything done for you (easy to use). And we want the easy to use to be default. Thus a tool to make it happen

[sokra]

Or we go raw-by-default and communicate in package.json which polyfills are needed. The build tool can handle adding these polyfills. This way it's easy to use and minimal in size.

// package.json
"module": "src/index.js",
"uses": [
  "Object.assign",
  "fetch",
  "Object.defineProperty"
]

A shared module which contains default mappings for the polyfills is used by the build tool by default, and app developer could override/disable polyfills if needed. Maybe defaults are choosen by environment (i. e. "last 2 versions").

[jacobangel]

Great discussion. Where did it move to?

[nolanlawson]

Nowhere for now... 😕

[KevinAst]

I very much appreciate all of the thoughts here, and the document that was updated for library considerations.

One point I would like to add to this discussion (albeit late) is that as a library author you have absolutely NO idea what your target env is going to be. I can attempt to apply polyfills to my library (with "non polluting" due diligence), however that is from a perspective of the potential targets that may be problematic. I may be completely unaware of an antiquated target that requires a polyfill.

In reality, we want babel to free us from this burden, and it is becoming better and better at doing this!

From an app perspective, applying the needed polyfills has become a fairly simple and straightforward process, with the advent of the "babel-polyfill" import, in conjunction with babel's "babel-preset-env" configured with useBuiltins:"usage".

That is why my opinion has progressed to the point of NOT applying polyfills at the library level, and merely documenting the potential need for the app to do this (depending on it's target env).

Thoughts?

[Mouvedia]

My take on this is to give the means to install the polyfills that might be necessary to the user (e.g. package.json dependency field) and only then explicitly require the mandatory ones out of these (e.g. import 'example';) so that the bundlers will detect them automatically. Concretely the library author gotta set a reasonable threshold on both the dependencies to install and the subset which is forcefully required. That leaves the burden of installing the optional ones to the user.