Require using AMD broken in 10.0.0

[jlchereau]

Loading via SystemJS in AMD used to work with v9.18.1. Broken in v10:

system.js:4 Uncaught (in promise) ReferenceError: hljs is not defined
    at eval (highlight.js:1566)
    at eval (highlight.js:6322)
    at eval (<anonymous>)
    at Qe (system.js:4)
    at system.js:4
    at async Promise.all (:63342/Kidoju.Widgets/src/js/widgets/index 10)

hljs is undefined when calling L1566

hljs.registerLanguage('apache', function () {
   ...
}());

In System.config.js, I have:

packages: {
    'src/js': {
        defaultExtension: 'js',
        meta: {
           '*.js': {
                babelOptions: {
                    es2015: false,
                    stage2: false,
                    stage3: false,
                },
                format: 'amd',
            },
            '*.es6': {
                format: 'esm',
            },
            '*.jsx': {
                format: 'esm',
            },
            '*.mjs': {
                format: 'esm',
            },
         },
      },
 },

A workaround for now is to add an exception in meta:

 'vendor/highlight/highlight.js': {
      // Note: AMD broken as of v10
      format: 'global',
},

[joshgoebel]

I'm not sure we ever were truly AMD compatible, the old init just did this:

  globalObject.hljs = factory({});

    // Finally register the global hljs with AMD.
    if(typeof define === 'function' && define.amd) {
      define([], function() {
        return globalObject.hljs;
      });
    }

IE, define a global, and then tell AMD to use that global... the new build system actually is actually behaving better (AMD wise) but the problem is now when we're trying to register the language modules there is no top-level hljs yet likely because I would assume AMD isn't (ever?) going to make one for us?

The language modules have always registered themselves via a top-level hljs global object.

So the behavior is different, but I'm not sure that's actually a problem - since it was "all globals all along" so far as I can tell. I'm not sure what the right answer is here because I don't think what we're doing before is actually correct... so perhaps someone with more knowledge can weigh in here.

[joshgoebel]

Perhaps we should just remove our AMD support? Perhaps someone super familiar with AMD could explain what we'd have to do with the language modules to make them truly AMD compatible (hard to say if that's viable without knowing what would be required)... just going back to how we had it before seems quite broken to me.

Do all the language modules need UMD wrappers? That would be annoying.

Some language modules also have their own built-in dependencies and expect to be loaded in the proper sequence... they would not do so well if the order were randomized, etc. (the build process takes care of this by inserting the registrations in the proper order in the final web build product).

[jlchereau]

I think the problem is in the structure of the build output file which is:

(function (global, factory) {
  typeof exports === 'object' && typeof module !== 'undefined' ? module.exports = factory() :
  typeof define === 'function' && define.amd ? define(factory) :
  (global = global || self, global.hljs = factory());
}(this, (function () { 'use strict';
  const HLJS = function(hljs) {
    ....
  }
  var highlight = HLJS({});
  return highlight;
})));

hljs.registerLanguage('apache', function () { // <-- hljs is undefined when using AMD
  ...
}());

After removing all hljs.registerLanguage, AMD loading works fine and returns the hljs object to the loader.

[joshgoebel]

Yes, I understand exactly why it is failing. The new AMD code fails to setup a global object (as it did before), so this now fails.

I'm less sure of the path forward.

This output is how the build process works (and always has). Languages are "modules" that are added AFTER the core library is setup by calling registerLanguage. This is also currently a requirement of the website builder/download tool too... the languages are just appeneded to the end of the core library.

[joshgoebel]

After removing all hljs.registerLanguage, AMD loading works fine and returns the hljs object to the loader.

And now you have no languages, so the highlighter won't work. :)

[joshgoebel]

From a different angle: What benefits do you gain by using AMD here vs having to declare it "global"?

[jlchereau]

I know but the problem is not with your UMD wrapper which is fine.

[joshgoebel]

I know but the problem is not with your UMD wrapper which is fine.

Well it depends how you define the problem. :-) You could also say the wrapper is EXACTLY the problem. The old wrapper setup a global, the new wrapper does not - and as a result it fails.

@egor-rogov Any thoughts here?

[jlchereau]

From my end, I can certainly use a workaround either with SystemJS (development) or Webpack (production). I'll always find a solution. I just thought it was nice to report an issue.

[joshgoebel]

I just thought it was nice to report an issue.

It certainly was. :-) And thanks. I'm just wondering the value of saying we support AMD when truly we don't and we register a global anyways... I have a feeling doing it "properly" might be time and effort better spent on other things...

Isn't part of the point of AMD to control/defer executing the JS modules?

Since Highlight.js has no external requirements I'm starting to wonder what AMD gives anyone here anyways? I guess you could express that other parts of your system are dependent on Highlight.js?

[jlchereau]

The issue is you are mixing modules and code outside of modules in the same file. I have also tried loading via CommonJS: same error:

Uncaught (in promise) ReferenceError: hljs is not defined
    at Object.eval (highlight.js:1566)
    at eval (highlight.js:6322)
    at eval (highlight.js:6323)
    at eval (<anonymous>)
    at Qe (system.js:4)
    at system.js:4
    at T (system.js:4)
    at U (system.js:4)
    at system.js:4
    at system.js:4

For this, simply configure SystemJS with:

'vendor/highlight/highlight.js': {
    format: 'cjs',
},

Your npm module works because it does not use the same code as the build output, highlight.pack.js. The npm module actually loads the module before loading languages:

var hljs = require('./core');
hljs.registerLanguage('apache', require('./languages/apache'));

In highlight.pack.js, the module id defined but not loaded before loading language packs, unless it has been assigned to global.

[jlchereau]

The cleaner option which I have just tested and which works is to change the structure as follows:

(function (global, factory) {
  typeof exports === 'object' && typeof module !== 'undefined' ? module.exports = factory() :
  typeof define === 'function' && define.amd ? define(factory) :
  (global = global || self, global.hljs = factory());
}(this, (function () { 'use strict';
  const HLJS = function(hljs) {
    ....
  }
  var highlight = HLJS({});

  highlight.registerLanguage('apache', function () {
      ...
  }());

  return highlight;
})));

This concatenates the file a bit differently but the result works both with AMD and CJS in addition to global = true UMD!

I have attached the working file for a diff: highlight.zip

[joshgoebel]

But that’s currently not possible for the website builder and would also break 3rd party languages that load later and expect a global. So if we are going to solve this we can’t just solve it half way.

I think we hack it like we did before or else it seems all the language modules need to become UMD. I’m not even sure how that might work. Surely AMD supports named dependencies?

[joshgoebel]

Do you know what the proper AMD way would be to call the registration at a later point?

Ie, how do I get the internal HLJS object back out of Amd or pass a callback that can access it?

[joshgoebel]

So if we give our module an id (which is easy enough):

typeof define === 'function' && define.amd ? define('highlight.js', factory) :

Then I think later you could do:

if (typeof define === 'function' && define.amd) {
  define(null, ["highlight.js"], (hljs) => { 
   hljs.registerLanguage('apache',...
  });
} else {
   hljs.registerLanguage('apache'...
}

Which is kind of what I expected... But I'm wondering what we gain by jumping thru all those hoops since no one has ever seemed to mind a global in the past?

Doing this would require all the 3rd party distributables to be recompiled to also include a UMD wrapper...

[jlchereau]

@yyyc514,

IMHO:

1. UMD loading in highlight.pack.js v10 is broken both for AMD and CommonJS,
2. highlight is a nice piece of code and you cannot have a broken feature: fix it or remove it,
3. your suggestion here above, is not the proper way to fix it for AMD and it does not fix it for CommonJS (I could explain but I am not sure it is worth the trouble).
4. In the long term, modules are now part of ECMAScript and are the way forward,
5. in the short term, most loaders including webpack can accommodate globals,
6. so in the end this is a question of product roadmap driven by user requirements.

I would recommend:

1. Until you have a proper fix, stick to global only to avoid misleading your users with features that do not work;
2. Give me a few days to look at the code for building highlight.pack.js and see if I can come up with an easy suggestion to fix this. If I can, I'll make a PR.

[jlchereau]

I just realized there is a lot of custom code to build highligh.js but why reinvent the wheel? i.e. why not use webpack to build your library? cf. https://webpack.js.org/guides/author-libraries/#expose-the-library. Any specific requirements? Just asking as you might consider delaying UMD as part of your product roadmap.

[egor-rogov]

@egor-rogov Any thoughts here?

No idea. Until today I was pretty sure that AMD is for Advanced Micro Devices (:

[joshgoebel]

Youur suggestion here above, is not the proper way to fix it for AMD

I was talking solely about how to do it "best" for AMD above... is that not how you would do it? Register the library with an ID and then make that id a dependency of the languages (which would also load via AMD)? Remember we have to support languages loaded at any time, not just those bundled with the JS file.

highlight.pack.js

Do you mean .min.js? There is no pack file anymore with version 10.

In the long term, modules are now part of ECMAScript and are the way forward,

If someone wanted to use the raw source it's all modules now and could be hooked into whatever build system, but the web library is built to a single simple file and the NPM library is built to CommonJS (require, etc), which Node still seems to be stuck on...

and CommonJS,

I'm not sure the web build has ever been CommonJS compatible (haha, and looks like it was because of how it bundled the languages - though i'm not sure the versions build by the website ever were), but feel free to correct me if I'm mistaken. Previous src/highlight.js happened to be since it ONLY included the library code, but that doesn't make sense now since that file is ES6 modules now, not a built file...

If someone wants to use CommonsJS they need to use the NPM package we provide.

[joshgoebel]

In the long term, modules are now part of ECMAScript and are the way forward,

Agree, and when Node.js natively supports them broadly then I think we'd look at publishing an NPM package that was ES6 modules vs require... until then anyone is free to use the raw source which is ahead of the curve (since v10).

[joshgoebel]

I just realized there is a lot of custom code to build highligh.js but why reinvent the wheel? i.e. why not use webpack to build your library? cf. https://webpack.js.org/guides/author-libraries/#expose-the-library. Any specific requirements? Just asking as you might consider delaying UMD as part of your product roadmap.

We have very specific requirements and the JS build code is all quite clear and easy to follow IMHO - it's all just functions, very little magic. You only need to know Javascript (and a bit of rollup). If we simplify it further in the future it'll likely be on top of rollup, not webpack. I've never cared for webpack; never seemed simple or intuitive to me at all.

The goal was never to ship a UMD build, just rather to "preserve" the existing AMD functionality from before... sadly I didn't realize how "hacked" it was, so we failed to do that.

---

If you want to explain how you feel the "proper" way to do AMD in this situation is, I will consider it but at this point I think the best way forward may be to just drop the previous hacky AMD support and not pretend to do AMD at all - and for the browser just ship a global version - which is really what we'd done all along anyways.

[joshgoebel]

So if we just fallback to IIFE for the browser build we'll have:

/*
  Highlight.js 10.0.0 (3875088a)
  License: BSD-3-Clause
  Copyright (c) 2006-2020, Ivan Sagalaev
*/
var hljs = (function () {
  'use strict';
  ...

Just a top level variable...

[joshgoebel]

@jlchereau Out of curiosity how were you using HLJS before? Just the default build with "common" languages built-in via AMD?

What will people's AMD pipelines do if we just switch to a pure global? Will they need to change their configs or will it just work despite not really being AMD?

[joshgoebel]

@jlchereau Can you take a quick look at the PR. It falls back to an IIFE build but also hooks up the variable to module.exports if it's around. I think this:

• Works in the browser (IIFE, global var hljs)
• Works for CommonJS (my Node.js test file can requite it and use it just fine)
• (therefore restoring the ability for someone to just grab the primary distributable and use it standalone with Node.js if they really wanted to)
• Drops support for AMD which we never truly supported properly anyways

[joshgoebel]

@jlchereau Any further thoughts? If not I'm going to move forward with just supporting CommonJS and IIFE/global and drop support for AMD.