fix(ts/js) use identifier to match potential keywords (#2519)

- (parser) Adds `keywords.$pattern` key to grammar definitions
- `lexemes` is now deprecated in favor of `keywords.$pattern` key
- enh(typescript) use identifier to match potential keywords, preventing false positives 
- enh(javascript) use identifier to match potential keywords, preventing false positives

Author: joshgoebel

CHANGES.md

@@ -2,6 +2,7 @@
 
 Parser Engine:
 
+- (parser) Adds `keywords.$pattern` key to grammar definitions (#2519) [Josh Goebel][]
 - (parser) Adds SHEBANG utility mode [Josh Goebel][]
 - (enh) Added `on:begin` callback for modes (#2261) [Josh Goebel][]
 - (enh) Added `on:end` callback for modes (#2261) [Josh Goebel][]
@@ -10,10 +11,12 @@ Parser Engine:
 
 Deprecations:
 
-- (deprecation) `endSameAsBegin` is now deprecated. (#2261) [Josh Goebel][]
+- `lexemes` is now deprecated in favor of `keywords.$pattern` key (#2519) [Josh Goebel][]
+- `endSameAsBegin` is now deprecated. (#2261) [Josh Goebel][]
 
 Language Improvements:
-
+- enh(typescript) use identifier to match potential keywords, preventing false positivites (#2519) [Josh Goebel][]
+- enh(javascript) use identifier to match potential keywords, preventing false positivites (#2519) [Josh Goebel][]
 - [enh] Add `OPTIMIZE:` and `HACK:` to the labels highlighted inside comments [Josh Goebel][]
 - enh(typescript/javascript/coffeescript/livescript) derive ECMAscript keywords from a common foudation (#2518) [Josh Goebel][]
 - enh(typescript) add setInterval, setTimeout, clearInterval, clearTimeout (#2514) [Josh Goebel][]
@@ -30,7 +33,7 @@ Language Improvements:
 [Vania Kucher]: https://github.com/qWici
 
 
-## Version 10.0.2 (pending)
+## Version 10.0.2
 
 Brower build:
 

docs/language-guide.rst

@@ -64,17 +64,19 @@ and most interesting parsing happens inside tags.
 Keywords
 --------
 
-In the simple case language keywords are defined in a string, separated by space:
+In the simple case language keywords can be defined with a string, separated by space:
 
 ::
 
   {
     keywords: 'else for if while'
   }
 
-Some languages have different kinds of "keywords" that might not be called as such by the language spec
-but are very close to them from the point of view of a syntax highlighter. These are all sorts of "literals", "built-ins", "symbols" and such.
-To define such keyword groups the attribute ``keywords`` becomes an object each property of which defines its own group of keywords:
+Some languages have different kinds of "keywords" that might not be called as
+such by the language spec but are very close to them from the point of view of a
+syntax highlighter. These are all sorts of "literals", "built-ins", "symbols"
+and such. To define such keyword groups the attribute ``keywords`` becomes an
+object each property of which defines its own group of keywords:
 
 ::
 
@@ -85,19 +87,25 @@ To define such keyword groups the attribute ``keywords`` becomes an object each
     }
   }
 
-The group name becomes then a class name in a generated markup enabling different styling for different kinds of keywords.
+The group name becomes the class name in the generated markup enabling different
+themeing for different kinds of keywords.
 
-To detect keywords highlight.js breaks the processed chunk of code into separate words — a process called lexing.
-The "word" here is defined by the regexp ``[a-zA-Z][a-zA-Z0-9_]*`` that works for keywords in most languages.
-Different lexing rules can be defined by the ``lexemes`` attribute:
+To detect keywords highlight.js breaks the processed chunk of code into separate
+words — a process called lexing. By default "words" are matched with the regexp
+``\w+``, and that works well for many languages. Different lexing rules can be
+defined by the magic ``$pattern`` attribute:
 
 ::
 
   {
-    lexemes: '-[a-z]+',
-    keywords: '-import -export'
+    keywords: {
+      $pattern: /-[a-z]+/,        // allow keywords to begin with dash
+      keyword: '-import -export'
+    }
   }
 
+Note: The older ``lexemes`` setting has been deprecated in favor of using
+``keywords.$pattern``. They are functionally identical.
 
 Sub-modes
 ---------

docs/mode-reference.rst

@@ -241,14 +241,19 @@ and ``endSameAsBegin: true``.
 
 .. _lexemes:
 
-lexemes
-^^^^^^^
+lexemes (now keywords.$pattern)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 **type**: regexp
 
-A regular expression that extracts individual lexemes from language text to find :ref:`keywords <keywords>` among them.
-Default value is ``hljs.IDENT_RE`` which works for most languages.
+A regular expression that extracts individual "words" from the code to compare
+against :ref:`keywords <keywords>`. The default value is ``\w+`` which works for
+many languages.
 
+Note: It's now recommmended that you use ``keywords.$pattern`` instead of
+``lexemes``, as this makes it easier to keep your keyword pattern associated
+with your keywords themselves, particularly if your keyword configuration is a
+constant that you repeat multiple times within different modes of your grammar.
 
 .. _keywords:
 
@@ -259,8 +264,8 @@ keywords
 
 Keyword definition comes in two forms:
 
-* ``'for while if else weird_voodoo|10 ... '`` -- a string of space-separated keywords with an optional relevance over a pipe
-* ``{'keyword': ' ... ', 'literal': ' ... '}`` -- an object whose keys are names of different kinds of keywords and values are keyword definition strings in the first form
+* ``'for while if|0 else weird_voodoo|10 ... '`` -- a string of space-separated keywords with an optional relevance over a pipe
+* ``{keyword: ' ... ', literal: ' ... ', $pattern: /\w+/ }`` -- an object that describes multiple sets of keywords and the pattern used to find them
 
 For detailed explanation see :doc:`Language definition guide </language-guide>`.
 

src/highlight.js

@@ -131,8 +131,8 @@ const HLJS = function(hljs) {
       }
 
       let last_index = 0;
-      top.lexemesRe.lastIndex = 0;
-      let match = top.lexemesRe.exec(mode_buffer);
+      top.keywordPatternRe.lastIndex = 0;
+      let match = top.keywordPatternRe.exec(mode_buffer);
       let buf = "";
 
       while (match) {
@@ -148,8 +148,8 @@ const HLJS = function(hljs) {
         } else {
           buf += match[0];
         }
-        last_index = top.lexemesRe.lastIndex;
-        match = top.lexemesRe.exec(mode_buffer);
+        last_index = top.keywordPatternRe.lastIndex;
+        match = top.keywordPatternRe.exec(mode_buffer);
       }
       buf += mode_buffer.substr(last_index);
       emitter.addText(buf);

src/languages/1c.js

@@ -5,7 +5,7 @@ Description: built-in language 1C:Enterprise (v7, v8)
 Category: enterprise
 */
 
-export default function(hljs){
+export default function(hljs) {
 
   // общий паттерн для определения идентификаторов
   var UNDERSCORE_IDENT_RE = '[A-Za-zА-Яа-яёЁ_][A-Za-zА-Яа-яёЁ_0-9]+';
@@ -446,9 +446,12 @@ export default function(hljs){
   // meta : инструкции препроцессора, директивы компиляции
   var META = {
     className: 'meta',
-    lexemes: UNDERSCORE_IDENT_RE,
+
     begin: '#|&', end: '$',
-    keywords: {'meta-keyword': KEYWORD + METAKEYWORD},
+    keywords: {
+      $pattern: UNDERSCORE_IDENT_RE,
+      'meta-keyword': KEYWORD + METAKEYWORD
+    },
     contains: [
       COMMENTS
     ]
@@ -463,7 +466,6 @@ export default function(hljs){
   // function : объявление процедур и функций
   var FUNCTION = {
     className: 'function',
-    lexemes: UNDERSCORE_IDENT_RE,
     variants: [
       {begin: 'процедура|функция', end: '\\)', keywords: 'процедура функция'},
       {begin: 'конецпроцедуры|конецфункции', keywords: 'конецпроцедуры конецфункции'}
@@ -474,9 +476,9 @@ export default function(hljs){
         contains: [
           {
             className: 'params',
-            lexemes: UNDERSCORE_IDENT_RE,
             begin: UNDERSCORE_IDENT_RE, end: ',', excludeEnd: true, endsWithParent: true,
             keywords: {
+              $pattern: UNDERSCORE_IDENT_RE,
               keyword: 'знач',
               literal: LITERAL
             },
@@ -496,8 +498,8 @@ export default function(hljs){
   return {
     name: '1C:Enterprise',
     case_insensitive: true,
-    lexemes: UNDERSCORE_IDENT_RE,
     keywords: {
+      $pattern: UNDERSCORE_IDENT_RE,
       keyword: KEYWORD,
       built_in: BUILTIN,
       class: CLASS,

src/languages/armasm.js

@@ -21,8 +21,8 @@ export default function(hljs) {
     name: 'ARM Assembly',
     case_insensitive: true,
     aliases: ['arm'],
-    lexemes: '\\.?' + hljs.IDENT_RE,
     keywords: {
+      $pattern: '\\.?' + hljs.IDENT_RE,
       meta:
         //GNU preprocs
         '.2byte .4byte .align .ascii .asciz .balign .byte .code .data .else .end .endif .endm .endr .equ .err .exitm .extern .global .hword .if .ifdef .ifndef .include .irp .long .macro .rept .req .section .set .skip .space .text .word .arm .thumb .code16 .code32 .force_thumb .thumb_func .ltorg '+

src/languages/avrasm.js

@@ -9,8 +9,8 @@ export default function(hljs) {
   return {
     name: 'AVR Assembly',
     case_insensitive: true,
-    lexemes: '\\.?' + hljs.IDENT_RE,
     keywords: {
+      $pattern: '\\.?' + hljs.IDENT_RE,
       keyword:
         /* mnemonic */
         'adc add adiw and andi asr bclr bld brbc brbs brcc brcs break breq brge brhc brhs ' +

src/languages/bash.js

@@ -81,8 +81,8 @@ export default function(hljs) {
   return {
     name: 'Bash',
     aliases: ['sh', 'zsh'],
-    lexemes: /\b-?[a-z\._]+\b/,
     keywords: {
+      $pattern: /\b-?[a-z\._]+\b/,
       keyword:
         'if then else elif fi for while in do done case esac function',
       literal:

src/languages/basic.js

@@ -11,8 +11,8 @@ export default function(hljs) {
     case_insensitive: true,
     illegal: '^\.',
     // Support explicitly typed variables that end with $%! or #.
-    lexemes: '[a-zA-Z][a-zA-Z0-9_\$\%\!\#]*',
     keywords: {
+        $pattern: '[a-zA-Z][a-zA-Z0-9_\$\%\!\#]*',
         keyword:
           'ABS ASC AND ATN AUTO|0 BEEP BLOAD|10 BSAVE|10 CALL CALLS CDBL CHAIN CHDIR CHR$|10 CINT CIRCLE ' +
           'CLEAR CLOSE CLS COLOR COM COMMON CONT COS CSNG CSRLIN CVD CVI CVS DATA DATE$ ' +

src/languages/clojure.js

@@ -7,8 +7,11 @@ Category: lisp
 */
 
 export default function(hljs) {
+  var SYMBOLSTART = 'a-zA-Z_\\-!.?+*=<>&#\'';
+  var SYMBOL_RE = '[' + SYMBOLSTART + '][' + SYMBOLSTART + '0-9/;:]*';
   var globals = 'def defonce defprotocol defstruct defmulti defmethod defn- defn defmacro deftype defrecord';
   var keywords = {
+    $pattern: SYMBOL_RE,
     'builtin-name':
       // Clojure keywords
       globals + ' ' +
@@ -41,8 +44,6 @@ export default function(hljs) {
       'lazy-seq spread list* str find-keyword keyword symbol gensym force rationalize'
   };
 
-  var SYMBOLSTART = 'a-zA-Z_\\-!.?+*=<>&#\'';
-  var SYMBOL_RE = '[' + SYMBOLSTART + '][' + SYMBOLSTART + '0-9/;:]*';
   var SIMPLE_NUMBER_RE = '[-+]?\\d+(\\.\\d+)?';
 
   var SYMBOL = {
@@ -86,7 +87,6 @@ export default function(hljs) {
   };
   var NAME = {
     keywords: keywords,
-    lexemes: SYMBOL_RE,
     className: 'name', begin: SYMBOL_RE,
     starts: BODY
   };