Merge pull request #36 from NodeSecure/documentation-enhancement

Documentation enhancement

Author: fraxken

README.md

@@ -10,7 +10,7 @@ JavaScript AST analysis. This package has been created to export the [Node-Secur
 
 The goal is to quickly identify dangerous code and patterns for developers and Security researchers. Interpreting the results of this tool will still require you to have a set of security notions.
 
-> 💖 I have no particular background in security. I'm simply becoming more and more interested and passionate about static code analysis. But I would be more than happy to learn that my work can help prevent potential future attacks (or leaks).
+> **Note** I have no particular background in security. I'm simply becoming more and more interested and passionate about static code analysis. But I would be more than happy to learn that my work can help prevent potential future attacks (or leaks).
 
 ## Goals
 The objective of the project is to successfully detect all potentially suspicious JavaScript codes.. The target is obviously codes that are added or injected for malicious purposes..
@@ -71,7 +71,7 @@ console.log(warnings);
 
 The analysis will return: `http` (in try), `crypto`, `util` and `fs`.
 
-> ⚠️ There is also a lot of suspicious code example in the root cases directory. Feel free to try the tool on these files.
+> **Warning** There is also a lot of suspicious code example in the `./examples` cases directory. Feel free to try the tool on these files.
 
 ## Warnings
 
@@ -106,26 +106,24 @@ import * as i18n from "@nodesecure/i18n";
 console.log(i18n.getToken(jsxray.warnings.parsingError.i18n));
 ```
 
-## Warnings Legends (v2.0+)
+## Warnings Legends
 
-> Node-secure versions equal or lower than 0.7.0 are no longer compatible with the warnings table below.
+> **Warning** versions of NodeSecure greather than v0.7.0 are no longer compatible with the warnings table below.
 
-This section describe all the possible warnings returned by JSXRay.
+This section describe all the possible warnings returned by JSXRay. Click on the warning **name** for additional information and examples.
 
-| name | description |
-| --- | --- |
-| parsing-error | An error occured when parsing the JavaScript code with meriyah. It mean that the conversion from string to AST as failed. If you encounter such an error, **please open an issue here**. |
-| unsafe-import | Unable to follow an import (require, require.resolve) statement/expr. |
-| unsafe-regex | A RegEx as been detected as unsafe and may be used for a ReDoS Attack. |
-| unsafe-stmt | Usage of dangerous statement like `eval()` or `Function("")`. |
-| unsafe-assign | Assignment of a protected global like `process` or `require`. |
-| encoded-literal | An encoded literal has been detected (it can be an hexa value, unicode sequence, base64 string etc) |
-| short-identifiers | This mean that all identifiers has an average length below 1.5. Only possible if the file contains more than 5 identifiers. |
-| suspicious-literal | This mean that the sum of suspicious score of all Literals is bigger than 3. |
-| obfuscated-code (**experimental**) | There's a very high probability that the code is obfuscated... |
-| weak-crypto (**experimental**) | The code probably contains a weak crypto algorithm ("md5...) |
-
-> 👀 More details on warnings and their implementations [here](./WARNINGS.md)
+| name | experimental | description |
+| --- | :-: | --- |
+| [parsing-error](./docs/parsing-error.md) | ❌ | The AST parser throw an error |
+| [unsafe-import](./docs/unsafe-import.md) | ❌ | Unable to follow an import (require, require.resolve) statement/expr. |
+| [unsafe-regex](./docs/unsafe-regex.md) | ❌ | A RegEx as been detected as unsafe and may be used for a ReDoS Attack. |
+| [unsafe-stmt](./docs//unsafe-stmt.md) | ❌ | Usage of dangerous statement like `eval()` or `Function("")`. |
+| [unsafe-assign](./docs/unsafe-assign.md) | ❌ | Assignment of a protected global like `process` or `require`. |
+| [encoded-literal](./docs/encoded-literal.md) | ❌ | An encoded literal has been detected (it can be an hexa value, unicode sequence or a base64 string) |
+| [short-identifiers](./docs/short-identifiers.md) | ❌ | This mean that all identifiers has an average length below 1.5. |
+| [suspicious-literal](./docs/suspicious-literal.md) | ❌ | A suspicious literal has been found in the source code. |
+| [obfuscated-code](./docs/obfuscated-code.md) | ✔️ | There's a very high probability that the code is obfuscated. |
+| [weak-crypto](./docs/weak-crypto.md) | ✔️ | The code probably contains a weak crypto algorithm (md5, sha1...) |
 
 ## API
 
@@ -153,6 +151,32 @@ interface Report {
 
 </details>
 
+<details>
+<summary>runASTAnalysisOnFile(pathToFile: string, options?: RuntimeFileOptions): Promise< ReportOnFile ></summary>
+
+```ts
+interface RuntimeOptions {
+    module?: boolean;
+    isMinified?: boolean;
+}
+```
+
+Run the SAST scanner on a given JavaScript file.
+
+```ts
+export type ReportOnFile = {
+  ok: true,
+  warnings: Warning<BaseWarning>[];
+  dependencies: ASTDeps;
+  isMinified: boolean;
+} | {
+  ok: false,
+  warnings: Warning<BaseWarning>[];
+}
+```
+
+</details>
+
 
 ## Contributors ✨
 

WARNINGS.md

@@ -0,0 +1,21 @@
+# Encoded literal
+
+| Code | Severity | i18n | Experimental |
+| --- | --- | --- | :-: |
+| encoded-literal | `Information` | `sast_warnings.encoded_literal` | ❌ | 
+
+## Introduction
+
+The SAST scanner assert all Literals in the tree and search for encoded values. JS-X-Ray currently supports three types of detection:
+- Hexadecimal sequence: `'\x72\x4b\x58\x6e\x75\x65\x38\x3d'`
+- Unicode sequence: `\u03B1`
+- Base64 encryption: `z0PgB0O=`
+
+Hexadecimal and Unicode sequence are tested directly on the raw Literal provided by meriyah. For base64 detection we use the npm package [is-base64](https://github.com/miguelmota/is-base64).
+
+Example of a JavaScript implementation:
+```js
+const hasHexadecimalSequence = /\\x[a-fA-F0-9]{2}/g.exec(node.raw) !== null;
+const hasUnicodeSequence = /\\u[a-fA-F0-9]{4}/g.exec(node.raw) !== null;
+const isBase64 = isStringBase64(node.value, { allowEmpty: false });
+```

docs/encoded-literal.md

@@ -0,0 +1,80 @@
+# Obfuscated code
+
+| Code | Severity | i18n | Experimental |
+| --- | --- | --- | :-: |
+| obfuscated-code | `Critical` | `sast_warnings.obfuscated_code` | ✔️ | 
+
+## Introduction
+
+An **experimental** warning capable of detecting obfuscation and sometimes the tool used. The scanner is capable to detect:
+
+- [freejsobfuscator](http://www.freejsobfuscator.com/)
+- [jjencode](https://utf-8.jp/public/jjencode.html)
+- [jsfuck](http://www.jsfuck.com/)
+- [obfuscator.io](https://obfuscator.io/)
+- morse
+- [trojan source](https://trojansource.codes/)
+
+Example of obfuscated code is in the root `examples` directory.
+
+### Technical note
+A complete G.Drive document has been written to describe the patterns of obfuscation tools and some way of detecting them:
+
+- [JSXRay - Patterns of obfuscated JavaScript code](https://docs.google.com/document/d/11ZrfW0bDQ-kd7Gr_Ixqyk8p3TGvxckmhFH3Z8dFoPhY/edit?usp=sharing)
+
+> **Note** There is no frozen implementation and this is an early implementation
+
+## Example
+
+The following code uses Morse code to obfuscate its real intent. This was used in an attack and I find it quite funny so i implemented morse detection 😂.
+
+```js
+function decodeMorse(morseCode) {
+  var ref = {
+    '.-': 'a',
+    '-...': 'b',
+    '-.-.': 'c',
+    '-..': 'd',
+    '.': 'e',
+    '..-.': 'f',
+    '--.': 'g',
+    '....': 'h',
+    '..': 'i',
+    '.---': 'j',
+    '-.-': 'k',
+    '.-..': 'l',
+    '--': 'm',
+    '-.': 'n',
+    '---': 'o',
+    '.--.': 'p',
+    '--.-': 'q',
+    '.-.': 'r',
+    '...': 's',
+    '-': 't',
+    '..-': 'u',
+    '...-': 'v',
+    '.--': 'w',
+    '-..-': 'x',
+    '-.--': 'y',
+    '--..': 'z',
+    '.----': '1',
+    '..---': '2',
+    '...--': '3',
+    '....-': '4',
+    '.....': '5',
+    '-....': '6',
+    '--...': '7',
+    '---..': '8',
+    '----.': '9',
+    '-----': '0',
+  };
+
+  return morseCode
+    .split('   ')
+    .map(a => a.split(' ').map(b => ref[b]).join(''))
+    .join(' ');
+}
+
+var decoded = decodeMorse(".-- --- .-. -..   .-- --- .-. -..");
+console.log(decoded);
+```

docs/obfuscated-code.md

@@ -0,0 +1,22 @@
+# Parsing Error
+
+| Code | Severity | i18n | Experimental |
+| --- | --- | --- | :-: |
+| ast-error | `Information` | `sast_warnings.ast_error` | ❌ | 
+
+## Introduction
+
+Parsing Error is throw when the library [meriyah](https://github.com/meriyah/meriyah) fail to parse the javascript source code into an AST. But it can also happen when the AST analysis fails because we don't manage a case properly.
+
+> **Note** If you are in the second case, please open an issue [here](https://github.com/NodeSecure/js-x-ray/issues)
+
+## Example
+
+```json
+{
+  "kind": "parsing-error",
+  "value": "[10:30]: Unexpected token: ','",
+  "location": [[0,0],[0,0]],
+  "file": "helpers\\asyncIterator.js"
+}
+```

docs/parsing-error.md

@@ -0,0 +1,30 @@
+# Short identifiers
+
+| Code | Severity | i18n | Experimental |
+| --- | --- | --- | :-: |
+| short-identifiers | `Warning` | `sast_warnings.short_identifiers` | ❌ | 
+
+## Introduction
+
+The SAST store in memory all Identifiers id so we are able later to sum the length of all ids. We are looking at several ESTree Node in the tree:
+- VariableDeclarator: `var boo;`
+- AssignmentExpression: `(boo = 5)`
+- FunctionDeclaration: `function boo() {}`
+- Property of ObjectExpression: `{ boo: 5 }`
+
+However, we do not take into consideration the properties of Objects for this warning. The warning is generated only if:
+
+- The file is not already declared as **Minified**.
+- There is more than **five** identifiers.
+- The sum of all identifiers name length is below `1.5`.
+
+## Example
+
+```json
+{
+  "kind": "short-identifiers",
+  "location": [[0,0], [0,0]],
+  "value": 1.5,
+  "file": "lib\\compile-env.js"
+}
+```

docs/short-identifiers.md

@@ -0,0 +1,55 @@
+# Suspicious literal
+
+| Code | Severity | i18n | Experimental |
+| --- | --- | --- | :-: |
+| suspicious-literal | `Warning` | `sast_warnings.suspicious_literal` | ❌ | 
+
+## Introduction
+
+Thats one of the most interesting JS-X-Ray warning. We designed it with the idea of detecting long strings of characters that are very common in malicious obfuscated/encrypted codes like in [smith-and-wesson-skimmer](https://badjs.org/posts/smith-and-wesson-skimmer/).
+
+The basic idea is to say that any string longer than 45 characters with no space is very suspicious... Then we establish a suspicion score that will be incremented according to several criteria:
+
+- if the string contains **space** in the first **45** characters then we set the score to `zero`, else we set the score to `one`.
+- if the string has more than **200 characters** then we add `1` to the score.
+- we add one to the score for each **750 characters**. So a length of __1600__ will add `two` to the score.
+- we add `two` point to the score if the string contains more than **70 unique characters**.
+
+So it's possible for a string with more than 45 characters to come out with a score of zero if:
+- there is space in the first 45 characters of the string.
+- less than 70 unique characters.
+
+The implementation is done in the [@nodesecure/sec-literal](https://github.com/NodeSecure/sec-literal/blob/main/src/utils.js) package and look like this:
+```js
+function stringCharDiversity(str, charsToExclude = []) {
+  const data = new Set(str);
+  charsToExclude.forEach((char) => data.delete(char));
+
+  return data.size;
+}
+
+// ---
+const kMaxSafeStringLen = 45;
+const kMaxSafeStringCharDiversity = 70;
+const kMinUnsafeStringLenThreshold = 200;
+const kScoreStringLengthThreshold = 750;
+
+function stringSuspicionScore(str) {
+  const strLen = stringWidth(str);
+  if (strLen < kMaxSafeStringLen) {
+    return 0;
+  }
+
+  const includeSpace = str.includes(" ");
+  const includeSpaceAtStart = includeSpace ? str.slice(0, kMaxSafeStringLen).includes(" ") : false;
+
+  let suspectScore = includeSpaceAtStart ? 0 : 1;
+  if (strLen > kMinUnsafeStringLenThreshold) {
+    suspectScore += Math.ceil(strLen / kScoreStringLengthThreshold);
+  }
+
+  return stringCharDiversity(str) >= kMaxSafeStringCharDiversity ? suspectScore + 2 : suspectScore;
+}
+```
+
+> **Note** The warning is generated only if the sum of all scores exceeds **three**.