Skip to content

Update @Ericbla encoder fork #243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 148 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
148 commits
Select commit Hold shift + click to select a range
e2d4af1
Implements encoding from Parser
Ericbla Feb 7, 2018
f756d09
Fix typo in Readme
Ericbla Feb 7, 2018
775553b
Replaces static Buffer with smart-buffer for internal encoding
Ericbla Feb 25, 2018
9285e9d
Fixes encoder option must not modify the content of object to encode
Ericbla Feb 26, 2018
9c64d23
Pass more context parameters to formatter and encoder options
Ericbla Feb 27, 2018
39e3c5b
Fixed random test error when using skip and encoding due to internal …
Ericbla Feb 27, 2018
b3e2c4b
Prepare to publish to npm under binary-parser-encoder package name
Ericbla Mar 1, 2018
dd75c74
Update note in Readme regarding forking project
Ericbla Mar 1, 2018
4443726
Fixes encoding array readUntil function and add encodeUntil option
Ericbla Aug 3, 2018
ad27cfe
Added "trim" and "padding" options for strings
Ericbla Sep 12, 2018
75379a7
Fixes #3 Array encoding
Ericbla Dec 21, 2018
340d57a
Bump version to 1.4.2
Ericbla Dec 21, 2018
b4ae885
Re-apply prettier format
Ericbla Dec 21, 2018
482fb0b
Fixes #16 Encding empty array
Ericbla Jan 9, 2019
6f267b6
Delivers #17 Add an option to set custom string padding char
Ericbla May 24, 2019
dbb1a9a
Update dependencies
Ericbla May 24, 2019
43893ca
Merge with upstream project master branch
Ericbla May 24, 2019
3107338
Merge remote-tracking branch 'upstream/master' into encoder
NotBobTheBuilder Jul 13, 2019
ee14f95
Tests run against dist
NotBobTheBuilder Jul 13, 2019
d4637c5
fix encoder ordering bug
NotBobTheBuilder Jul 13, 2019
2097f98
types are lowercase
NotBobTheBuilder Jul 13, 2019
fa74c9b
fix bad indent
NotBobTheBuilder Jul 13, 2019
204705b
array readuntil bugfix
NotBobTheBuilder Jul 13, 2019
1f8bdfa
Resolve missing typeof
NotBobTheBuilder Jul 13, 2019
3819684
Issue #19 Little endianess incorrect
Ericbla Oct 11, 2019
6f66bf2
Update dependencies
Ericbla Nov 23, 2019
a3a6cf8
Update to merge PR #18 from @NotBobTheBuilder
Ericbla Nov 23, 2019
0287201
Merge PR #18 from
Ericbla Nov 23, 2019
e34dc64
Added typescripts types definition in the distribution
Ericbla Nov 23, 2019
f571a59
Fixes #20 String padded with space (0x20) instead of null (0x0) when …
Ericbla Dec 15, 2019
16d272c
fix: ts type definitions
mspasov Mar 12, 2020
4733c3f
Merge pull request #21 from mspasov/type-fixes
Ericbla Mar 14, 2020
42612ee
Added a unit test for issue #23
Ericbla Oct 23, 2020
63c718a
Added .vscode to .gitignore
Ericbla Oct 30, 2020
6aeb70b
Merged with keichi/binary-parser master branch
Ericbla Oct 30, 2020
295c6ec
Fixes github build workflow
Ericbla Oct 30, 2020
398cdcf
Fix: Publish typings (.d.ts) files to NPM registry
ovr Sep 18, 2019
627f6e4
fix(TextEncoding): Binary Parser not work in old browsers
szaboge Dec 16, 2020
dc81af8
fix(TextEncoding): use `fast-text-encoding` instead of `text-encoding`
yuhr Mar 22, 2021
11712f9
Add VSCode workspace settings
yuhr Mar 22, 2021
fc040fa
Drop node 8 support
keichi Mar 23, 2021
1895f6a
add parcel to dev dependencies
abductedPlatypus Apr 8, 2021
4439b84
use strict equal in tests
abductedPlatypus Apr 8, 2021
3e6afda
add wrapper
abductedPlatypus Apr 8, 2021
3714a59
add wrapper test
abductedPlatypus Apr 8, 2021
b4a34b1
add missing strict equal
abductedPlatypus Apr 8, 2021
48ae061
fix missing )'s for hex encoded string
abductedPlatypus Apr 8, 2021
bcfb547
lint code & add missing cleanup
abductedPlatypus Apr 8, 2021
987601d
update package-lock.json
abductedPlatypus Apr 8, 2021
a9d908b
remove parcel from package
abductedPlatypus Apr 18, 2021
4fdaed1
add docs for wrapped
abductedPlatypus Apr 18, 2021
bf089e5
fix package-lock.json
abductedPlatypus Apr 18, 2021
1c50eaa
Update to lockfileVersion 2
keichi Apr 22, 2021
435de98
Update y18n to fix a vulnerability
keichi Apr 22, 2021
b4fceac
Update dev dependencies
keichi Apr 22, 2021
2f68fb4
Bump to v1.8.0
keichi Apr 22, 2021
700d24a
Update lodash to fix a vulnerability
keichi May 10, 2021
af7bea0
Add CI test for node 16
keichi May 10, 2021
fbc2c6c
Fixed some function signatures
Rzial Jun 3, 2021
bd36806
Removed trailing blank spaces
Rzial Jun 3, 2021
8382ab3
Added context variables on parsing process
Rzial Jun 3, 2021
7ba57ae
Added unit tests for parsing context variables
Rzial Jun 9, 2021
d8338a9
Fix #76
keichi Jun 10, 2021
fc3f816
Fixed formatting
Rzial Jun 12, 2021
e0e2e61
Added `$index` context variable
Rzial Jun 12, 2021
f77ecf2
Bump to v1.9.0
keichi Jun 19, 2021
d9ac644
Added missing semicolons on generated code
Rzial Jun 19, 2021
d19af5f
Bump to v1.9.1
keichi Jun 22, 2021
2778734
Removed typo on code generation
Rzial Jul 3, 2021
b9d24a6
Update binary_parser.ts
Rechdan Jul 4, 2021
7590e78
Update dependencies
keichi Jul 5, 2021
d88ee9e
Bump to v1.9.2
keichi Jul 6, 2021
96366f2
Add benchmark script
keichi Sep 13, 2021
c12589d
Update dependencies
keichi Sep 13, 2021
d38bb8e
Disable context variables by default
keichi Sep 18, 2021
17baac8
Use prettier's default formatting style
keichi Sep 18, 2021
b105fc7
Use double quotes in README
keichi Sep 21, 2021
0866236
Update binary parsing packages to compare against
keichi Sep 21, 2021
78c6e35
feat: remove IE11 & node.js 10 support
manzt Sep 22, 2021
8dedff3
feat: export esm entrypoint
manzt Sep 21, 2021
21add19
chore: combine source
manzt Sep 26, 2021
76ac35e
Update dependencies
keichi Sep 26, 2021
6f5fcdc
Add package script for running CJS build
keichi Sep 26, 2021
8345513
Bump version to 2.0.0
keichi Sep 26, 2021
8f77ddf
chore: set shared tsc target
manzt Sep 26, 2021
97b690c
fix: target ES6
manzt Sep 27, 2021
20beac2
Add hand-written parser as a benchmark target
keichi Sep 21, 2021
3507c82
Use let and const instead of var
keichi Sep 26, 2021
f87c314
Update README
keichi Sep 27, 2021
07aa9c9
Remove unused binary package from benchmark/package.json
keichi Sep 27, 2021
ac9755c
Fix typo
keichi Sep 27, 2021
dfb1e91
Bump to 2.0.1
keichi Sep 28, 2021
4473836
Rename LICENSE.md to LICENSE
keichi Sep 29, 2021
4ed5d3d
Enable strict type checking
keichi Sep 30, 2021
29a5353
Improve error messages
keichi Sep 30, 2021
9a1b73a
Translate unit tests to TypeScript
keichi Oct 2, 2021
8729eae
Run tests directly using ts-node
keichi Oct 4, 2021
2bc412b
Translate examples to TypeScript
keichi Oct 4, 2021
b3a6eaa
Make examples work from any directory
keichi Oct 4, 2021
ec9994c
Adjust build script
keichi Oct 4, 2021
d1f4782
Remove nyc
keichi Oct 4, 2021
78b293d
Run unit test on headless Chrome using Karma
keichi Oct 4, 2021
97116ee
Run style check with prettier on CI
keichi Oct 12, 2021
7e837fd
Add documentation for sizeOf() in README
keichi Oct 27, 2021
b4080d7
Fix assert as method definition
mohd-akram Nov 23, 2021
6543684
Bump to v2.0.2
keichi Nov 24, 2021
b52a54c
Bump karma from 6.3.4 to 6.3.14
dependabot[bot] Feb 11, 2022
6065daf
Bump follow-redirects from 1.14.4 to 1.14.8
dependabot[bot] Feb 14, 2022
61446da
Bump log4js from 6.3.0 to 6.4.1
dependabot[bot] Feb 15, 2022
60d7b60
Update dependencies
keichi Feb 15, 2022
ca50ae7
Escape assert strings
vexcat Apr 8, 2022
625d232
Bump minimist from 1.2.5 to 1.2.6
dependabot[bot] Apr 9, 2022
acdfbc3
Bump async from 3.2.1 to 3.2.3
dependabot[bot] Apr 13, 2022
4656f52
Format source code
keichi Apr 14, 2022
b63ca61
Fix typo (endianess -> endianness)
keichi May 9, 2022
f9672b8
Update dependencies
keichi May 9, 2022
6ab9435
Bump to v2.0.3
keichi May 9, 2022
11e1a49
Allow long bit field sequences
mohd-akram May 18, 2022
2b75d23
Bump to 2.1.0
keichi Jun 5, 2022
b7ffee6
feat: make `name` parameter optional for `wrapped`
wpyoga Sep 2, 2022
a8fca1e
fix: remove unnecessary null guard
wpyoga Sep 2, 2022
adce05f
test: add test for `wrapped` with empty `name` and without `name` arg…
wpyoga Sep 2, 2022
c3743aa
docs: update readme to reflect the new feature
wpyoga Sep 2, 2022
72b1e91
Bump to v2.2.0
keichi Sep 5, 2022
695243e
Add source map lib files
cmdcolin Sep 14, 2022
fe7a312
Add all dist folder
cmdcolin Sep 14, 2022
d87fc4b
docs: explain how to call `.useContextVars()`
wpyoga Sep 16, 2022
04d5f23
Update dependencies
keichi Sep 20, 2022
9f95d6b
Bump to v2.2.1
keichi Sep 20, 2022
2ba775b
Fix typos
mohd-akram Oct 4, 2022
786be0b
Add CodeQL workflow for GitHub code scanning
lgtm-migrator Nov 9, 2022
6cf521b
Bump socket.io-parser from 4.0.4 to 4.0.5
dependabot[bot] Nov 9, 2022
e5bf1e5
Bump qs and body-parser
dependabot[bot] Dec 8, 2022
21d16be
Bump minimatch from 3.0.4 to 3.1.2
dependabot[bot] Dec 9, 2022
c4731e5
Bump engine.io and socket.io
dependabot[bot] Dec 9, 2022
f6ee616
Bump json5 from 2.2.0 to 2.2.3
dependabot[bot] Jan 7, 2023
c527629
Bump ua-parser-js from 0.7.31 to 0.7.33
dependabot[bot] Jan 27, 2023
b4aa970
Additional checks to determine if next field is a bit field or not
nick-hunter Jun 19, 2023
033ef4a
Fix formatting issues. Add tests
nick-hunter Jun 20, 2023
1c6e68b
Update dependencies
keichi Jul 17, 2023
d6bc4f8
Bump node.js version requirement
keichi Jul 17, 2023
ab81ad7
Fix code formatting
keichi Jul 17, 2023
6ea611b
Merge remote-tracking branch 'upstream/master' into encoder
stereokai Aug 26, 2023
1e92ea8
Add smart-buffer to package-lock.json
stereokai Aug 26, 2023
65e8e39
MAke some corrections to get the build to pass
stereokai Aug 26, 2023
9120a20
Fix formatting of encoder text files
stereokai Aug 26, 2023
40de365
Fix bugs &ensure compatibility w/ ContextVariables
stereokai Aug 26, 2023
51d99c4
Fix bug: assign initial context variables properly
stereokai Aug 27, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/nodejs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@ name: test-nodejs

on:
push:
branches: [ master ]
branches: [ encoder]
pull_request:
branches: [ master ]
branches: [ encoder ]

jobs:
build:
Expand Down
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -18,5 +18,5 @@ node_modules
coverage/
.nyc_output/
dist/

.vscode/
.cache/
118 changes: 101 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,18 @@
[![npm](https://img.shields.io/npm/v/binary-parser)](https://www.npmjs.com/package/binary-parser)
[![license](https://img.shields.io/github/license/keichi/binary-parser)](https://github.com/keichi/binary-parser/blob/master/LICENSE)

Binary-parser is a parser builder for JavaScript that enables you to write
efficient binary parsers in a simple and declarative manner.
Until the *encoding* feature is merged in baseline of original project,
this branch is published under the name: **binary-parser-encoder** in [npm](https://npmjs.org/).

[![build](https://github.com/Ericbla/binary-parser/workflows/build/badge.svg)](https://github.com/Ericbla/binary-parser/actions?query=workflow%3Abuild)
[![npm](https://img.shields.io/npm/v/binary-parser-encoder)](https://www.npmjs.com/package/binary-parser-encoder)

Binary-parser is a parser/encoder builder for JavaScript that enables you to write
efficient binary parsers/encoders in a simple and declarative manner.

It supports all common data types required to analyze a structured binary
data. Binary-parser dynamically generates and compiles the parser code
on-the-fly, which runs as fast as a hand-written parser (which takes much more
data. Binary-parser dynamically generates and compiles the parser and encoder code
on-the-fly, which runs as fast as a hand-written parser/encoder (which takes much more
time and effort to write). Supported data types are:

- [Integers](#uint8-16-32-64le-bename-options) (8, 16, 32 and 64 bit signed
Expand All @@ -32,12 +38,14 @@ and [binary](https://github.com/substack/node-binary).
## Quick Start

1. Create an empty `Parser` object with `new Parser()` or `Parser.start()`.
2. Chain methods to build your desired parser. (See [API](#api) for detailed
2. Chain methods to build your desired parser and/or encoder. (See [API](#api) for detailed
documentation of each method)
3. Call `Parser.prototype.parse` with a `Buffer`/`Uint8Array` object passed as
its only argument.
4. The parsed result will be returned as an object.
- If parsing failed, an exception will be thrown.
5. Or call `Parser.prototype.encode` with an object passed as argument.
6. Encoded result will be returned as a `Buffer` object.

```javascript
// Module import
Expand Down Expand Up @@ -73,6 +81,23 @@ const buf = Buffer.from("450002c5939900002c06ef98adc24f6c850186d1", "hex");

// Parse buffer and show result
console.log(ipHeader.parse(buf));

var anIpHeader = {
version: 4,
headerLength: 5,
tos: 0,
packetLength: 709,
id: 37785,
offset: 0,
fragOffset: 0,
ttl: 44,
protocol: 6,
checksum: 61336,
src: [ 173, 194, 79, 108 ],
dst: [ 133, 1, 134, 209 ] };

// Encode an IP header object and show result as hex string
console.log(ipHeader.encode(anIpHeader).toString("hex"));
```

## Installation
Expand All @@ -87,14 +112,22 @@ The npm package provides entry points for both CommonJS and ES modules.

## API

### new Parser()
### new Parser([options])
Create an empty parser object that parses nothing.
`options` is an optional object to pass options to this declarative
parser.
- `smartBufferSize` The chunk size of the encoding (smart)buffer (when encoding is used) (default is 256 bytes).

### parse(buffer)
Parse a `Buffer`/`Uint8Array` object `buffer` with this parser and return the
resulting object. When `parse(buffer)` is called for the first time, the
associated parser code is compiled on-the-fly and internally cached.

### encode(obj)
Encode an `Object` object `obj` with this parser and return the resulting
`Buffer`. When `encode(obj)` is called for the first time, encoder code is
compiled on-the-fly and internally cached.

### create(constructorFunction)
Set the constructor function that should be called to create the object
returned from the `parse` method.
Expand Down Expand Up @@ -151,12 +184,23 @@ the following keys:
- `length ` - (Optional) Length of the string. Can be a number, string or a
function. Use number for statically sized arrays, string to reference
another variable and function to do some calculation.
Note: When encoding the string is padded with a `padd` charecter to fit the length requirement.
- `zeroTerminated` - (Optional, defaults to `false`) If true, then this parser
reads until it reaches zero.
reads until it reaches zero (or the specified `length`). When encoding, a *null* character is inserted at end of
the string (if the optional `length` allows it).
- `greedy` - (Optional, defaults to `false`) If true, then this parser reads
until it reaches the end of the buffer. Will consume zero-bytes.
until it reaches the end of the buffer. Will consume zero-bytes. (Note: has
no effect on encoding function)
- `stripNull` - (Optional, must be used with `length`) If true, then strip
null characters from end of the string.
null characters from end of the string. (Note: When encoding, this will also set the **default** `padd` character
to null instead of space)
- `trim` - (Optional, default to `false`) If true, then trim() (remove leading and trailing spaces)
the parsed string.
- `padding` - (Optional, Only used for encoding, default to `right`) If `left` then the string
will be right aligned (padding left with `padd` char or space) depending of the `length` option
- `padd` - (Optional, Only used for encoding with `length` specified) A string from which first character (1 Byte)
is used as a padding char if necessary (provided string length is less than `length` option). Note: Only 'ascii'
or utf8 < 0x80 are alowed. Note: The default padd character is *space* (or *null* when `stripNull` is used).

### buffer(name[, options])
Parse bytes as a buffer. Its type will be the same as the input to
Expand All @@ -175,7 +219,8 @@ keys:
calculation.
- `readUntil` - (either `length` or `readUntil` is required) If `"eof"`, then
this parser will read till it reaches the end of the `Buffer`/`Uint8Array`
object. If it is a function, this parser will read the buffer until the
object. (Note: has no effect on encoding.)
If it is a function, this parser will read the buffer until the
function returns true.

### array(name, options)
Expand All @@ -194,6 +239,12 @@ keys:
- `readUntil` - (either `length`, `lengthInBytes`, or `readUntil` is required)
If `"eof"`, then this parser reads until the end of the `Buffer`/`Uint8Array`
object. If function it reads until the function returns true.
**<u>Note</u>**: When encoding,
the `buffer` second parameter of `readUntil` function is the buffer already encoded
before this array. So no *read-ahead* is possible.
- `encodeUntil` - a function (item, object), only used when encoding, that replaces
the `readUntil` function when present and allow limit the number of encoded items
by returning true based on *item* values or other *object* properies.

```javascript
const parser = new Parser()
Expand Down Expand Up @@ -331,7 +382,7 @@ const parser = new Parser()
### seek(relOffset)
Move the buffer offset for `relOffset` bytes from the current position. Use a
negative `relOffset` value to rewind the offset. This method was previously
named `skip(length)`.
named `skip(length)`. (Note: when encoding, the skipped bytes will be filled with zeros)

### endianness(endianness)
Define what endianness to use in this parser. `endianness` can be either
Expand All @@ -348,6 +399,21 @@ const parser = new Parser()
.int32("c");
```

### encoderSetOptions(opts)
Set specific options for encoding.
Current supported `opts` object may contain:
- bitEndianess: true|false (default false) When true, tell the encoder to respect endianess BITs order, so that
encoding is exactly the reverse of the parsing process for bits fields.

```javascript
var parser = new Parser()
.endianess("little")
.encoderSetOptions({bitEndianess: true}) // Use BITs endianess for bits fields
.bit4("a")
.bit4("b")
.uint16("c");
```

### namely(alias)
Set an alias to this parser, so that it can be referred to by name in methods
like `.array`, `.nest` and `.choice`, without the requirement to have an
Expand Down Expand Up @@ -495,26 +561,44 @@ mainParser.parse(buffer);
Returns how many bytes this parser consumes. If the size of the parser cannot
be statically determined, a `NaN` is returned.

### compile()
### compile() and compileEncode()
Compile this parser on-the-fly and cache its result. Usually, there is no need
to call this method directly, since it's called when `parse(buffer)` is
to call this method directly, since it's called when `parse(buffer)` or `encode(obj)` is
executed for the first time.

### getCode()
Dynamically generates the code for this parser and returns it as a string.
### getCode() and getCodeEncode()
Dynamically generates the code for this parser/encoder and returns it as a string.
Useful for debugging the generated code.

### Common options
These options can be used in all parsers.

- `formatter` - Function that transforms the parsed value into a more desired
form.
form. *formatter*(value, obj, buffer, offset) &rarr; *new value* \
where `value` is the value to be formatted, `obj` is the current object being generated, `buffer` is the buffer currently beeing parsed and `offset` is the current offset in that buffer.
```javascript
const parser = new Parser().array("ipv4", {
type: uint8,
length: "4",
formatter: function(arr, obj, buffer, offset) {
return arr.join(".");
}
});
```

- `encoder` - Function that transforms an object property into a more desired
form for encoding. This is the opposite of the above `formatter` function. \
*encoder*(value) &rarr; *new value* \
where `value` is the value to be encoded (de-formatted) and `obj` is the object currently being encoded.
```javascript
const parser = new Parser().array("ipv4", {
type: uint8,
length: "4",
formatter: function(arr) {
formatter: function(arr, obj, buffer, offset) {
return arr.join(".");
},
encoder: function(str, obj) {
return str.split(".");
}
});
```
Expand Down
Loading