- Improved writing operator compatibility
Add a typeNarrowing property when parsing to dictate when a value should be attempted to be narrowed to a more specific type (ie: number, boolean, etc). The values are "all" (default), "unquoted", and "none"
- Correctly author ESM types for
/slimusers - Bump target of output destined for bundlers to ES2022
- Expose additional types, previously unexposed
- Bump internal parser to handle additional text formats
write_hidden_object_starthas been replaced withstart_mixed_mode- One can now write the equals operator (to be used with mixed mode only though)
- Remove wee_alloc in favor of standard memory allocator
- Performance improvements in json and js object creation (avoid heap allocation when a grouped field consist of only one element)
- the duplicate key mode (previously
disambiguatevalues have been renamed):- none -> group
- keys -> preserve
- typed -> key-value-pairs
- Fix return type of
parseTextto be aRecord<string, any>instead of anObject - Fix return type of
parseTextwhen provided a callback
- Quoted keys will now retain quotes
- Written output no longer ends in a newline
- Fix botched release of 0.6.1
- Fix bundling to make the library easier to use with bundlers
- Fix possibility of double initialization
- Distribute library with conditional exports
- Main distributable is now a UMD package instead CJS
- Add a
jomini/slimentrypoint for downstream developers to customize the Wasm used in initialization
- Support deserializing dates with a zero year
- Support CK2 exact operator (==)
- Support CK3 not equal operator (!=)
- Support skipping over semicolons when parsing text
- Support parsing text with UTF-8 BOM
- Support parsing files with missing brackets
- Small performance improvement to text parsing
- Improve support for parsing operators
- Add support for decoding numbers with leading plus sign
- Fix improperly writing escaped quoted values that end with a quote
- Add low level write API. See readme for more info
- Skip creating empty objects values that don't have an associated key. The example below shows the new behavior
// foo={bar=val {}} { } me=you
{
foo: { bar: 'val' },
me: "you"
}- Internal method of
Query::freeis no longer exposed. - Minor parsing performance improvement
Support for prehistoric Eu4 Leviathan monument dates
Support for Ck3 interpolated variable syntax:
position = @[1-leopard_x]
Support for Eu4 parameter definition syntax:
each_estate_effect = {
if = {
limit = {
has_estate = estate_brahmins
}
[[effect] $effect$]
[[estate_action]
$estate_action$ = estate_brahmins
]
}
}
Allow extraneous closing brace at any point. Previously extraneous closing braces could only be at the end of the document (something seen in Vic II saves), but there are EU4 game files where an extraneous closing brace ocurrs in the middle (looking at you verona.txt):
a = { 1 }
}
b = 2
This release makes the parser purposely accept invalid documents in an effort to be as flexible as possible. Any file that is parseable by PDS should be parseable by us.
Fix inaccuracy in float parsing numbers in the range of (-1, 0). Previously
foo=-0.5
and
foo=0.5
would parse to the same value. This has been fixed.
Added the ability to properly expose a document with an object trailer:
area = { color = { 10 } 1 2 }
The trailing "1 2" is referred to as the object trailer and will now be
exposed as the trailer property:
area: { color: [10], trailer: [1, 2] }- Bug fixes to internal parser to properly handle more edge cases (unseen in the wild) instead of panicking
- Add option to prettify JSON output
- Add option to define how to deal with duplicate keys when generating JSON
- Enhance types on
Queryobject
Fix wasm not being bundled inside the JS distribution
New API: requesting the parsed file be converted to JSON:
const parser = await Jomini.initialize();
const jsonString = parser.parseText(
buffer,
{ encoding: "windows1252" },
(q) => q.json()
);The JSON will contain object keys in the order as they appear in the file (instead of it being browser dependent). The JSON API is fast too, producing the JSON and then parsing the JSON is still 3x faster than parsing the file directly to a JS object.
Additionally integers that can't losslessly represented by a 64 bit floating point number are now represented as strings instead of numbers with a loss of precision.
Support for parsing Victoria II saves that contain an extraneous closing brace
Support for parsing negative dates
Fix some typescript typing on optional arguments to parseText
Aggregate fields that are out of order under a single field
name=aaa
name=bbb
core=123
name=ccc
name=ddd
is parsed into:
{
name: ["aaa", "bbb", "ccc", "ddd"],
core: 123,
}Fix path to typescript types
10-20% parsing performance improvement
Fix parsing of the following format that can occur in EU4 saves:
history = {
{}
1689.10.2={
decision="abc123"
}
}
- Fix webpack compatibility
- Wasm inlined as base64 so browser users don't need to fetch an additional file
This library has been rewritten and hopefully by the end there'll be enough evidence to make the rewrite worth it.
The two major reasons why the library was rewritten:
- Performance: Prior versions of the library are unable to parse large save files, as this would often exhaust node's heap space and parsing could take up to 30 seconds.
- Accuracy: Some inputs would cause the parser to crash for unknown reasons and the parser generator used didn't allow enough insight to easily debug the situation.
So it was decided to adopt the parser that underpins Rakaly, the EU4 achievement leaderboard, and the Paradox Game Converters's conversion from ironman to plaintext.
Given the same input, the new parser is over 3x faster and uses a third of the memory. And since this parser is much more flexible, it should handle inputs that previous versions could not.
What could be a real game changer is that this version introduces a query api on the parsed input that allow this version to be 200x faster than previously.
So while one can compute the player's prestige from an EU4 save with:
const buffer = readFileSync(args[0]);
const parser = await Jomini.initialize();
const save = parser.parseText(buffer, { encoding: "windows1252" });
const player = save.player;
const prestige = save.countries[player].prestige;It is 40x faster to declare upfront what you want to extract from the save file.
const buffer = readFileSync(args[0]);
const parser = await Jomini.initialize();
const { player, prestige } = parser.parseText(
buffer,
{ encoding: "windows1252" },
(query) => {
const player = query.at("/player");
const prestige = query.at(`/countries/${player}/prestige`);
return { player, prestige };
}
);Couple of important changes:
- The encoding of byte data must be supplied so that the parser knows how to decode strings like "Jåhkåmåhkke"
- The parser is implemented in web assembly and so the runtime environment is restricted to Node 12+ and >90% of browsers
- The parser must be initialized before use. This is a one time cost that is automatically reused on future invocations.
And breaking changes:
color = rgb { 100 200 150 }
is now parsed to
out = {
color: { rgb: [100, 200, 150] },
}Same with hsv. This change has been done as the object / array header (rgb / hsv) can be arbitrary and new games introduce new headers. So in order to be more future proof, this change was made.
- Fix parsing to support members that have numeric keys and empty values (eg: 'unit={ 2={ } 14={ } }')
- Add typescript types
- Allow escaped quotes in string fields
- Support additional object operators.
has_level >= 2
The output is
{'has_level': { 'GREATER_THAN_EQUAL': 2 }}- Allow parsing of dashed identifiers
- Fix keys for rgb colors
- Handle Stellaris variables
- Handle Stellaris colors
Add compatibility with Stellaris (#1)
Remove extraneous packaged files that (#1)
A near complete rewrite with Jison that ditches the handwritten stream based approach taken previously for an all in one parser generator. While I believe this sacrifices performance, it makes up with it in accuracy. I couldn't test the two implementations because parser generator one was the only one that could parse all the files given.
- Added
toDateto the API - Implemented parsing of dates that contain hour information
- Implemented simplified
parseAPI - Implemented
toArrayto force properties to be an array - Dates are now stored as dates in object and not ISO 8601 strings
- Removed dependency on lodash