Add tokenMap props and convertTokenData utility

[jorenbroekema]

• Adjust / add docs for tokenMap property and flattenTokens / convertToTokenObject/Map utils
• Refactor internals away from flattenTokens as prep for deprecation
• Add TODOs where relevant for next PR to make full use of the tokenMap prop for perf gains
• Add changeset

For next PR:

• use tokenMap for transform hook (significant perf gains for large sets)
• use tokenMap for reference resolutions (huge perf gains for large sets especially with loads of chained refs)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[aws-amplify-us-west-1]

This pull request is automatically being deployed by Amplify Hosting (learn more).

Access this pull request here: https://pr-1397.d16eby4ekpss5y.amplifyapp.com

[Delagen]

@dbanksdesign this change introduces top-level token copy which can broke some use cases. convertTokenData process tokens the same way

This can be broken change after 4.2.0 version

[jorenbroekema]

to_ret.push(/** @type {Token} */ (slice[name]));

is equivalent to

result.push({ .../** @type {T} */ (slice[key]) })

given the token is an object type.. All we added is a key property to it.

I'm not sure why that would be a breaking change, could you elaborate or show an example?

[Delagen]

It's not the same, second is equivalent of Object.assign({}, slice[key])

It's not breaking change for me, but for someone may be.

for example

for(const token of flattenTokens(dictionary.tokens)) {
  // ...some modification of token
}

in 4.2 will affect real tokens in dictionary
in 4.3 will ignore top properties change, for example token.value

[jorenbroekema]

Hmm I see. The flattenTokens util should return a value that isn't coupled to the input value imo, so mutating the result should not mutate the input, this is unintentional and a bad practice if you ask me.

I would consider this an unintentional bugfix then; while people may rely on the "bug" (unintended behavior) in their code, it doesn't make the bugfix warrant a major bump.

[Delagen]

I understand that. I fixed it in my pipeline.

But to make function totally pure you should use full clone instead of just shallow clone.
Because current behavior allow of update for example subproperties of token, i.e attributes, properties of object values etc.

[jorenbroekema]

True, I think in a lot of places I'm already using structuredClone() but we should verify for any function that is doing some kind of token handling that these functions are pure. https://github.com/amzn/style-dictionary/blob/main/lib/utils/flattenTokens.js#L62 is a good example where it's not yet being done.
Created an issue for this #1412