`include` the same token does not do a "deep merge"?

[chris-dura]

From Docs:

Style Dictionary takes all the files it finds in the include and source arrays and performs a deep merge on them.

I do not know if what I'm seeing is a bug, or a misunderstanding of what "deep merge" means to StyleDictionary...

I assumed "deep merge" meant that metadata fields, like $extensions or $description, that were NOT overridden would still be in the output... IOW, "inherited" by the files later in the precedence order.

However, this does not seem to be the case if the same token, stored in two separate files, is listed in the include config...

/* default.tokens.json */

{
  "colors": {
    "primary": {
      "$value": "#ff0000",
      "$type": "color",
      "$description: "The primary color."
      "$extensions": {
          "com.foo": true
      }
    }
  }
}

/* overrides.tokens.json */

{
  "colors": {
    "primary": {
      "$value": "#0000ff",
      "$type": "color"
    }
  }
}

/* config.json */

{
  "include": [
    "default.tokens.json",
    "overrides.tokens.json
  ],
  "platforms": {
    "json": {
      "files": [
        {
          "destination": "tokens.json",
          "format": "json"
        }
      ]
    }
  }
}

/* output.tokens.json */

{
  "colors": {
    "primary": {
      "$value": "#0000ff",
      "$type": "color",
      "filePath": "overrides.tokens.json",
      "isSource": true,
      "original": {
        "$value": "#0000ff",
        "$type": "color"
      },
      "name": "primary",
      "attributes": {},
      "path": [
        "colors",
        "primary"
      ]
    }
  }
}

/* Expected output.tokens.json */

{
  "colors": {
    "primary": {
      "$value": "#0000ff",
      "$type": "color",
      "$description": "The primary color.",
      "$extensions": {
        "com.foo": true
      },
      "filePath": "overrides.tokens.json",
      "isSource": true,
      "original": {
        "$value": "#0000ff",
        "$type": "color",
        "$description": "The primary color.",
        "$extensions": {
          "com.foo": true
        }
      },
      "name": "primary",
      "attributes": {},
      "path": [
        "colors",
        "primary"
      ]
    }
  }
}

However, if I move the overrides.tokens.json to source, instead of include, the output is what I would expect.

/* config.json */

{
  "include": [
    "default.tokens.json"
  ],
  "source": [
    "overrides.tokens.json"
  ],
  ...
}

This seems perhaps to be the intended usage of source; however, the problem arises when you have multiple overrides for the same token in source, for example when trying to generate "derivative" themes, or dark modes, because a glut of "red herring" Collision warnings are generated.

/* config.json */

{
  "include": [
    "default.tokens.json"
  ],
  "source": [
    "overrides.tokens.json",
    "dark.tokens.json"
  ],
  ...
}

/* dark.tokens.json */

{
  "colors": {
    "primary": {
      "$value": "#ffffff",
      "$type": "color"
    }
  }
}

Token collisions detected (4):

Collision detected at: colors.primary! Original value: #0000ff, New value: #ffffff
Collision detected at: colors.primary! Original value: color, New value: color
Collision detected at: colors.primary! Original value: dark.tokens.json, New value: dark.tokens.json
Collision detected at: colors.primary! Original value: true, New value: true

Before I go silencing ALL collision warnings (some of which might actually be wanted ☹️), I wanted to make sure that the above is expected behavior, and in order to inherit metadata, then ALL my "override" files need to be in the source config?

• Tangentially related logging-level issue: Suppress filtered out reference warnings at the getReferences() level? #1287

[hadrien-xbto]

👋
Im having a similar kind of issue with collision warning, and I was wondering if there is a hook or action, that could be used to handle those collision on a case by case basis ?

In my case, I have a desktop.json file and a mobile.json files, where most tokens are in fact intentional duplicates (which I'd like to skip), and some of the duplicates should trigger the creation of a namespacing in my output tokens for instance.

Is that something that seems achievable with the current state of SD library ?

[chris-dura]

@hadrien-xbto -- In the current state of the logging configuration, probably not...

I think there's been some discussion about making logging configuration more granular, which could help your scenario a little bit maybe. For example, eventually, maybe you'd be able to silence Collision warnings, but not Broken Reference warnings... kind of like "per-rule" configuration in ESlint?

However, I think it'd end up being overly-complex to enable "ignore THIS collision warning, but not this OTHER collision warning", so, not sure the SD team would put that much time into reworking the logging architecture.

The best I've been able to do in current state is configure logging on a per-Platform basis... so, if there is a particular scenario that I wanted to ignore Collision Warnings, I'd just put that scenario in its own "platform" and turn off the logging for that platform... but currently, it's still an all-or-none scenario... you cannot distinguish one collision from another collision.

[hadrien-xbto]

@chris-dura Have you explored custom parsers ? I was wondering if it could be used to import the JSON file and add namespaces for each imported file, then manage the tokens on my own afterwards with a combination of transforms and formats. But although I managed to have my own parser, as soon as I add a namespace, it breaks all the references like that :

mobile-color.input.border.default.$value tries to reference color.border.primary, which is not defined.

[jorenbroekema]

Alright so there's quite a couple of things here, and perhaps there should be a deep-dive section in the docs explaining this in more detail.

I do not know if what I'm seeing is a bug, or a misunderstanding of what "deep merge" means to StyleDictionary...

As it turns out, both 😅 . The bug is that the "include" vs "source" do not currently behave the exact same way, also when combining the two. This is because sets within include OR source are merged through the "combineJSON" util which uses the deepExtend util. However, when combining the include & source objects together, the deepExtend util is used directly with a slightly different configuration, these need to be aligned. So I'm labelling this issue as a bug for this.

The misunderstanding lies in how token level collisions are handled, your "actual" output is "correct".
When you have a token collision between 2 files, the file later in the cascade "wins". What we mean with "win" aka "override" is that this token in its entirety should override the other one, without its properties getting mingled together; the latter is usually unintended and unwanted. E.g. token A's $extensions metadata, it may contain some information about token A, but that doesn't mean this metadata should be copied/merged into token B when token B overrides token A, token A's metadata doesn't belong to token B after all. In summary: token groups get deep-merged, tokens get overridden.

As for the "original" property, this is constructed to keep track of the token's original state after being parsed and preprocessed. It's there as a reference to a token's pre-transformed state, but merging the separate token files into a single dictionary is considered its original/initial state. We don't keep track currently of whether a token has collided after the fact, in the parsing lifecycle.

Token collisions detected (4):

Collision detected at: colors.primary! Original value: #0000ff, New value: #ffffff
Collision detected at: colors.primary! Original value: color, New value: color
Collision detected at: colors.primary! Original value: dark.tokens.json, New value: dark.tokens.json
Collision detected at: colors.primary! Original value: true, New value: true

This should be a single collision warning, since it's only 1 token. It's throwing a warning for every property of the token 😓 this is definitely a bug of which I'm not really sure there's an open issue yet. Should be relatively easy to fix though.

@hadrien-xbto

Im having a similar kind of issue with collision warning, and I was wondering if there is a hook or action, that could be used to handle those collision on a case by case basis

There's isn't something at the moment to hook into the collision behavior, at least not in the public API of Style Dictionary. Internally we have a collision callback that we call to throw those collision warnings you see in the console, and there's an overrideKeys option we use internally to ensure that token "value" properties don't get merged together, e.g. if Token A and Token B are both typography values (object type), you don't want deep-merging behavior of the typography object values, you just want Token B to take precedence fully.
I'm not super keen on making these public unless I see a good use case for it.

There's an open issue discussing how tokensets can be layered/composed in a way which lets you silence collision warnigns if the collisions are intended. E.g. a collision in 1 layer with a token in another would be "intended", but if such a collision happens within the same layer, it would be a warning. #1170 I would love to get more feedback on this one.

I think there's been some discussion about making logging configuration more granular, which could help your scenario a little bit maybe. For example, eventually, maybe you'd be able to silence Collision warnings, but not Broken Reference warnings... kind of like "per-rule" configuration in ESlint?

https://styledictionary.com/reference/logging/ It's pretty granular now, warnings can be turned off, unfortunately not so granular yet that you can pick and choose which warnings exactly are turned off (or warn, or throw). That should be easy to add though, feel free to raise a feature request. We started going in that direction already by adding granular control for the broken references errors to be thrown non-fatally, and we can follow it for warnings too, and perhaps also some of the other errors that don't have to be fatal (because we can think of fallback behavior that makes sense).

However, I think it'd end up being overly-complex to enable "ignore THIS collision warning, but not this OTHER collision warning", so, not sure the SD team would put that much time into reworking the logging architecture.

Yeah, on a per-collision basis is probably overly granular and complex, but I can imagine with correct layering as proposed here #1170 that this would be a way to achieve that, marking certain sets relations as being "okay to collide" but same-layer sets "not okay to collide".

[chris-dura]

@jorenbroekema -- thanks for your responses!

However, when combining the include & source objects together, the deepExtend util is used directly with a slightly different configuration, these need to be aligned. So I'm labelling this issue as a bug for this.

When you align these, will there be a mechanism to choose between "merge" (keep metadata) vs "clobber" (default, overwrite everything)?

You mention...

There's an open issue discussing how tokensets can be layered/composed in a way which lets you silence collision warnigns if the collisions are intended

Naively, I'm inferring that if a collision is "intended", that it might be preferred that collision does a "merge" instead of a "clobber"?

[chris-dura]

Unfortunately, I think I completely missed this point, and thought I could get away with adding everything to source, but it didn't work when ONLY source files were used...

In summary: token groups get deep-merged, tokens get overridden.

@jorenbroekema -- so, just to confirm, does this mean it is expected that if I want to write a $description for a semantic token, I will need to duplicate that $description field in every file that overrides that semantic token?

/* config.json */

{
  "source": [
    "base.tokens.json",
    "override1.tokens.json",
    "override2.tokens.json",
    "dark.tokens.json"
  ],
}

/* base.tokens.json */
{
  "primary": {
    "$value": "#ff0000",
    "$description": "The primary color."
  }
}

/* override1.tokens.json */
{
  "primary": {
    "$value": "#00ff00",
    "$description": "The primary color." /* WET */
  }
}

/* override2.tokens.json */
{
  "primary": {
    "$value": "#0000ff",
    "$description": "The primary color." /* WET */
  }
}

/* dark.tokens.json */
{
  "primary": {
    "$value": "#ffffff",
    "$description": "The primary color." /* WET */
  }
}

[jorenbroekema]

so, just to confirm, does this mean it is expected that if I want to write a $description for a semantic token, I will need to duplicate that $description field in every file that overrides that semantic token?

Yes, correct, from a design philosophy perspective, tokens are generally "isolated" (although can reference other tokens), so even if a token is targeting to override another, that coupling is not something the tokens should rely on individually.

will there be a mechanism to choose between "merge" (keep metadata) vs "clobber" (default, overwrite everything)?

No, it will be "clobber" always for the token level, and "merge" for token groups.

I'd have to see a strong use case in order to change my mind and make this configurable.
Generally speaking, we try to avoid token overrides by using "theming", we're currently working on a sister spec to the DTCG spec for theming / defining tokensets but what it comes down to in your example is that your JSON files belong to different theme variants, so in one permutation you may have dark, while light in the other, and you have a separate SD instance for each. So the tokens don't override one another, because which JSON set is active depends on the themes setup.
https://resolver-spec.netlify.app/ this is very WIP and the API is subject to change, significantly, and the docs are a bit lacking still, but this is the general direction we're going into.

[hadrien-xbto]

@jorenbroekema Here is our usecase :

• Most of our tokens are common between web and mobile (colors, spacing, ...)
• fontsize and lineheight have different base sizes
• Each Figma for web and mobile export its own set of tokens, resulting in duplication (same values) and duplication with different values
• We are building universal components, meaning those distinct values must both be available in the tokens to be able to pick the necessary one for each platform

For now, I have found a workaround (that creates its own set of problems though) : when loading the JSON file, I namespace the contents of .desktop and .mobile files, then do the merging part manually to be able to export values like {web: 24px, mobile: 20px}

Is it a weird usecase and we could architecture things differently, or it seems legitimate to you ?

[chris-dura]

... we're currently working on a sister spec to the DTCG spec for theming / defining tokensets but what it comes down to in your example is that your JSON files belong to different theme variants, so in one permutation you may have dark, while light in the other, and you have a separate SD instance for each.

Yeah, this will be interesting... I've struggled coalescing the concepts of "theme or theme variant" vs "mode"... they can either be treated as completely separate entities or basically the same. In our current setup, "modes" are "subsets" of the higher-level "themes", and are "combinatorial". Most folks pigeon-hole "modes" to be "dark/light", but we currently have 3 "modes" (with a handful more to come) in our system...

• dark | light
• compact | cozy | comfy
• reduce-motion

So, you can't apply BOTH the maroon.theme AND the orange.theme, but you certainly could have both dark and cozy "modes" applied 🤷🏻

Generally speaking, we try to avoid token overrides by using "theming",

Yeah, I guess what I'm butting up against is more about "token extension" than "token overriding" -- or, I just stop thinking about "modes" as being "subsets" -- because if modes ARE subsets, I think it is inherent (implied?) that a "dark mode" (as opposed to a "dark theme") would necessarily inherit some token aspects from its "parent theme".

I mean, heck... we just need to look at the "Appearance Settings" in GitHub vs GitLab to see a concrete representation of completely different opinions on "theme" vs "mode" 😅 😅

... anyway, I digress, thanks again for the input and I'll keep an eye on that Resolver!

[chris-dura]

No, it will be "clobber" always for the token level, and "merge" for token groups.

As is prone to happen over the weekend, I thought more about this... when thinking about "semantic" tokens and the DTCG properties, I still feel like "clobber/overriding" is the 20% use case, and 80% would be "merging / mingling" overrides, at least for "intended" collisions?

If there is an intended overriding of a semantic token, usually, the goal is to update the $value, and that's it.

The chief purpose of semantic-level tokens is to define an agnostic intent... color.primary is still color.primary whether you're in a "red theme" or "dark mode". The "meaning/purpose" of the token isn't likely going to change between overrides or contexts... otherwise, you'd likely name it something other than color.primary.

So, when looking at the $ metadata fields, which are primarily used to describe additional "meaning" of the token, not just the $value... How likely is it that someone will want to change anything other than the $value? It would be very unlikely to change the $type of a token in dark mode, and even $description would be not critical? And, $extensions (if following the DTCG spec's recommendation) will likely be specifically namespaced already, so collisions and pollution there will be somewhat mitigated.

/* base.tokens.json */
{
  "primary": {
    "$value": "#ff0000",
    "$type": "color",
    "$description": "The primary color.",
    "$extensions": {
      "theme.A": {
        foo: true
      }
    }
  }
}

/* dark.tokens.json */
{
  "primary": {
    "$value": "#00000",
    "$extensions": {
      "theme.B": {
        foo: false
      }
    }
  }
}

The only potential "gotcha" with "merging" in this (typical?) case is mitigated by the implicit/recommended namespacing in the $extensions, but again, I would think it's safe to assume that theme.B would just ignore theme.A scoped properties 🤷🏻

/* output.tokens.json */

{
  "primary": {
    "$value": "#00000",
    "$type": "color",
    "$description": "The primary color.",
    "$extensions": {
      "theme.A": {
        foo: true
      },
      "theme.B": {
        foo: false
      }
    }
  }
}

Anyway -- I think I've come to the point where token "overrides" sit firmly in the "merging" camp instead of "clobber". Even outside of semantic tokens, at the primitive and component-level, the majority use case for intended collisions/overrides is to simply change the $value... not adjust the secondary $type, $description or $extensions fields. And staying DRY in the tokens source provides a non-trivial value for easing the maintenance of any tokensets that get beyond a few hundred tokens.

[jorenbroekema]

Is it a weird usecase and we could architecture things differently, or it seems legitimate to you ?

I would create a theme dimension called "density" -> "mobile" vs "web" (or "compact" or "dense" or "sparse"), so following the multi-dimensional theming approach that's also used in the resolver spec and these SD examples:

• https://github.com/amzn/style-dictionary/tree/main/examples/advanced/multi-brand-multi-platform
• https://github.com/tokens-studio/lion-example

[chris-dura]

Possibly related...

I noticed there's another merge behavior I don't quite understand, when a token $value is an array, like cubicBezier, it seems that the "order" of the source paths can coincidentally affect the final output where the array overwrites the previous $value, even when the source doesn't include the file with the array in it.

I've reproduced the issue by contriving an example where I simply duplicate the paths multiple times. It's probably best practice to avoid duplicate paths, but I'm curious about the behavior, because if the duplicate paths were the root issue, I would expect it to happen with other $value types, not just arrays.

// tokens/globals/index.js
export default {
  border: {
    style: {
      dashed: {
        $value: "globals-dashed",
      },
    },
  },
};

// tokens/brands/brand-1/index.js
export default {
  border: {
    style: {
      dashed: {
        $value: "brand-1-dashed"
      },
    },
  },
};

// tokens/brands/brand-2/index.js
export default {
  border: {
    style: {
      dashed: {
        $value: "brand-2-dashed"
      },
    },
  },
};

// config.mjs
export const getConfig = (brand) => {

  const cfg = {
    source: [
      `tokens/globals/**/*.js`,
      `tokens/brands/${brand}/index.js`,

      // Just duplicate the paths to emulate generation via a loop,
      // similar to the multi-brand multi-platform example.
      `tokens/globals/**/*.js`,
      `tokens/brands/${brand}/**/*.js`,
    ],
    platforms: {
      "json": {
        files: [
          {
            destination: `output-${brand}.json`,
            format: 'json',
          },
        ],
      },
    },
  }

  return cfg;
};

// build-tokens.mjs
["brand-1", "brand-2"].map((brand) => {
  const cfg = getConfig(brand);
  const sd = new StyleDictionary(cfg);
  sd.buildAllPlatforms();
});

// output-brand-1.json
{
  "border": {
    "style": {
      "dashed": {
        "$value": "brand-1-dashed",
        "filePath": "tokens/brands/brand-1/index.js",
        "isSource": true,
        "original": {
          "$value": "brand-1-dashed"
        },
        "name": "dashed",
        "attributes": {},
        "path": [
          "border",
          "style",
          "dashed"
        ]
      }
    }
  }
}

// output-brand-2.json
{
  "border": {
    "style": {
      "dashed": {
        "$value": "brand-2-dashed",
        "filePath": "tokens/brands/brand-2/index.js",
        "isSource": true,
        "original": {
          "$value": "brand-2-dashed"
        },
        "name": "dashed",
        "attributes": {},
        "path": [
          "border",
          "style",
          "dashed"
        ]
      }
    }
  }
}

When I change brand-2 to use an cubicBezier array, the output-brand-2.json changes, but so does the output-brand-1.json.

// tokens/brands/brand-2/index.js
export default {
  border: {
    style: {
      dashed: {
        $value: [0.4, 0, 0.2, 1]
      },
    },
  },
};

// output-brand-1.json
{
  "border": {
    "style": {
      "dashed": {
        "$value": [
          0.4,
          0,
          0.2,
          1
        ],
        "filePath": "tokens/brands/brand-1/index.js",
        "isSource": true,
        "original": {
          "$value": [
            0.4,
            0,
            0.2,
            1
          ]
        },
        "name": "dashed",
        "attributes": {},
        "path": [
          "border",
          "style",
          "dashed"
        ]
      }
    }
  }
}

// output-brand-2.json
{
  "border": {
    "style": {
      "dashed": {
        "$value": [
          0.4,
          0,
          0.2,
          1
        ],
        "filePath": "tokens/brands/brand-1/index.js",
        "isSource": true,
        "original": {
          "$value": [
            0.4,
            0,
            0.2,
            1
          ]
        },
        "name": "dashed",
        "attributes": {},
        "path": [
          "border",
          "style",
          "dashed"
        ]
      }
    }
  }
}

Token collisions detected (9):

Collision detected at: border.style.dashed! Original value: globals-dashed, New value: brand-2-dashed
Collision detected at: border.style.dashed! Original value: tokens/brands/brand-2/index.js, New value: tokens/brands/brand-2/index.js
Collision detected at: border.style.dashed! Original value: true, New value: true
Collision detected at: border.style.dashed! Original value: tokens/brands/brand-2/index.js, New value: tokens/brands/brand-1/index.js
Collision detected at: border.style.dashed.$value! Original value: 0.4, New value: 0.4
Collision detected at: border.style.dashed.$value! Original value: 0, New value: 0
Collision detected at: border.style.dashed.$value! Original value: 0.2, New value: 0.2
Collision detected at: border.style.dashed.$value! Original value: 1, New value: 1
Collision detected at: border.style.dashed! Original value: tokens/brands/brand-1/index.js, New value: tokens/brands/brand-1/index.js

Commenting out the one duplicate globals path creates the expected proper values for output-brand-1.json and output-brand-2.json. The best solution is to ensure that there aren't any (accidentally) duplicated paths, but when generating configs in loops, I think that could probably happen fairly easily if you're not careful... and, I think the bigger smell is that it seems the duplicate paths are only a problem if you coincidentally have an array in the "wrong spot" in the source config.

  source: [
    `tokens/globals/index.js`,
    `tokens/brands/${brand}/index.js`,
    // `tokens/globals/index.js`,
    `tokens/brands/${brand}/index.js`,
  ],

[jorenbroekema]

Since you have this reproduction, would it be okay for you to create a github repo I can clone, or a stackblitz link or something? It would save me some time recreating your example. I'm not sure why this behavior is happening, I'd definitely have to investigate it.