fix d3-color and potential security issue

[ananzh]

Description

Addresses potential ReDoS issue from d3-color version < 3.1.0

d3-color is used as a sub-dependency of elastic/chart package. since there is no function change form 1.4 to 3.1.0, we could add resolution in package.json to overwrite the version.

yarn why d3-color
yarn why v1.22.19
[1/4] Why do we have the module "d3-color"...?
[2/4] Initialising dependency graph...
warning Resolution field "typescript@4.0.2" is incompatible with requested version "typescript@~4.5.2"
[3/4] Finding dependency...
[4/4] Calculating file sizes...
=> Found "d3-color@1.4.1"
info Has been hoisted to "d3-color"
info Reasons this module exists
   - "workspace-aggregator-26115f01-dd15-42ff-8fa9-cd8d075aae02" depends on it
   - Hoisted from "_project_#@elastic#charts#d3-color"
   - Hoisted from "_project_#@elastic#charts#d3-interpolate#d3-color"
   - Hoisted from "_project_#@elastic#charts#d3-scale#d3-color"
   - Hoisted from "_project_#@elastic#charts#d3-scale#d3-interpolate#d3-color"

Check List

• All tests pass
  • yarn test:jest
  • yarn test:jest_integration
  • yarn test:ftr
• New functionality includes testing.
• New functionality has been documented.
• Update CHANGELOG.md
• Commits are signed per the DCO using --signoff

[kavilla]

Looks like it's require snapshot update?

[ananzh]

Looks like it's require snapshot update?

Not just snapshot. but see a lot of

Jest SyntaxError: Unexpected token 'export' 

due to d3-color pre-compiled source code. fix that. Then see locally some failing tests are from src/plugins/data_source. since this is a security issue and will backport to 1.x. I think we should have another issue or PR to fix the following unit tests. what do you think? @kavilla

Summary of all failing tests
 FAIL  src/plugins/data_source/server/cryptography/cryptography_client.test.ts
  ● Test encrpyt and decrypt module › Positive test cases › Encrypt and Decrypt with same in memory keyring

    Unsupported plaintext

      55 |    */
      56 |   public async encryptAndEncode(plainText: string): Promise<string> {
    > 57 |     const result = await this.encrypt(this.keyring, plainText);
         |                               ^
      58 |     return result.result.toString(ENCODING_STRATEGY);
      59 |   }
      60 |

      at Object.encrypt (node_modules/@aws-crypto/encrypt-node/src/encrypt.ts:54:11)
      at CryptographyClient.encryptAndEncode (src/plugins/data_source/server/cryptography/cryptography_client.ts:57:31)
      at Object.<anonymous> (src/plugins/data_source/server/cryptography/cryptography_client.test.ts:32:50)

  ● Test encrpyt and decrypt module › Positive test cases › Encrypt and Decrypt with two different keyrings with exact same identifiers

    Unsupported plaintext

      55 |    */
      56 |   public async encryptAndEncode(plainText: string): Promise<string> {
    > 57 |     const result = await this.encrypt(this.keyring, plainText);
         |                               ^
      58 |     return result.result.toString(ENCODING_STRATEGY);
      59 |   }
      60 |

      at Object.encrypt (node_modules/@aws-crypto/encrypt-node/src/encrypt.ts:54:11)
      at CryptographyClient.encryptAndEncode (src/plugins/data_source/server/cryptography/cryptography_client.ts:57:31)
      at Object.<anonymous> (src/plugins/data_source/server/cryptography/cryptography_client.test.ts:42:51)

  ● Test encrpyt and decrypt module › Negative test cases › Encrypt and Decrypt with different key names

    Unsupported plaintext

      55 |    */
      56 |   public async encryptAndEncode(plainText: string): Promise<string> {
    > 57 |     const result = await this.encrypt(this.keyring, plainText);
         |                               ^
      58 |     return result.result.toString(ENCODING_STRATEGY);
      59 |   }
      60 |

      at Object.encrypt (node_modules/@aws-crypto/encrypt-node/src/encrypt.ts:54:11)
      at CryptographyClient.encryptAndEncode (src/plugins/data_source/server/cryptography/cryptography_client.ts:57:31)
      at Object.<anonymous> (src/plugins/data_source/server/cryptography/cryptography_client.test.ts:65:51)

  ● Test encrpyt and decrypt module › Negative test cases › Encrypt and Decrypt with different key namespaces

    Unsupported plaintext

      55 |    */
      56 |   public async encryptAndEncode(plainText: string): Promise<string> {
    > 57 |     const result = await this.encrypt(this.keyring, plainText);
         |                               ^
      58 |     return result.result.toString(ENCODING_STRATEGY);
      59 |   }
      60 |

      at Object.encrypt (node_modules/@aws-crypto/encrypt-node/src/encrypt.ts:54:11)
      at CryptographyClient.encryptAndEncode (src/plugins/data_source/server/cryptography/cryptography_client.ts:57:31)
      at Object.<anonymous> (src/plugins/data_source/server/cryptography/cryptography_client.test.ts:84:51)

  ● Test encrpyt and decrypt module › Negative test cases › Encrypt and Decrypt with different wrapping keys

    Unsupported plaintext

      55 |    */
      56 |   public async encryptAndEncode(plainText: string): Promise<string> {
    > 57 |     const result = await this.encrypt(this.keyring, plainText);
         |                               ^
      58 |     return result.result.toString(ENCODING_STRATEGY);
      59 |   }
      60 |

      at Object.encrypt (node_modules/@aws-crypto/encrypt-node/src/encrypt.ts:54:11)
      at CryptographyClient.encryptAndEncode (src/plugins/data_source/server/cryptography/cryptography_client.ts:57:31)
      at Object.<anonymous> (src/plugins/data_source/server/cryptography/cryptography_client.test.ts:103:51)


Test Suites: 1 failed, 1 skipped, 1495 passed, 1496 of 1497 total
Tests:       5 failed, 24 skipped, 9 todo, 11329 passed, 11367 total
Snapshots:   2504 passed, 2504 total
Time:        136.72 s

[codecov-commenter]

Codecov Report

Merging #2454 (a0f54f9) into main (a528965) will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #2454   +/-   ##
=======================================
  Coverage   66.75%   66.75%           
=======================================
  Files        3200     3200           
  Lines       60886    60886           
  Branches     9250     9250           
=======================================
+ Hits        40644    40646    +2     
+ Misses      18030    18028    -2     
  Partials     2212     2212           

Impacted Files	Coverage Δ
...ared/static/forms/hook_form_lib/hooks/use_field.ts	66.66% <0.00%> (+0.96%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[ananzh]

I only see the above 5 failed unit tests locally
CI doesn't catch this.

[seraphjiang]

@noCharger i saw some src/plugins/data_source UT failed, could you take a look

[joshuarrrr]

looks good, just a couple small questions.

[ananzh]

add a tracking issue for the failed local unit tests:
#2459

[noCharger]

@noCharger i saw some src/plugins/data_source UT failed, could you take a look

will do

[opensearch-trigger-bot]

The backport to 1.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.x 1.x
# Navigate to the new working tree
cd .worktrees/backport-1.x
# Create a new branch
git switch --create backport/backport-2454-to-1.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 d802a3ef5b3c75418e2cf4d5f480e48274842dfd
# Push it to GitHub
git push --set-upstream origin backport/backport-2454-to-1.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.x

Then, create a pull request where the base branch is 1.x and the compare/head branch is backport/backport-2454-to-1.x.

[opensearch-trigger-bot]

The backport to 1.3 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.3 1.3
# Navigate to the new working tree
cd .worktrees/backport-1.3
# Create a new branch
git switch --create backport/backport-2454-to-1.3
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 d802a3ef5b3c75418e2cf4d5f480e48274842dfd
# Push it to GitHub
git push --set-upstream origin backport/backport-2454-to-1.3
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.3

Then, create a pull request where the base branch is 1.3 and the compare/head branch is backport/backport-2454-to-1.3.

[joshuarrrr]

not backportable to any 1.x due to node version conflicts