Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve bulk Stix loader to support 100K+ relations loading in one bulk #6386

Closed
richard-julien opened this issue Mar 19, 2024 · 0 comments · Fixed by #6387
Closed

Improve bulk Stix loader to support 100K+ relations loading in one bulk #6386

richard-julien opened this issue Mar 19, 2024 · 0 comments · Fixed by #6387
Assignees
Labels
bug use for describing something not working as expected export Functional scope : EXPORT solved use to identify issue that has been solved (must be linked to the solving PR)
Milestone

Comments

@richard-julien
Copy link
Member

Description

Improve bulk Stix loader to support 100K+ relations loading in one bulk.
The error is due to a nodejs limitation where function doesnt support more than 100K parameters. This limitation throw the error "Maximum call size exceeded" when we use spread operators to add result in the global accumulation array.

Reproducible Steps

Steps to create the smallest reproducible scenario:

  1. Get a platform with lot of massive reports
  2. Create a taxii sharing focus on reports
  3. Consume the taxii collection

Expected Output

List of reports correctly exported.

Actual Output

Execution failure for "Maximum call size exceeded"

Additional information

Improvement is done with 2 different approaches.

  1. Accumulate the result in a callback instead of spread the complete list at the end.
  2. Improve nested connections internal filter to be based on query terms to prevent field explosions
@richard-julien richard-julien added the bug use for describing something not working as expected label Mar 19, 2024
@richard-julien richard-julien added this to the Release 6.0.8 milestone Mar 19, 2024
@richard-julien richard-julien self-assigned this Mar 19, 2024
@aHenryJard aHenryJard assigned aHenryJard and unassigned aHenryJard Mar 20, 2024
@Kedae Kedae added the solved use to identify issue that has been solved (must be linked to the solving PR) label Mar 20, 2024
@Jipegien Jipegien added the export Functional scope : EXPORT label Jul 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug use for describing something not working as expected export Functional scope : EXPORT solved use to identify issue that has been solved (must be linked to the solving PR)
Projects
None yet
4 participants