-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Security Solution] Added concurrency limits and request throttling t…
…o prebuilt rule routes (#209551) **Resolves: #208357 **Resolves: #208355 ## Summary To prevent possible OOM errors, we need to limit concurrent requests to prebuilt rule routes (see attached tickets for more details). - `installation/_perform` and `upgrade/_perform` endpoints - Concurrency is limited to one parallel call. If another call is made simultaneously, the server responds with 429 Too Many Requests. - On the front end, all rule install and upgrade operations are retried in case of a 429 response. This ensures proper handling when a user clicks multiple times an update or install rule buttons - `prebuilt_rules/_bootstrap` endpoint - Install prebuilt rules and endpoint packages sequentially instead of in parallel to prevent from having them both downloaded into memory simultaneously. - Added a 30-minute socket timeout to prevent the proxy from closing the connection while rule installation is in progress. - Introduced a `throttleRequests` wrapper, ensuring the endpoint handler is called only once when multiple concurrent requests are received. - The first request triggers the handler, while subsequent requests wait for the first one to complete and reuse its result. - This prevents costly prebuilt rule package installation from running in parallel. - Reusing the response ensures the frontend correctly invalidates cached prebuilt rule queries. Since concurrent frontend requests should receive the same installed package information, responding with 421 and using the retry logic as in cases above is not an option here because the second request would receive a package installation skipped response leading to no cache invalidation. - `installation/_review` and `upgrade/_review` endpoints - Concurrency is limited to one parallel call. If another call is made simultaneously, the server responds with 429 Too Many Requests. - On the front end, all rule install and upgrade operations are retried in case of a 429 response. This ensures proper handling when a user clicks multiple times an update or install rule buttons (cherry picked from commit c5557f3) # Conflicts: # x-pack/plugins/security_solution/public/detection_engine/rule_management/api/hooks/prebuilt_rules/capped_exponential_backoff.ts # x-pack/plugins/security_solution/public/detection_engine/rule_management/api/hooks/prebuilt_rules/retry_on_rate_limited_error.ts # x-pack/plugins/security_solution/public/detection_engine/rule_management/api/hooks/prebuilt_rules/use_bootstrap_prebuilt_rules.ts # x-pack/plugins/security_solution/public/detection_engine/rule_management/api/hooks/prebuilt_rules/use_perform_specific_rules_upgrade_mutation.ts # x-pack/plugins/security_solution/public/detection_engine/rule_management/api/hooks/use_bootstrap_prebuilt_rules.ts # x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/bootstrap_prebuilt_rules/bootstrap_prebuilt_rules_handler.ts # x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/perform_rule_upgrade_route.ts # x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/review_rule_installation/review_rule_installation_handler.ts # x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/review_rule_installation/review_rule_installation_route.ts # x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/review_rule_upgrade/review_rule_upgrade_handler.ts # x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/review_rule_upgrade/review_rule_upgrade_route.ts # x-pack/plugins/security_solution/server/utils/throttle_requests.test.ts # x-pack/plugins/security_solution/server/utils/throttle_requests.ts # x-pack/solutions/security/plugins/security_solution/public/detection_engine/rule_management/api/hooks/use_bootstrap_prebuilt_rules.ts # x-pack/solutions/security/plugins/security_solution/public/detection_engine/rule_management_ui/components/rules_table/upgrade_prebuilt_rules_table/translations.tsx # x-pack/solutions/security/plugins/security_solution/public/detection_engine/rule_management_ui/components/rules_table/upgrade_prebuilt_rules_table/upgrade_prebuilt_rules_table_context.tsx
- Loading branch information
Showing
21 changed files
with
565 additions
and
263 deletions.
There are no files selected for viewing
20 changes: 20 additions & 0 deletions
20
...c/detection_engine/rule_management/api/hooks/prebuilt_rules/capped_exponential_backoff.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
/* | ||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
* or more contributor license agreements. Licensed under the Elastic License | ||
* 2.0; you may not use this file except in compliance with the Elastic License | ||
* 2.0. | ||
*/ | ||
|
||
const MAX_BACKOFF = 30000; | ||
|
||
/** | ||
* Calculates a backoff delay using an exponential growth formula, capped by a | ||
* predefined maximum value. | ||
* | ||
* @param failedAttempts - The number of consecutive failed attempts. | ||
* @returns The calculated backoff delay, in milliseconds. | ||
*/ | ||
export const cappedExponentialBackoff = (failedAttempts: number) => { | ||
const backoff = Math.min(1000 * 2 ** failedAttempts, MAX_BACKOFF); | ||
return backoff; | ||
}; |
22 changes: 22 additions & 0 deletions
22
.../detection_engine/rule_management/api/hooks/prebuilt_rules/retry_on_rate_limited_error.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
/* | ||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
* or more contributor license agreements. Licensed under the Elastic License | ||
* 2.0; you may not use this file except in compliance with the Elastic License | ||
* 2.0. | ||
*/ | ||
|
||
import { get } from 'lodash'; | ||
|
||
/* | ||
Prebuilt rule operations like install and upgrade are rate limited to one at a | ||
time. In most cases it is fine to wait for the other operation to finish using | ||
the retry logic. | ||
429 can be caused by a user clicking multiple times on the install or upgrade | ||
rule buttons and in most cases the operations can be performed in succession | ||
without any conflicts. | ||
*/ | ||
export const retryOnRateLimitedError = (failureCount: number, error: unknown) => { | ||
const statusCode = get(error, 'response.status'); | ||
return statusCode === 429; | ||
}; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.