Skip to content

Commit

Permalink
[Security Solution] Added concurrency limits and request throttling t…
Browse files Browse the repository at this point in the history
…o prebuilt rule routes (elastic#209551)

**Resolves: elastic#208357
**Resolves: elastic#208355

## Summary  

To prevent possible OOM errors, we need to limit concurrent requests to
prebuilt rule routes (see attached tickets for more details).

- `installation/_perform` and `upgrade/_perform` endpoints
- Concurrency is limited to one parallel call. If another call is made
simultaneously, the server responds with 429 Too Many Requests.
- On the front end, all rule install and upgrade operations are retried
in case of a 429 response. This ensures proper handling when a user
clicks multiple times an update or install rule buttons

- `prebuilt_rules/_bootstrap` endpoint
- Install prebuilt rules and endpoint packages sequentially instead of
in parallel to prevent from having them both downloaded into memory
simultaneously.
- Added a 30-minute socket timeout to prevent the proxy from closing the
connection while rule installation is in progress.
- Introduced a `throttleRequests` wrapper, ensuring the endpoint handler
is called only once when multiple concurrent requests are received.
- The first request triggers the handler, while subsequent requests wait
for the first one to complete and reuse its result.
- This prevents costly prebuilt rule package installation from running
in parallel.
- Reusing the response ensures the frontend correctly invalidates cached
prebuilt rule queries. Since concurrent frontend requests should receive
the same installed package information, responding with 421 and using
the retry logic as in cases above is not an option here because the
second request would receive a package installation skipped response
leading to no cache invalidation.

- `installation/_review` and `upgrade/_review` endpoints
- Concurrency is limited to one parallel call. If another call is made
simultaneously, the server responds with 429 Too Many Requests.
- On the front end, all rule install and upgrade operations are retried
in case of a 429 response. This ensures proper handling when a user
clicks multiple times an update or install rule buttons
  • Loading branch information
xcrzx authored Feb 11, 2025
1 parent ed19705 commit c5557f3
Show file tree
Hide file tree
Showing 23 changed files with 574 additions and 287 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

const MAX_BACKOFF = 30000;

/**
* Calculates a backoff delay using an exponential growth formula, capped by a
* predefined maximum value.
*
* @param failedAttempts - The number of consecutive failed attempts.
* @returns The calculated backoff delay, in milliseconds.
*/
export const cappedExponentialBackoff = (failedAttempts: number) => {
const backoff = Math.min(1000 * 2 ** failedAttempts, MAX_BACKOFF);
return backoff;
};
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { get } from 'lodash';

/*
Prebuilt rule operations like install and upgrade are rate limited to one at a
time. In most cases it is fine to wait for the other operation to finish using
the retry logic.
429 can be caused by a user clicking multiple times on the install or upgrade
rule buttons and in most cases the operations can be performed in succession
without any conflicts.
*/
export const retryOnRateLimitedError = (failureCount: number, error: unknown) => {
const statusCode = get(error, 'response.status');
return statusCode === 429;
};
Original file line number Diff line number Diff line change
Expand Up @@ -6,18 +6,18 @@
*/
import type { UseMutationOptions } from '@tanstack/react-query';
import { useMutation } from '@tanstack/react-query';
import { BOOTSTRAP_PREBUILT_RULES_URL } from '../../../../../common/api/detection_engine';
import type { BootstrapPrebuiltRulesResponse } from '../../../../../common/api/detection_engine/prebuilt_rules/bootstrap_prebuilt_rules/bootstrap_prebuilt_rules.gen';
import { PREBUILT_RULES_PACKAGE_NAME } from '../../../../../common/detection_engine/constants';
import { bootstrapPrebuiltRules } from '../api';
import { useInvalidateFetchPrebuiltRulesInstallReviewQuery } from './prebuilt_rules/use_fetch_prebuilt_rules_install_review_query';
import { useInvalidateFetchPrebuiltRulesStatusQuery } from './prebuilt_rules/use_fetch_prebuilt_rules_status_query';
import { useInvalidateFetchPrebuiltRulesUpgradeReviewQuery } from './prebuilt_rules/use_fetch_prebuilt_rules_upgrade_review_query';
import { BOOTSTRAP_PREBUILT_RULES_URL } from '../../../../../../common/api/detection_engine';
import type { BootstrapPrebuiltRulesResponse } from '../../../../../../common/api/detection_engine/prebuilt_rules/bootstrap_prebuilt_rules/bootstrap_prebuilt_rules.gen';
import { PREBUILT_RULES_PACKAGE_NAME } from '../../../../../../common/detection_engine/constants';
import { bootstrapPrebuiltRules } from '../../api';
import { useInvalidateFetchPrebuiltRulesInstallReviewQuery } from './use_fetch_prebuilt_rules_install_review_query';
import { useInvalidateFetchPrebuiltRulesStatusQuery } from './use_fetch_prebuilt_rules_status_query';
import { useInvalidateFetchPrebuiltRulesUpgradeReviewQuery } from './use_fetch_prebuilt_rules_upgrade_review_query';

export const BOOTSTRAP_PREBUILT_RULES_KEY = ['POST', BOOTSTRAP_PREBUILT_RULES_URL];

export const useBootstrapPrebuiltRulesMutation = (
options?: UseMutationOptions<BootstrapPrebuiltRulesResponse, Error>
options?: UseMutationOptions<BootstrapPrebuiltRulesResponse>
) => {
const invalidatePrePackagedRulesStatus = useInvalidateFetchPrebuiltRulesStatusQuery();
const invalidatePrebuiltRulesInstallReview = useInvalidateFetchPrebuiltRulesInstallReviewQuery();
Expand All @@ -26,7 +26,7 @@ export const useBootstrapPrebuiltRulesMutation = (
return useMutation(() => bootstrapPrebuiltRules(), {
...options,
mutationKey: BOOTSTRAP_PREBUILT_RULES_KEY,
onSettled: (...args) => {
onSuccess: (...args) => {
const response = args[0];
if (
response?.packages.find((pkg) => pkg.name === PREBUILT_RULES_PACKAGE_NAME)?.status ===
Expand All @@ -40,8 +40,8 @@ export const useBootstrapPrebuiltRulesMutation = (
invalidatePrebuiltRulesUpdateReview();
}

if (options?.onSettled) {
options.onSettled(...args);
if (options?.onSuccess) {
options.onSuccess(...args);
}
},
});
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ import { reviewRuleInstall } from '../../api';
import { REVIEW_RULE_INSTALLATION_URL } from '../../../../../../common/api/detection_engine/prebuilt_rules/urls';
import type { ReviewRuleInstallationResponseBody } from '../../../../../../common/api/detection_engine/prebuilt_rules';
import { DEFAULT_QUERY_OPTIONS } from '../constants';
import { retryOnRateLimitedError } from './retry_on_rate_limited_error';
import { cappedExponentialBackoff } from './capped_exponential_backoff';

export const REVIEW_RULE_INSTALLATION_QUERY_KEY = ['POST', REVIEW_RULE_INSTALLATION_URL];

Expand All @@ -26,6 +28,8 @@ export const useFetchPrebuiltRulesInstallReviewQuery = (
{
...DEFAULT_QUERY_OPTIONS,
...options,
retry: retryOnRateLimitedError,
retryDelay: cappedExponentialBackoff,
}
);
};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ import { reviewRuleUpgrade } from '../../api';
import { REVIEW_RULE_UPGRADE_URL } from '../../../../../../common/api/detection_engine/prebuilt_rules/urls';
import type { ReviewRuleUpgradeResponseBody } from '../../../../../../common/api/detection_engine/prebuilt_rules';
import { DEFAULT_QUERY_OPTIONS } from '../constants';
import { retryOnRateLimitedError } from './retry_on_rate_limited_error';
import { cappedExponentialBackoff } from './capped_exponential_backoff';

export const REVIEW_RULE_UPGRADE_QUERY_KEY = ['POST', REVIEW_RULE_UPGRADE_URL];

Expand All @@ -26,6 +28,8 @@ export const useFetchPrebuiltRulesUpgradeReviewQuery = (
{
...DEFAULT_QUERY_OPTIONS,
...options,
retry: retryOnRateLimitedError,
retryDelay: cappedExponentialBackoff,
}
);
};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,15 @@ import type { UseMutationOptions } from '@tanstack/react-query';
import { useMutation } from '@tanstack/react-query';
import type { PerformRuleInstallationResponseBody } from '../../../../../../common/api/detection_engine/prebuilt_rules';
import { PERFORM_RULE_INSTALLATION_URL } from '../../../../../../common/api/detection_engine/prebuilt_rules/urls';
import { useInvalidateFetchPrebuiltRulesStatusQuery } from './use_fetch_prebuilt_rules_status_query';
import { useInvalidateFindRulesQuery } from '../use_find_rules_query';
import { performInstallAllRules } from '../../api';
import { useInvalidateFetchCoverageOverviewQuery } from '../use_fetch_coverage_overview_query';
import { useInvalidateFetchRuleManagementFiltersQuery } from '../use_fetch_rule_management_filters_query';
import { useInvalidateFetchRulesSnoozeSettingsQuery } from '../use_fetch_rules_snooze_settings_query';
import { useInvalidateFindRulesQuery } from '../use_find_rules_query';
import { retryOnRateLimitedError } from './retry_on_rate_limited_error';
import { useInvalidateFetchPrebuiltRulesInstallReviewQuery } from './use_fetch_prebuilt_rules_install_review_query';
import { performInstallAllRules } from '../../api';
import { useInvalidateFetchCoverageOverviewQuery } from '../use_fetch_coverage_overview_query';
import { useInvalidateFetchPrebuiltRulesStatusQuery } from './use_fetch_prebuilt_rules_status_query';
import { cappedExponentialBackoff } from './capped_exponential_backoff';

export const PERFORM_ALL_RULES_INSTALLATION_KEY = [
'POST',
Expand All @@ -23,7 +25,7 @@ export const PERFORM_ALL_RULES_INSTALLATION_KEY = [
];

export const usePerformAllRulesInstallMutation = (
options?: UseMutationOptions<PerformRuleInstallationResponseBody, Error>
options?: UseMutationOptions<PerformRuleInstallationResponseBody>
) => {
const invalidateFindRulesQuery = useInvalidateFindRulesQuery();
const invalidateFetchRulesSnoozeSettings = useInvalidateFetchRulesSnoozeSettingsQuery();
Expand All @@ -33,7 +35,7 @@ export const usePerformAllRulesInstallMutation = (
const invalidateRuleStatus = useInvalidateFetchPrebuiltRulesStatusQuery();
const invalidateFetchCoverageOverviewQuery = useInvalidateFetchCoverageOverviewQuery();

return useMutation<PerformRuleInstallationResponseBody, Error>(() => performInstallAllRules(), {
return useMutation<PerformRuleInstallationResponseBody>(() => performInstallAllRules(), {
...options,
mutationKey: PERFORM_ALL_RULES_INSTALLATION_KEY,
onSettled: (...args) => {
Expand All @@ -49,5 +51,7 @@ export const usePerformAllRulesInstallMutation = (
options.onSettled(...args);
}
},
retry: retryOnRateLimitedError,
retryDelay: cappedExponentialBackoff,
});
};
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,17 @@ import type {
PerformRuleInstallationResponseBody,
} from '../../../../../../common/api/detection_engine/prebuilt_rules';
import { PERFORM_RULE_INSTALLATION_URL } from '../../../../../../common/api/detection_engine/prebuilt_rules/urls';
import { useInvalidateFetchPrebuiltRulesStatusQuery } from './use_fetch_prebuilt_rules_status_query';
import { useInvalidateFindRulesQuery } from '../use_find_rules_query';
import { useInvalidateFetchRuleManagementFiltersQuery } from '../use_fetch_rule_management_filters_query';
import { useInvalidateFetchRulesSnoozeSettingsQuery } from '../use_fetch_rules_snooze_settings_query';
import { useInvalidateFetchPrebuiltRulesInstallReviewQuery } from './use_fetch_prebuilt_rules_install_review_query';
import type { BulkAction } from '../../api';
import { performInstallSpecificRules } from '../../api';
import { useInvalidateFetchCoverageOverviewQuery } from '../use_fetch_coverage_overview_query';
import { useBulkActionMutation } from '../use_bulk_action_mutation';
import { useInvalidateFetchCoverageOverviewQuery } from '../use_fetch_coverage_overview_query';
import { useInvalidateFetchRuleManagementFiltersQuery } from '../use_fetch_rule_management_filters_query';
import { useInvalidateFetchRulesSnoozeSettingsQuery } from '../use_fetch_rules_snooze_settings_query';
import { useInvalidateFindRulesQuery } from '../use_find_rules_query';
import { retryOnRateLimitedError } from './retry_on_rate_limited_error';
import { useInvalidateFetchPrebuiltRulesInstallReviewQuery } from './use_fetch_prebuilt_rules_install_review_query';
import { useInvalidateFetchPrebuiltRulesStatusQuery } from './use_fetch_prebuilt_rules_status_query';
import { cappedExponentialBackoff } from './capped_exponential_backoff';

export const PERFORM_SPECIFIC_RULES_INSTALLATION_KEY = [
'POST',
Expand All @@ -35,7 +37,7 @@ export interface UsePerformSpecificRulesInstallParams {
export const usePerformSpecificRulesInstallMutation = (
options?: UseMutationOptions<
PerformRuleInstallationResponseBody,
Error,
unknown,
UsePerformSpecificRulesInstallParams
>
) => {
Expand All @@ -51,7 +53,7 @@ export const usePerformSpecificRulesInstallMutation = (

return useMutation<
PerformRuleInstallationResponseBody,
Error,
unknown,
UsePerformSpecificRulesInstallParams
>(
(rulesToInstall: UsePerformSpecificRulesInstallParams) =>
Expand Down Expand Up @@ -81,6 +83,8 @@ export const usePerformSpecificRulesInstallMutation = (
options.onSettled(...args);
}
},
retry: retryOnRateLimitedError,
retryDelay: cappedExponentialBackoff,
}
);
};
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,16 @@ import type { UseMutationOptions } from '@tanstack/react-query';
import { useMutation } from '@tanstack/react-query';
import type { PerformRuleUpgradeResponseBody } from '../../../../../../common/api/detection_engine/prebuilt_rules';
import { PERFORM_RULE_UPGRADE_URL } from '../../../../../../common/api/detection_engine/prebuilt_rules/urls';
import { useInvalidateFetchPrebuiltRulesStatusQuery } from './use_fetch_prebuilt_rules_status_query';
import { useInvalidateFindRulesQuery } from '../use_find_rules_query';
import { useInvalidateFetchRuleManagementFiltersQuery } from '../use_fetch_rule_management_filters_query';
import { useInvalidateFetchRulesSnoozeSettingsQuery } from '../use_fetch_rules_snooze_settings_query';
import type { PerformUpgradeRequest } from '../../api';
import { performUpgradeSpecificRules } from '../../api';
import { useInvalidateFetchPrebuiltRulesUpgradeReviewQuery } from './use_fetch_prebuilt_rules_upgrade_review_query';
import { useInvalidateFetchCoverageOverviewQuery } from '../use_fetch_coverage_overview_query';
import { useInvalidateFetchRuleManagementFiltersQuery } from '../use_fetch_rule_management_filters_query';
import { useInvalidateFetchRulesSnoozeSettingsQuery } from '../use_fetch_rules_snooze_settings_query';
import { useInvalidateFindRulesQuery } from '../use_find_rules_query';
import { useInvalidateFetchPrebuiltRulesStatusQuery } from './use_fetch_prebuilt_rules_status_query';
import { useInvalidateFetchPrebuiltRulesUpgradeReviewQuery } from './use_fetch_prebuilt_rules_upgrade_review_query';
import { retryOnRateLimitedError } from './retry_on_rate_limited_error';
import { cappedExponentialBackoff } from './capped_exponential_backoff';

export const PERFORM_SPECIFIC_RULES_UPGRADE_KEY = [
'POST',
Expand All @@ -24,7 +26,7 @@ export const PERFORM_SPECIFIC_RULES_UPGRADE_KEY = [
];

export const usePerformSpecificRulesUpgradeMutation = (
options?: UseMutationOptions<PerformRuleUpgradeResponseBody, Error, PerformUpgradeRequest>
options?: UseMutationOptions<PerformRuleUpgradeResponseBody, unknown, PerformUpgradeRequest>
) => {
const invalidateFindRulesQuery = useInvalidateFindRulesQuery();
const invalidateFetchRulesSnoozeSettings = useInvalidateFetchRulesSnoozeSettingsQuery();
Expand All @@ -35,7 +37,7 @@ export const usePerformSpecificRulesUpgradeMutation = (
const invalidateRuleStatus = useInvalidateFetchPrebuiltRulesStatusQuery();
const invalidateFetchCoverageOverviewQuery = useInvalidateFetchCoverageOverviewQuery();

return useMutation<PerformRuleUpgradeResponseBody, Error, PerformUpgradeRequest>(
return useMutation<PerformRuleUpgradeResponseBody, unknown, PerformUpgradeRequest>(
(args: PerformUpgradeRequest) => {
return performUpgradeSpecificRules(args);
},
Expand All @@ -56,6 +58,8 @@ export const usePerformSpecificRulesUpgradeMutation = (
options.onSettled(...args);
}
},
retry: retryOnRateLimitedError,
retryDelay: cappedExponentialBackoff,
}
);
};
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import { useEffect } from 'react';
import {
BOOTSTRAP_PREBUILT_RULES_KEY,
useBootstrapPrebuiltRulesMutation,
} from '../api/hooks/use_bootstrap_prebuilt_rules';
} from '../api/hooks/prebuilt_rules/use_bootstrap_prebuilt_rules';

/**
* Install or upgrade the security packages (endpoint and prebuilt rules)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,8 @@ export const AddPrebuiltRulesTableContextProvider = ({
rules: [{ rule_id: ruleId, version: rule.version }],
enable,
});
} catch {
// Error is handled by the mutation's onError callback, so no need to do anything here
} finally {
setLoadingRules((prev) => prev.filter((id) => id !== ruleId));
}
Expand All @@ -182,6 +184,8 @@ export const AddPrebuiltRulesTableContextProvider = ({
setLoadingRules((prev) => [...prev, ...rulesToUpgrade.map((r) => r.rule_id)]);
try {
await installSpecificRulesRequest({ rules: rulesToUpgrade, enable });
} catch {
// Error is handled by the mutation's onError callback, so no need to do anything here
} finally {
setLoadingRules((prev) =>
prev.filter((id) => !rulesToUpgrade.some((r) => r.rule_id === id))
Expand All @@ -197,6 +201,8 @@ export const AddPrebuiltRulesTableContextProvider = ({
setLoadingRules((prev) => [...prev, ...rules.map((r) => r.rule_id)]);
try {
await installAllRulesRequest();
} catch {
// Error is handled by the mutation's onError callback, so no need to do anything here
} finally {
setLoadingRules([]);
setSelectedRules([]);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,12 +61,6 @@ export const UPDATE_BUTTON_LABEL = i18n.translate(
defaultMessage: 'Update',
}
);
export const UPDATE_ERROR = i18n.translate(
'xpack.securitySolution.detectionEngine.ruleDetails.updateError',
{
defaultMessage: 'Update error',
}
);

export const UPDATE_FLYOUT_PER_FIELD_TOOLTIP_DESCRIPTION = i18n.translate(
'xpack.securitySolution.detectionEngine.ruleDetails.perFieldTooltip',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@ import type {
RuleUpgradeSpecifier,
} from '../../../../../../common/api/detection_engine';
import { usePrebuiltRulesCustomizationStatus } from '../../../../rule_management/logic/prebuilt_rules/use_prebuilt_rules_customization_status';
import { useAppToasts } from '../../../../../common/hooks/use_app_toasts';
import type { RuleUpgradeState } from '../../../../rule_management/model/prebuilt_rule_upgrade';
import { RuleUpgradeTab } from '../../../../rule_management/components/rule_details/three_way_diff';
import { PerFieldRuleDiffTab } from '../../../../rule_management/components/rule_details/per_field_rule_diff_tab';
Expand Down Expand Up @@ -128,8 +127,6 @@ export const UpgradePrebuiltRulesTableContextProvider = ({
tags: [],
ruleSource: [],
});
const { addError } = useAppToasts();

const isUpgradingSecurityPackages = useIsUpgradingSecurityPackages();

const {
Expand Down Expand Up @@ -196,21 +193,15 @@ export const UpgradePrebuiltRulesTableContextProvider = ({
pickVersion: 'MERGED',
rules: ruleUpgradeSpecifiers,
});
} catch (err) {
addError(err, { title: i18n.UPDATE_ERROR });
} catch {
// Error is handled by the mutation's onError callback, so no need to do anything here
} finally {
const upgradedRuleIdsSet = new Set(upgradingRuleIds);

setLoadingRules((prev) => prev.filter((id) => !upgradedRuleIdsSet.has(id)));
}
},
[
confirmLegacyMLJobs,
confirmConflictsUpgrade,
rulesUpgradeState,
upgradeSpecificRulesRequest,
addError,
]
[confirmLegacyMLJobs, confirmConflictsUpgrade, rulesUpgradeState, upgradeSpecificRulesRequest]
);

const upgradeRulesToTarget = useCallback(
Expand All @@ -233,15 +224,15 @@ export const UpgradePrebuiltRulesTableContextProvider = ({
pickVersion: 'TARGET',
rules: ruleUpgradeSpecifiers,
});
} catch (err) {
addError(err, { title: i18n.UPDATE_ERROR });
} catch {
// Error is handled by the mutation's onError callback, so no need to do anything here
} finally {
const upgradedRuleIdsSet = new Set(ruleIds);

setLoadingRules((prev) => prev.filter((id) => !upgradedRuleIdsSet.has(id)));
}
},
[confirmLegacyMLJobs, rulesUpgradeState, upgradeSpecificRulesRequest, addError]
[confirmLegacyMLJobs, rulesUpgradeState, upgradeSpecificRulesRequest]
);

const upgradeRules = useCallback(
Expand Down
Loading

0 comments on commit c5557f3

Please sign in to comment.