Added minimum timeout for transforms search of 10 minutes (#1033)

Joshua152 · tanqiuliu · bowenlan-amzn · web-flow · commit 60a8513011df · 2023-12-12T09:25:42.000-08:00
* Added minimum timeout for transforms search of 10 minutes Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Extracted cancel minimum code to function Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Fixed transform code to use cluster setting Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Removed log statements Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Changed timeout logic Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Switched to basing off seconds Signed-off-by: Joshua Au <joshuahyau@gmail.com> * [Feature] Support Transform as an ISM action (#760) * Initial impl Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * fix style Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * end to end functional Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * ISM transform unit tests & integ tests Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * Fix after core #8157 (#857) Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Upgrade the backport workflow (#862) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * Added 2.9 release notes. (#851) * Added 2.9 release notes. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Added 2.9 release notes. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Added 2.9 release notes. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> --------- Signed-off-by: AWSHurneyt <hurneyt@amazon.com> Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * Handle NPE in isRollupIndex (#855) * Handle NPE in isRollupIndex `metadata.index()` can return `null`, so handle that case by returning `false`. Signed-off-by: Bryce Lampe <brycelampe@gmail.com> * unit test Signed-off-by: Bryce Lampe <brycelampe@gmail.com> --------- Signed-off-by: Bryce Lampe <brycelampe@gmail.com> Co-authored-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * Fix core XcontentType refactor (#873) Signed-off-by: Hailong Cui <ihailong@amazon.com> Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * fix for max & min aggregations when no metric property exist (#870) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * core refactor change (#884) Signed-off-by: Hailong Cui <ihailong@amazon.com> Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * update backport branch name (#885) Signed-off-by: Hailong Cui <ihailong@amazon.com> Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * core refactor change (#887) Signed-off-by: Hailong Cui <ihailong@amazon.com> Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * Fix breaking change by core refactor (#888) Signed-off-by: Hailong Cui <ihailong@amazon.com> Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * fix core breaking (#906) Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * Support copy alias in rollover (#907) * Support copy alias in rollover Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * 2.10 Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> --------- Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * Set preference to _primary when searching control-center index (#911) * Set preference to _primary when searching control-center index Signed-off-by: gaobinlong <gbinlong@amazon.com> * Use _primary_first instead Signed-off-by: gaobinlong <gbinlong@amazon.com> --------- Signed-off-by: gaobinlong <gbinlong@amazon.com> Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * Add primary first preference to all search requests (#912) Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * fix intelliJ IDEA gradle sync error (#916) Signed-off-by: Hailong Cui <ihailong@amazon.com> Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * make control center index as system index (#919) Signed-off-by: Hailong Cui <ihailong@amazon.com> Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * Updates demo certs used in integ tests (#921) Signed-off-by: Darshit Chanpura <dchanp@amazon.com> Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * Added 2.10 release notes (#925) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * Bump bwc version (#930) Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * fix integ tests; upgrade mappings versions Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * Fix DCO Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * Addressed pr comments; Add integ test case for re-execute the same transform action Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * Addressed detekt error Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * Added ISMTransform writeable test Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> * Addressed comments; Moved updateTransformStartTime to IndexManagementRestTestCase Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> --------- Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> Signed-off-by: Bryce Lampe <brycelampe@gmail.com> Signed-off-by: Hailong Cui <ihailong@amazon.com> Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: gaobinlong <gbinlong@amazon.com> Signed-off-by: Darshit Chanpura <dchanp@amazon.com> Co-authored-by: bowenlan-amzn <bowenlan23@gmail.com> Co-authored-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: AWSHurneyt <hurneyt@amazon.com> Co-authored-by: Bryce Lampe <brycelampe@gmail.com> Co-authored-by: Hailong Cui <ihailong@amazon.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> Co-authored-by: gaobinlong <gbl_long@163.com> Co-authored-by: Darshit Chanpura <35282393+DarshitChanpura@users.noreply.github.com> Signed-off-by: Joshua Au <joshuahyau@gmail.com> * [Test] increase the wait time after transform job triggered (#999) Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Drafted 2.11 release notes. (#1004) * Drafted 2.11 release notes. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Drafted 2.11 release notes. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> --------- Signed-off-by: AWSHurneyt <hurneyt@amazon.com> Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Refactor change policy API and the policy in managed index to be non-null (#967) * Refactor the policy to be non null in managed index config Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Update Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * fix bug Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> --------- Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Add more error notification at fail points (#1000) * Add more error notification at fail points Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Handle exception gracefully Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * small fix Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> --------- Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Joshua Au <joshuahyau@gmail.com> * fix the race condition in test reset action start time (#1007) Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Bump bwc version after 2.11 release (#1015) Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Joshua Au <joshuahyau@gmail.com> * added type check for pipeline aggregator types in Transform initialization (#1014) Signed-off-by: n-dohrmann <noah@dohrmann.org> Co-authored-by: Joanne Wang <109310487+jowg-amazon@users.noreply.github.com> Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Improve security plugin enabling check (#1017) Signed-off-by: Hailong Cui <ihailong@amazon.com> Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Onboard jenkins prod docker images to github actions (#1025) * Onboard jenkins prod docker images to github actions Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Add more Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Add more Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> --------- Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Support switch aliases in shrink action. (#987) Signed-off-by: Oleg Kravchuk <oleg.ikibo@gmail.com> Co-authored-by: ikibo <oleg.kravchuk@logz.io> Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Transform pipeline aggr test (#1027) * tester code: pipeline aggr. transform job Signed-off-by: n-dohrmann <noah@dohrmann.org> * made test case for pipeline aggregator in transform job Signed-off-by: n-dohrmann <noah@dohrmann.org> * removed unnec. test lines Signed-off-by: n-dohrmann <noah@dohrmann.org> * re-added method call on Transform obj Signed-off-by: n-dohrmann <noah@dohrmann.org> --------- Signed-off-by: n-dohrmann <noah@dohrmann.org> Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Added unit test for switchAliases method. (#1035) * Added unit test for switchAliases method. Signed-off-by: Oleg Kravchuk <oleg.ikibo@gmail.com> * Added unit test for switchAliases method checking the flow when switchAliases is disabled. Signed-off-by: Oleg Kravchuk <oleg.ikibo@gmail.com> --------- Signed-off-by: Oleg Kravchuk <oleg.ikibo@gmail.com> Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Interval schedule should take start time from the request, should not set it to the current time of request execution. (#1036) Signed-off-by: Oleg Kravchuk <oleg.ikibo@gmail.com> Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Added minimum for search.cancel_after_time_interval setting for rollups (#1026) * Added minimum for search.cancel_after_time_interval setting for rollups Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Added constant for cancel_after_time_interval for rollup search Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Handled case of default value for cancel interval Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Added comment explanation for default rollup cancel after time interval Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Fixed github workflow checks Signed-off-by: Joshua Au <joshuahyau@gmail.com> --------- Signed-off-by: Joshua Au <joshuahyau@gmail.com> Co-authored-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Update 2.11.1 release note (#1042) Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Interval schedule should take start time from the request, should not… (#1040) * Interval schedule should take start time from the request, should not set it to the current time of request execution. Signed-off-by: Oleg Kravchuk <oleg.ikibo@gmail.com> * Changed the "delayed continuous execution test" to be more expressive about what it should test. Signed-off-by: Oleg Kravchuk <oleg.ikibo@gmail.com> * fixed the NPE if schedule.startTime is NULL Signed-off-by: Oleg Kravchuk <oleg.ikibo@gmail.com> * fixed the NPE if schedule.startTime is NULL Signed-off-by: Oleg Kravchuk <oleg.ikibo@gmail.com> * fixed styling Signed-off-by: Oleg Kravchuk <oleg.ikibo@gmail.com> * - removed null checks from RollUp and Transforms - fixed comments in the "delayed execution" test Signed-off-by: Oleg Kravchuk <oleg.ikibo@gmail.com> --------- Signed-off-by: Oleg Kravchuk <oleg.ikibo@gmail.com> Signed-off-by: Joshua Au <joshuahyau@gmail.com> * Removed unused imports Signed-off-by: Joshua Au <joshuahyau@gmail.com> --------- Signed-off-by: Joshua Au <joshuahyau@gmail.com> Signed-off-by: Tanqiu Liu <liutanqiu@gmail.com> Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> Signed-off-by: Bryce Lampe <brycelampe@gmail.com> Signed-off-by: Hailong Cui <ihailong@amazon.com> Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: gaobinlong <gbinlong@amazon.com> Signed-off-by: Darshit Chanpura <dchanp@amazon.com> Signed-off-by: n-dohrmann <noah@dohrmann.org> Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> Signed-off-by: Oleg Kravchuk <oleg.ikibo@gmail.com> Co-authored-by: Tanqiu Liu <liutanqiu@gmail.com> Co-authored-by: bowenlan-amzn <bowenlan23@gmail.com> Co-authored-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: AWSHurneyt <hurneyt@amazon.com> Co-authored-by: Bryce Lampe <brycelampe@gmail.com> Co-authored-by: Hailong Cui <ihailong@amazon.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> Co-authored-by: gaobinlong <gbl_long@163.com> Co-authored-by: Darshit Chanpura <35282393+DarshitChanpura@users.noreply.github.com> Co-authored-by: n-dohrmann <87952011+n-dohrmann@users.noreply.github.com> Co-authored-by: Joanne Wang <109310487+jowg-amazon@users.noreply.github.com> Co-authored-by: Peter Zhu <zhujiaxi@amazon.com> Co-authored-by: Oleg Kravchuk <oleg.ikibo@gmail.com> Co-authored-by: ikibo <oleg.kravchuk@logz.io>
diff --git a/src/main/kotlin/org/opensearch/indexmanagement/transform/TransformSearchService.kt b/src/main/kotlin/org/opensearch/indexmanagement/transform/TransformSearchService.kt
@@ -16,6 +16,7 @@ import org.opensearch.action.bulk.BackoffPolicy
 import org.opensearch.action.index.IndexRequest
 import org.opensearch.action.search.SearchRequest
 import org.opensearch.action.search.SearchResponse
+import org.opensearch.action.search.TransportSearchAction.SEARCH_CANCEL_AFTER_TIME_INTERVAL_SETTING
 import org.opensearch.client.Client
 import org.opensearch.cluster.metadata.IndexMetadata
 import org.opensearch.cluster.service.ClusterService
@@ -47,6 +48,7 @@ import org.opensearch.indexmanagement.util.IndexUtils.Companion.LUCENE_MAX_CLAUS
 import org.opensearch.indexmanagement.util.IndexUtils.Companion.ODFE_MAGIC_NULL
 import org.opensearch.indexmanagement.util.IndexUtils.Companion.hashToFixedSize
 import org.opensearch.core.rest.RestStatus
+import org.opensearch.indexmanagement.transform.settings.TransformSettings.Companion.MINIMUM_CANCEL_AFTER_TIME_INTERVAL_SECONDS
 import org.opensearch.search.aggregations.Aggregation
 import org.opensearch.search.aggregations.bucket.composite.CompositeAggregation
 import org.opensearch.search.aggregations.bucket.composite.CompositeAggregationBuilder
@@ -78,11 +80,17 @@ class TransformSearchService(
     @Volatile private var backoffPolicy =
         BackoffPolicy.constantBackoff(TRANSFORM_JOB_SEARCH_BACKOFF_MILLIS.get(settings), TRANSFORM_JOB_SEARCH_BACKOFF_COUNT.get(settings))
 
+    @Volatile private var cancelAfterTimeInterval = SEARCH_CANCEL_AFTER_TIME_INTERVAL_SETTING.get(settings)
+
     init {
         clusterService.clusterSettings.addSettingsUpdateConsumer(TRANSFORM_JOB_SEARCH_BACKOFF_MILLIS, TRANSFORM_JOB_SEARCH_BACKOFF_COUNT) {
                 millis, count ->
             backoffPolicy = BackoffPolicy.constantBackoff(millis, count)
         }
+
+        clusterService.clusterSettings.addSettingsUpdateConsumer(SEARCH_CANCEL_AFTER_TIME_INTERVAL_SETTING) {
+            cancelAfterTimeInterval = it
+        }
     }
 
     @Suppress("RethrowCaughtException")
@@ -187,7 +195,11 @@ class TransformSearchService(
             val searchStart = Instant.now().epochSecond
             val searchResponse = backoffPolicy.retryTransformSearch(logger, transformContext.transformLockManager) {
                 val pageSizeDecay = 2f.pow(retryAttempt++)
-                val searchRequestTimeoutInSeconds = transformContext.getMaxRequestTimeoutInSeconds()
+
+                var searchRequestTimeoutInSeconds = transformContext.getMaxRequestTimeoutInSeconds()
+                if (searchRequestTimeoutInSeconds == null) {
+                    searchRequestTimeoutInSeconds = getCancelAfterTimeIntervalSeconds(cancelAfterTimeInterval.seconds)
+                }
 
                 client.suspendUntil { listener: ActionListener<SearchResponse> ->
                     // If the previous request of the current transform job execution was successful, take the page size of previous request.
@@ -224,6 +236,16 @@ class TransformSearchService(
         }
     }
 
+    private fun getCancelAfterTimeIntervalSeconds(givenIntervalSeconds: Long): Long {
+        // The default value for the cancelAfterTimeInterval is -1 and so, in this case
+        // we should ignore processing on the value
+        if (givenIntervalSeconds == -1L) {
+            return -1
+        }
+
+        return max(givenIntervalSeconds, MINIMUM_CANCEL_AFTER_TIME_INTERVAL_SECONDS)
+    }
+
     companion object {
         const val failedSearchErrorMessage = "Failed to search data in source indices"
         const val modifiedBucketsErrorMessage = "Failed to get the modified buckets in source indices"
diff --git a/src/main/kotlin/org/opensearch/indexmanagement/transform/settings/TransformSettings.kt b/src/main/kotlin/org/opensearch/indexmanagement/transform/settings/TransformSettings.kt
@@ -14,6 +14,7 @@ class TransformSettings {
     companion object {
         const val DEFAULT_RENEW_LOCK_RETRY_COUNT = 3
         const val DEFAULT_RENEW_LOCK_RETRY_DELAY = 1000L
+        const val MINIMUM_CANCEL_AFTER_TIME_INTERVAL_SECONDS = 600L
 
         val TRANSFORM_JOB_SEARCH_BACKOFF_COUNT: Setting<Int> = Setting.intSetting(
             "plugins.transform.internal.search.backoff_count",
