Making snapshot name to scripted input in template #77

thalurur · 2021-06-10T17:26:52Z

Signed-off-by: Ravi Thaluru ravi1092@gmail.com

Issue #, if available:
https://discuss.opendistrocommunity.dev/t/universal-snapshot-policy-for-all-indexes/4815

Description of changes:
Processing snapshot field as a script instead of string so snapshot name can be derived from run time context - the two attributes supported during runtime are index and indexuuid

CheckList:
[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

src/test/resources/mappings/cached-opendistro-ism-config.json

cryptk · 2021-09-20T22:35:17Z

Is there something that is holding this up? This feature would be greatly appreciated

…ed from the index or any runtime attributes instead of being constant Signed-off-by: Ravi Thaluru <ravi1092@gmail.com>

codecov-commenter · 2021-10-01T17:01:21Z

Codecov Report

Merging #77 (096aff4) into main (dafd3bb) will decrease coverage by 0.06%.
The diff coverage is 93.75%.

@@             Coverage Diff              @@
##               main      #77      +/-   ##
============================================
- Coverage     75.88%   75.81%   -0.07%     
- Complexity     1922     1924       +2     
============================================
  Files           260      260              
  Lines         10992    11002      +10     
  Branches       1720     1723       +3     
============================================
  Hits           8341     8341              
- Misses         1699     1710      +11     
+ Partials        952      951       -1

Impacted Files	Coverage Δ
...atemanagement/step/snapshot/AttemptSnapshotStep.kt	`89.02% <92.85%> (+0.13%)`	⬆️
...ment/indexstatemanagement/action/SnapshotAction.kt	`46.15% <100.00%> (ø)`
...atemanagement/model/action/SnapshotActionConfig.kt	`82.85% <100.00%> (ø)`
...nt/indexstatemanagement/model/destination/Slack.kt	`40.90% <0.00%> (-13.64%)`	⬇️
...anagement/indexstatemanagement/model/Transition.kt	`63.07% <0.00%> (-4.62%)`	⬇️
...arch/indexmanagement/rollup/RollupSearchService.kt	`57.40% <0.00%> (-3.71%)`	⬇️
...ent/indexstatemanagement/util/ManagedIndexUtils.kt	`77.92% <0.00%> (-1.30%)`	⬇️
...earch/indexmanagement/transform/model/Transform.kt	`87.00% <0.00%> (-0.50%)`	⬇️
...nt/indexstatemanagement/model/destination/Chime.kt	`54.54% <0.00%> (+13.63%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dafd3bb...096aff4. Read the comment docs.

...lin/org/opensearch/indexmanagement/indexstatemanagement/step/snapshot/AttemptSnapshotStep.kt

dbbaughe · 2021-10-01T18:45:50Z

...lin/org/opensearch/indexmanagement/indexstatemanagement/step/snapshot/AttemptSnapshotStep.kt

@@ -159,6 +173,7 @@ class AttemptSnapshotStep(
    }

    companion object {
+        val validTopContextFields = setOf("index", "indexUuid")


Just to confirm is this all users have asked for? (Access to index/indexUuid in the snapshot name)

the users only asked for indexName in the issue, I have added indexUuid too since its readily available

dbbaughe · 2021-10-01T18:47:04Z

...lin/org/opensearch/indexmanagement/indexstatemanagement/step/snapshot/AttemptSnapshotStep.kt

-                        .format(DateTimeFormatter.ofPattern("uuuu.MM.dd-HH:mm:ss.SSS", Locale.ROOT))
-                )
+            val snapshotScript = Script(ScriptType.INLINE, Script.DEFAULT_TEMPLATE_LANG, config.snapshot, mapOf())
+            snapshotName = compileTemplate(snapshotScript, managedIndexMetaData, indexName).plus(snapshotNamePrefix)


Any behavior differences for users w/ existing snapshot actions or they should continue to work (and behave) as before this fix?

i.e. their snapshot names should not change somehow because of this change if they are not using the ctx variable and are just using a string.

They should work the same way, the mustache template compiler will return the same string value if there is no placeholder fields

dbbaughe · 2021-10-01T18:48:58Z

...lin/org/opensearch/indexmanagement/indexstatemanagement/step/snapshot/AttemptSnapshotStep.kt

+        val compiledValue = scriptService.compile(template, TemplateScript.CONTEXT)
+            .newInstance(template.params + mapOf("ctx" to contextMap))
+            .execute()
+        return if (compiledValue.isBlank()) defaultValue else compiledValue


When does compiledValue.isBlank() occur?

if someone puts in a template with just a replaceable field and that is not valid e.g. {{ctx.name}} or {{ctx.anything_not_allowed}} this will be empty string

If someone specifies the template as {{ctx.anything_not_allowed}}_snapshot then the compiled value is just _snapshot

If someone specifies the template as {ctx.anything_not_allowed} since this is not a valid template placeholder it will be compiled as {ctx.anything_not_allowed}

Hmm can someone specify an empty string as the snapshot name currently or do we block that?
i.e. if currently sameone has "" and it's allowed then instead of "" + the postfix stuff it'll change to $indexName + postfix stuff (since it looks like we add the index name as the default to use if this returns blank)

good point, there is no check for it being non empty at the moment the field needs to be present in the policy but it can be empty string.

I can update the default value to empty string in that case - i.e if the original name is empty string

Signed-off-by: Ravi Thaluru <ravi1092@gmail.com>

* Add integTest script to the repo (#94) Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Removing Usages of Action Get Call and using listeners (#100) Signed-off-by: Aditya Jindal <aditjind@amazon.com> * Enhance ISM template (#105) Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Explain response still use old opendistro policy id (#109) * Explain response still use old opendistro policy id * Use hardcoded policyid setting in tests for explain response * Trying to fix flaky tests * Storing user information as part of the job when security plugin is installed (#113) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com> * ISM/Notification channel support (#117) * Updates NAME of transport actions * Upgrades Kotlin version, updates dependencies on other OS plugins, adds notification plugin as a test resource and includes it in test clusters * Adds support for Channels in error notifications and notification actions * Adds support for sending notifications to channels * Adds support for publishing notifications to the legacy destinations through the Notification plugin and some cleanup * Removes notification alerting jar dependency * Adds compile only dep on commons codec for digest utils sha1 method in ism rollup * Updates Error Notification to make channel/destination nullable, and adds helper methods for publish calls * Constructs URL for legacy custom webhook Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Fixes Feature enum and dep Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Trying something else Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Addresses comments Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Storing user object in all APIs and enabling filter of response based on user (#115) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com> * Upgrade dependencies to 1.1 and build snapshot by default. (#121) Signed-off-by: dblock <dblock@amazon.com> * Security improvements (#126) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com> * Removes support for notification plugin (#136) Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Updating security filtering logic (#137) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com> * Release note for 1.1.0.0 release. (#139) * Release note for 1.1.0.0 release. Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Correct copyright notices * Uses published daily snapshot dependencies (#141) Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * License header check (#142) * Provide default copyright header using IDE feature Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Address #103 history write index is rolled over even if the history indices are disabled * Removed integtest.sh. (#148) Signed-off-by: dblock <dblock@dblock.org> * Adds mavenLocal back to repositories (#158) Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Making snapshot name to scripted input in template (#77) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com> * Fix issues with security changes in rollup runnner (#161) * Updates index management version to 1.2 (#157) * Updates index management version to 1.2 * Updates job scheduler snapshot to 1.2 in test resources Signed-off-by: Robert Downs <downsrob@amazon.com> * Adds setting to search all rollup jobs on a target index (#165) * Adds cluster setting to search all rollup jobs Signed-off-by: Clay Downs <downsrob@amazon.com> * Adds implementation for the delay feature in rollup jobs (#147) * Adds delay implementation for rollup jobs * Removes non-continuous delay implementation * Adds additional rollup delay tests Signed-off-by: Clay Downs <downsrob@amazon.com> * Updates testCompile mockito version, adds AwaitsFix annotation to MetadataRegressionIT tests (#168) * Updates testCompile mockito version to match OpenSearch changes * AwaitsFix on the failing MetadataRegressionIT tests Signed-off-by: Robert Downs <downsrob@amazon.com> * Adds cluster setting to configure index state management jitter (#153) * Adds jitter cluster setting, sets jitter to 0 for ISM tests Signed-off-by: Clay Downs <downsrob@amazon.com> * Allows out of band rollovers on an index without causing ISM to fail (#180) * Allows out of band rollovers on an index without causing ISM to fail Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Fixes detekt issue Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Remove policy API on read only indices (#182) * In explain API not showing the total count to all users (#185) Co-authored-by: Peter Zhu <zhujiaxi@amazon.com> Co-authored-by: Aditya Jindal <13850971+aditjind@users.noreply.github.com> Co-authored-by: Bowen Lan <62091230+bowenlan-amzn@users.noreply.github.com> Co-authored-by: Ravi <6005951+thalurur@users.noreply.github.com> Co-authored-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org>

…t#77) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com>

thalurur marked this pull request as draft June 10, 2021 17:27

thalurur mentioned this pull request Jun 10, 2021

Making snapshot name a scripted field opendistro-for-elasticsearch/index-management#425

Closed

thalurur changed the title ~~Making snapshot name to script template so snapshot name can be derived from the index or any runtime attributes instead of being constant~~ Making snapshot name to scripted input in template Jun 10, 2021

dbbaughe reviewed Jun 10, 2021

View reviewed changes

src/test/resources/mappings/cached-opendistro-ism-config.json Outdated Show resolved Hide resolved

thalurur added 2 commits October 1, 2021 09:27

Making snapshot name to script template so snapshot name can be deriv…

87df7ea

…ed from the index or any runtime attributes instead of being constant Signed-off-by: Ravi Thaluru <ravi1092@gmail.com>

Not changing mapping type of snapshot name

6636c68

thalurur force-pushed the rsnap branch from b25a9fd to 6636c68 Compare October 1, 2021 16:28

Correctly building mustache template script in the attempt step

c7af351

thalurur marked this pull request as ready for review October 1, 2021 16:38

dbbaughe reviewed Oct 1, 2021

View reviewed changes

...lin/org/opensearch/indexmanagement/indexstatemanagement/step/snapshot/AttemptSnapshotStep.kt Outdated Show resolved Hide resolved

dbbaughe reviewed Oct 1, 2021

View reviewed changes

thalurur added 2 commits October 1, 2021 12:59

PR comments

5b7b8c3

Signed-off-by: Ravi Thaluru <ravi1092@gmail.com>

Honoring empty snapshot names

096aff4

dbbaughe approved these changes Oct 1, 2021

View reviewed changes

thalurur merged commit 905c16e into opensearch-project:main Oct 1, 2021

wuychn pushed a commit to ochprince/index-management that referenced this pull request Mar 16, 2023

Making snapshot name to scripted input in template (opensearch-projec…

3958175

…t#77) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Making snapshot name to scripted input in template #77

Making snapshot name to scripted input in template #77

thalurur commented Jun 10, 2021 •

edited

Loading

cryptk commented Sep 20, 2021

codecov-commenter commented Oct 1, 2021 •

edited

Loading

dbbaughe Oct 1, 2021

thalurur Oct 1, 2021

dbbaughe Oct 1, 2021

dbbaughe Oct 1, 2021

thalurur Oct 1, 2021

dbbaughe Oct 1, 2021

thalurur Oct 1, 2021 •

edited

Loading

dbbaughe Oct 1, 2021

thalurur Oct 1, 2021 •

edited

Loading

Making snapshot name to scripted input in template #77

Making snapshot name to scripted input in template #77

Conversation

thalurur commented Jun 10, 2021 • edited Loading

cryptk commented Sep 20, 2021

codecov-commenter commented Oct 1, 2021 • edited Loading

Codecov Report

dbbaughe Oct 1, 2021

Choose a reason for hiding this comment

thalurur Oct 1, 2021

Choose a reason for hiding this comment

dbbaughe Oct 1, 2021

Choose a reason for hiding this comment

dbbaughe Oct 1, 2021

Choose a reason for hiding this comment

thalurur Oct 1, 2021

Choose a reason for hiding this comment

dbbaughe Oct 1, 2021

Choose a reason for hiding this comment

thalurur Oct 1, 2021 • edited Loading

Choose a reason for hiding this comment

dbbaughe Oct 1, 2021

Choose a reason for hiding this comment

thalurur Oct 1, 2021 • edited Loading

Choose a reason for hiding this comment

thalurur commented Jun 10, 2021 •

edited

Loading

codecov-commenter commented Oct 1, 2021 •

edited

Loading

thalurur Oct 1, 2021 •

edited

Loading

thalurur Oct 1, 2021 •

edited

Loading