Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create TTL in the UpdateWorkflowExecution cycles. #5243

Merged
merged 50 commits into from
Jun 29, 2023

Conversation

agautam478
Copy link
Contributor

@agautam478 agautam478 commented May 2, 2023

What changed?
Adding TTL to the CreateWorkflowRequest is not enough because it gets overwritten by the 'UpdateWorkflowRequest' and the final value of the TTL is persisted.
So we reimplemented the code and added TTL into the UpdateworkflowRequest instead.
Supporting document:
https://docs.google.com/document/d/14Bn3W32DdJEP9dVi2MDTyGD0joUG8K_i4wCFEfFZJns/edit?usp=sharing

Why?
Calculate the TTL in the Update Workflow context and update the subsequent queries down the persistence layer. Adding TTL to the CreateWorkflowRequest is not enough because it gets overwritten by the 'UpdateWorkflowRequest' and the TTL is removed. Cassandra does not allow persistence of the TTL on its tables. If you perform an update without TTL then the TTL will be lost. In other words, we had to insert the TTL during the Workflow state update.

Moved the TTL to Workflow mutation

How did you test it?
Tested locally and unit tests.

Potential risks
We might experience data loss if the TTL value is not appropriate or if the TTL accidentally cause data deletion.

Release notes
NA

Documentation Changes
NA

@agautam478 agautam478 changed the title Create TTL in the UpdateWorkflowExecution cycles. Do not merge: Create TTL in the UpdateWorkflowExecution cycles. May 2, 2023
@coveralls
Copy link

coveralls commented May 2, 2023

Pull Request Test Coverage Report for Build 01890345-07e3-4b4f-8663-5aac7638262d

  • 51 of 140 (36.43%) changed or added relevant lines in 7 files are covered.
  • 78 unchanged lines in 16 files lost coverage.
  • Overall coverage decreased (-0.04%) to 57.23%

Changes Missing Coverage Covered Lines Changed/Added Lines %
common/persistence/persistence-tests/persistenceTestBase.go 0 1 0.0%
common/persistence/nosql/nosqlplugin/cassandra/workflowUtils.go 17 19 89.47%
service/history/execution/mutable_state_builder.go 25 29 86.21%
common/persistence/persistence-tests/executionManagerTest.go 0 82 0.0%
Files with Coverage Reduction New Missed Lines %
common/persistence/nosql/nosqlplugin/cassandra/workflowUtils.go 1 74.5%
service/history/execution/context.go 1 69.09%
service/history/queue/timer_queue_processor_base.go 1 77.26%
service/history/queue/transfer_queue_processor_base.go 1 77.62%
common/persistence/sql/common.go 2 50.65%
common/persistence/sql/sqlplugin/mysql/db.go 2 83.33%
service/history/task/transfer_active_task_executor.go 2 72.24%
service/matching/matcher.go 2 90.29%
common/log/tag/tags.go 3 49.04%
service/history/queue/timer_gate.go 3 95.83%
Totals Coverage Status
Change from base Build 0188e07b-5773-43c7-910d-9b29e3a515c6: -0.04%
Covered Lines: 87094
Relevant Lines: 152183

💛 - Coveralls

@agautam478 agautam478 changed the title Do not merge: Create TTL in the UpdateWorkflowExecution cycles. Create TTL in the UpdateWorkflowExecution cycles. May 3, 2023
@agautam478 agautam478 requested a review from Shaddoll June 13, 2023 20:30

time.Sleep(65)

_, err = s.GetWorkflowExecutionInfo(ctx, domainID, workflowExecution)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a not-found error should be expected.

@agautam478 agautam478 force-pushed the updateworkflow branch 2 times, most recently from d7484c8 to 8291df6 Compare June 15, 2023 00:17
@agautam478 agautam478 requested a review from Shaddoll June 27, 2023 20:24
//default TTL Value. 0 TTL means no ttl is set, hence your records will persist forever unless explicitly deleted.
ttlInSeconds := 0
//Only fires when the workflow is closing.
if execution.CloseStatus == 1 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you should check state == WorkflowStateCompleted instead

@agautam478 agautam478 requested a review from Shaddoll June 28, 2023 18:30
@agautam478 agautam478 merged commit bc8bebe into uber:master Jun 29, 2023
agautam478 added a commit to agautam478/cadence that referenced this pull request Aug 16, 2023
agautam478 added a commit that referenced this pull request Aug 17, 2023
ketsiambaku pushed a commit to ketsiambaku/cadence that referenced this pull request Aug 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants