Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store mutable state checksum in SQL storage #5649

Merged
merged 3 commits into from
Feb 12, 2024
Merged

Conversation

Shaddoll
Copy link
Contributor

@Shaddoll Shaddoll commented Feb 8, 2024

What changed?
Store mutable state checksum in SQL storage

Why?
Bug fix

How did you test it?
unit tests & persistence tests

Potential risks

Release notes

Documentation Changes

@Shaddoll Shaddoll force-pushed the checksum branch 3 times, most recently from f6e273e to 95b34cc Compare February 9, 2024 01:40
@coveralls
Copy link

coveralls commented Feb 9, 2024

Pull Request Test Coverage Report for Build 018d9e5a-2d00-4f90-b8c0-fd90255b73c4

Details

  • -16 of 73 (78.08%) changed or added relevant lines in 7 files are covered.
  • 162 unchanged lines in 24 files lost coverage.
  • Overall coverage decreased (-0.03%) to 62.73%

Changes Missing Coverage Covered Lines Changed/Added Lines %
common/persistence/persistence-tests/persistenceTestBase.go 0 2 0.0%
common/persistence/serializer.go 12 16 75.0%
common/persistence/sql/sql_execution_store_util.go 12 16 75.0%
common/persistence/executionManager.go 13 19 68.42%
Files with Coverage Reduction New Missed Lines %
service/history/queue/timer_queue_processor_base.go 1 78.23%
common/persistence/historyManager.go 2 66.67%
common/persistence/sql/common.go 2 72.73%
common/persistence/sql/sqlplugin/mysql/db.go 2 83.33%
common/persistence/sql/sqlplugin/mysql/task.go 2 73.68%
common/persistence/sql/sqlplugin/postgres/db.go 2 85.0%
common/persistence/sql/sqlplugin/postgres/task.go 2 73.4%
service/history/task/transfer_standby_task_executor.go 2 87.04%
service/matching/db.go 2 73.23%
common/log/tag/tags.go 3 50.46%
Totals Coverage Status
Change from base Build 018d901e-1899-4f89-a983-058a17ceae06: -0.03%
Covered Lines: 92372
Relevant Lines: 147254

💛 - Coveralls

@3vilhamster
Copy link
Contributor

Could you elaborate on the issue?
So we were not storing side effects data SQL persistence storage layer.
Should we have a integration test that ensures that we store everything for SQL and NonSQL storages?

Copy link
Contributor

@taylanisikdemir taylanisikdemir left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a mysql integration test that verifies checksum written/read properly?

common/persistence/dataStoreInterfaces.go Show resolved Hide resolved
if len(newResponse.State.Checksum.Value) == 0 {
newResponse.State.Checksum, err = m.serializer.DeserializeChecksum(response.State.ChecksumData)
if err != nil {
return nil, err
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does the err indicate it's checksum deserialization that failed or will it be just "invalid character etc." kind of error? If so maybe wrap the err with more context

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The serializer provides the context.

@@ -635,6 +642,10 @@ func (m *executionManagerImpl) SerializeWorkflowMutation(
if err != nil {
return nil, err
}
checksumData, err := m.serializer.SerializeChecksum(input.Checksum, common.EncodingTypeJSON)
if err != nil {
return nil, err
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here about err message

@Shaddoll
Copy link
Contributor Author

Shaddoll commented Feb 9, 2024

Could you elaborate on the issue? So we were not storing side effects data SQL persistence storage layer. Should we have a integration test that ensures that we store everything for SQL and NonSQL storages?

Yeah, the checksum is not stored in SQL databases. We have persistence test verifying the storage. But previously, it didn't compare checksum if the checksum read from database is empty.

@Shaddoll Shaddoll enabled auto-merge (squash) February 12, 2024 17:25
@Shaddoll Shaddoll merged commit d3bff7d into uber:master Feb 12, 2024
16 checks passed
@Shaddoll Shaddoll deleted the checksum branch February 12, 2024 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants