workflow/record: Add metadata for debug and replication lag resilience #120
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
echarrod
reviewed
Mar 17, 2025
echarrod
reviewed
Mar 17, 2025
echarrod
approved these changes
Mar 17, 2025
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The following have been added in a new type called
workflow.Metaand will require a small migration process before upgrading to the version that contains these changes. This is only acceptable as workflow is pre-V1. These are some very old ideas that have numerously come back over and over again as invaluable to a production system. I dont like extending workflow but these are some ideas I believe are intrinsic to having a solution that provides the developers the correct tools to debug issues. One fundamental change is that of the addition of versioning.Workflow is currently exposed to an issue where it will discard an event if the record store is lagging behind the event streamer. Due to the outbox pattern being implemented this meant it was unlikely happening but the reality is that it's a risk to the systems guaranteed once delivery and processing and should be mitigated.
Here are the fields that are found in Meta and their corresponding purposes:
Version: Ensures data consistency by validating that the record's version matches the event version, preventing issues with stale or out-of-order data that can take place when reading from a replica that is lagging behind the origin of the event.
RunStateReason: Human-readable explanation for the current run state, aiding in debugging by providing context behind state transitions (e.g., "Paused", "Canceled").
StatusDescription: A string representation of the Status field, making it easier to interpret status values (e.g., "In Progress", "Completed") without referencing status codes. Camel case string values are normalised to have spaces again for improved readability
TraceOrigin: Captures the origin or source of the workflow trigger, offering traceability for debugging and auditing purposes.