Skip to content

feat: prevent potential deadlocks #768

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 53 commits into from

Conversation

gfyrag
Copy link
Contributor

@gfyrag gfyrag commented Mar 14, 2025

No description provided.

flemzord and others added 30 commits January 2, 2025 15:47
* feat: Add Flake

* ci: Rework tests

- Add ginkgo to the list of dependencies in flake.nix
- Remove unnecessary CACHE directives in Earthfile
- Simplify 'compile' section in Earthfile
- Simplify 'tests' section in Earthfile
- Simplify 'lint' section in Earthfile
- Simplify 'pre-commit' section in Earthfile
- Simplify 'pre-commit-nix' section in Earthfile
- Simplify 'tidy' section in Earthfile
- Simplify 'export-docs-events' section in Earthfile

feat/nix

* ci: Rework config

* ci: Rework config

Reworked config to use 'nix develop' with 'earthly' command. Added 'pre-commit-nix'
hook. Removed unnecessary flags and secrets. (feat/nix)

* ci: Rework config

Reworked config to use 'nix develop' with 'earthly' command. Added 'pre-commit-nix'
hook. Removed unnecessary flags and secrets. (feat/nix)

* ci: Rework config

Reworked config to use 'nix develop' with 'earthly' command. Added 'pre-commit-nix'
hook. Removed unnecessary flags and secrets. (feat/nix)

* ci: Rework config

Reworked config to use 'nix develop' with 'earthly' command. Added 'pre-commit-nix'
hook. Removed unnecessary flags and secrets. (feat/nix)

* ci: Rework config

Reworked config to use 'nix develop' with 'earthly' command. Added 'pre-commit-nix'
hook. Removed unnecessary flags and secrets. (feat/nix)

* ci: Rework config

Reworked config to use 'nix develop' with 'earthly' command. Added 'pre-commit-nix'
hook. Removed unnecessary flags and secrets. (feat/nix)

* ci: Rework config

Reworked config to use 'nix develop' with 'earthly' command. Added
'pre-commit-nix' hook. Removed unnecessary flags and secrets. (feat/nix)

* ci: Rework config

Reworked config to use 'nix develop' with 'earthly' command. Added
'pre-commit-nix' hook. Removed unnecessary flags and secrets. (feat/nix)

* feat(ledger): optimize release script in Earthfile (issue #123)

* Add .envrc and update .gitignore to track it

Added a new .envrc file with configuration for direnv and updated .gitignore to include .envrc while still ignoring .env files. This ensures proper local environment setup using direnv without accidentally committing sensitive .env files.

* Add missing `isgomock` struct to mock test files

This change ensures all mock test files include the `isgomock` struct, standardizing the structure of generated mocks. This addition aligns with gomock's conventions and facilitates internal consistency across mock definitions.

* chore: migrate from Earthly to Just for build and workflow automation

This change replaces Earthly with Just by removing Earthfiles and
introducing a Justfile. Benefits include streamlining the dependency
management, simplifying workflow automation, and improving build
process readability.

Build and CI workflows (e.g., pre-commit, tests, linting) have been
adapted to utilize Just commands, ensuring consistency across
development processes.

* build(workflow): update CI to simplify release commands

Replaced complex earthly-based release command with a `just` target in
the CI workflow for better maintainability. Added new `release`
commands to the Justfile for local, CI, and standard releases. This
streamlines the release process and enhances clarity.

* refactor: simplify and streamline build and lint processes

- Removed unused 'group' annotation in Justfile to enhance clarity.
- Consolidated COPY commands in Earthfile for better maintainability.
- Updated 'compile' stage in Earthfile to use 'sources' for consistency.

These changes reduce redundancy, improve readability, and ensure less
complexity in maintenance.

* fix(justfile): reorder pre-commit tasks for consistent execution flow

Reordered tasks in the pre-commit command to ensure `generate` and
`earthly` execute before `tidy`, `lint`, and `export-docs-events`.
This improves maintainability and ensures prerequisites are met.

* chore(ci): update release workflow to use nix and just for releases

Replaced `earthly` with `nix` and `just` in the release workflow.
This improves build consistency and leverages flakes for reproducibility.

* test(mock): add `isgomock` struct to mock types

Added `isgomock` struct to mock types in tests to enhance clarity
and identification of GoMock-generated code structure.
* chore: upgrade dependencies

* fix: pre commit
* fix: coverage

* fix(ci): add EARTHLY_SECRETS for improved build compatibility

Added EARTHLY_SECRETS environment variable to streamline integration
with Earthly by consolidating critical secrets, improving CI builds.

* fix(ci): remove redundant EARTHLY_SECRETS and use explicit secrets

Removed unused EARTHLY_SECRETS from GitHub Actions for simplification.
Updated Earthly command in Justfile to explicitly pass required secrets,
enhancing security and maintainability in CI workflows.

---------

Co-authored-by: Maxence Maireaux <maxence@maireaux.fr>
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 4.2.0 to 5.1.2.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](codecov/codecov-action@v4.2.0...v5.1.2)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore: integrate OpenAPI generation into `Justfile` and simplify `Earthfile`

- Added `openapi` task to `Justfile` for merging specs and generating docs.
- Removed OpenAPI-related tasks from `Earthfile` in favor of `Justfile`.
- Streamlined pre-commit process by centralizing OpenAPI steps in `Justfile`.

This simplifies the OpenAPI generation workflow and reduces duplication,
making the development process more efficient and maintainable.

* feat: integrate speakeasy CLI and improve build workflows

- Add speakeasy CLI as an installable package in the Nix Flake.
- Support multiple platforms with pre-defined hashes and binaries.
- Enhance Nix devShells by including speakeasy for supported systems.
- Simplify Justfile tasks by removing redundant Earthly build steps.
- Update speakeasy generate command in Justfile for better usability.
- Refactor workflows to eliminate unnecessary steps in Earthfile.

These changes streamline SDK generation, improve multi-platform support,
and reduce reliance on Earthly for local builds.
Added a new task `openapi` to manage OpenAPI specification by copying
`openapi.yaml` and saving it as an artifact. This enhances automation
and ensures consistent handling of OpenAPI files.
* feat: remove earthly usage for benchmarks

* chore: remove commited js files

* feat: use ephemeral postgres for benchmarks (cause regression error)

* feat(benchmarks): use compare recipe instead of run on prs

* feat: properly clean postgres servers

* fix: create output directory

* feat: increase benchmarks timeout

* feat: benchmarks and add inputs on workflow dispatch

* feat: use nix to setup just

* fix: nix usage
was comparing self with self
* fix: migration using azure postgres with no superuser provileges

* fix: next migration
* feat: added numscript feature flag

* fix: allow passing the flag

* refactor: use builtin function for slice string

* refactor: changed flags list representation
* chore: bump numscript version

* fix: run go mod tidy on generator
gfyrag and others added 21 commits February 26, 2025 13:41
* refactor: simplify cli args handling

* feat: remove bad comment
Improve performance of rollback detection by processing buckets concurrently using a worker pool. Added configurable parallel workers with a default of 100, and implemented error handling for multiple potential failures during the process.
#766)

* feat: upgrade skeapeasy and use streaming file system for logs imports

* feat: use speakeasy workflow to regenerate the sdk

* fix: add staticcheck for speakeasy
@gfyrag gfyrag requested a review from a team as a code owner March 14, 2025 07:53
Copy link

coderabbitai bot commented Mar 14, 2025

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)
  • main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

This change introduces a new deadlock prevention mechanism in the GetBalances method of the Store struct inside the ledger package. It adds a dependency on the slices package and implements a stable sorting of the accountsVolumes slice based on the Account and Asset fields. This ensures a consistent locking order during database operations.

Changes

File Summary
internal/.../ledger/balances.go Added a stable sort via slices.SortStableFunc to the accountsVolumes array to enforce a consistent locking order and prevent deadlocks.

Sequence Diagram(s)

sequenceDiagram
    participant C as Client
    participant G as GetBalances
    participant S as Sorting (slices package)
    participant D as Database

    C->>G: Call GetBalances()
    G->>S: Sort accountsVolumes by Account, Asset
    S-->>G: Return sorted slice
    G->>D: Acquire locks in consistent order
    D-->>G: Process balance retrieval
    G->>C: Return balances
Loading

Possibly related PRs

Suggested reviewers

  • flemzord

Poem

I'm a rabbit, hopping with glee,
Sorting through accounts so deadlocks flee.
Each lock in order, a tidy parade,
Preventing mishaps, precision well-made.
Cheers to code that’s sharp and bright!
🐰✨


🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@gfyrag gfyrag changed the base branch from main to release/v2.2 March 14, 2025 07:54
@gfyrag gfyrag enabled auto-merge (rebase) March 14, 2025 07:56
Copy link

codecov bot commented Mar 14, 2025

Codecov Report

Attention: Patch coverage is 80.22814% with 104 lines in your changes missing coverage. Please review.

Please upload report for BASE (release/v2.2@b1cb3cf). Learn more about missing BASE report.
Report is 24 commits behind head on release/v2.2.

Files with missing lines Patch % Lines
internal/storage/driver/driver.go 67.04% 21 Missing and 8 partials ⚠️
internal/worker/async_block.go 80.32% 10 Missing and 2 partials ⚠️
internal/api/router.go 38.88% 10 Missing and 1 partial ⚠️
cmd/serve.go 74.07% 6 Missing and 1 partial ⚠️
internal/storage/ledger/balances.go 74.07% 6 Missing and 1 partial ⚠️
cmd/config.go 40.00% 4 Missing and 2 partials ⚠️
cmd/worker.go 82.85% 4 Missing and 2 partials ⚠️
internal/api/bulking/handler_stream_json.go 89.58% 5 Missing ⚠️
internal/worker/fx.go 73.68% 3 Missing and 2 partials ⚠️
internal/controller/ledger/numscript_parser.go 66.66% 2 Missing and 1 partial ⚠️
... and 6 more
Additional details and impacted files
@@               Coverage Diff               @@
##             release/v2.2     #768   +/-   ##
===============================================
  Coverage                ?   81.80%           
===============================================
  Files                   ?      136           
  Lines                   ?     7306           
  Branches                ?        0           
===============================================
  Hits                    ?     5977           
  Misses                  ?     1022           
  Partials                ?      307           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@gfyrag gfyrag force-pushed the hotfix/v2.2/prevent-some-deadlocks branch from 963a0b3 to 3ae6856 Compare March 14, 2025 07:59
@gfyrag gfyrag closed this Mar 14, 2025
auto-merge was automatically disabled March 14, 2025 08:01

Pull request was closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants