Stream CheckTx Order Fills #1380

jonfung-dydx · 2024-04-10T21:30:51Z

Summary by CodeRabbit

New Features
- Enhanced the orderbook streaming functionality to include detailed updates on order placements, updates, and fills.
Documentation
- Updated descriptions for methods related to streaming orderbook updates.
Refactor
- Split the function handling orderbook updates into two distinct functions for better clarity and focused operations.
Tests
- Added mock functions and methods across various test utilities to support the new orderbook update functionalities.

coderabbitai · 2024-04-10T21:31:02Z

Important

Auto Review Skipped

Auto reviews are disabled on base/target branches other than the default branch. Please add the base/target branch pattern to the list of additional branches to be reviewed in the settings.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository.

To trigger a single review, invoke the @coderabbitai review command.

Walkthrough

The recent updates focus on enhancing the streaming and handling of orderbook updates within a Central Limit Order Book (CLOB). New functionalities include detailed tracking and streaming of order placements, updates, and fills. Key changes involve splitting existing functions to specialize in match fills and general updates, alongside the introduction of new data structures and interfaces to support these operations, ensuring more granular data flow and processing in the system.

Changes

File Path	Change Summary
`.../v4-protos/src/codegen/dydxprotocol/clob/query.*`	Updated to include detailed descriptions and new interfaces for streaming orderbook updates with a focus on order placements and fills.
`proto/dydxprotocol/clob/query.proto`	Added new message types and updated RPC descriptions to enhance orderbook data handling.
`.../grpc/grpc_streaming_manager.go`	Split `SendOrderbookUpdates` into two functions for handling match fills and general orderbook updates separately.
`.../grpc/noop_streaming_manager.go`, `.../grpc/types/manager.go`, `.../memclob/keeper.go`, `.../x/clob/keeper/keeper.go`, `.../x/clob/types/mem_clob_keeper.go`	Introduced `SendOrderbookMatchFillUpdates` method across various components to manage and stream orderbook match fills.
`.../x/clob/memclob/memclob_grpc_streaming.go`	Added method to wrap clob matches into a structured format for streaming.
`protocol/mocks/MemClobKeeper.go`, `protocol/x/clob/memclob/memclob.go`	Enhanced logic to handle and send orderbook updates with fills.
`protocol/x/clob/types/operations_to_propose.go`	Updated method to handle match operations more effectively within the operations queue.

This structured approach not only clarifies the roles of different components but also enhances the scalability and robustness of the orderbook management system.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

mergify · 2024-04-10T21:31:32Z

⚠️ The sha of the head commit of this PR conflicts with #1376. Mergify cannot evaluate rules on this PR. ⚠️

mergify · 2024-04-10T21:34:28Z

⚠️ The sha of the head commit of this PR conflicts with #1376. Mergify cannot evaluate rules on this PR. ⚠️

mergify · 2024-04-10T21:35:26Z

⚠️ The sha of the head commit of this PR conflicts with #1376. Mergify cannot evaluate rules on this PR. ⚠️

jayy04 · 2024-04-24T16:05:55Z

protocol/x/clob/memclob/memclob.go

+	internalOperation := m.operationsToPropose.MustAddMatchToOperationsQueue(takerOrder, makerFillWithOrders)
+
+	// If orderbook updates are on, send an orderbook update with the fill to grpc streams.
+	if m.generateOrderbookUpdates {
+		// Collect all maker orders.
+		makerOrders := lib.MapSlice(
+			makerFillWithOrders,
+			func(mfwo types.MakerFillWithOrder) types.Order {
+				return mfwo.Order
+			},
+		)
+		clobMatch := internalOperation.GetMatch()
+		orderbookMatchFill := m.GenerateOrderBookMatchFill(ctx, *clobMatch, takerOrder, makerOrders)
+		m.clobKeeper.SendOrderbookMatchFillUpdates(ctx, []types.OrderBookMatchFill{orderbookMatchFill})
+	}


mega nit: these should be atomic, right? let's add a function to wrap these to be more future proof

jayy04 · 2024-04-24T16:06:51Z

protocol/x/clob/memclob/memclob.go

+		// Collect all maker orders.
+		makerOrders := lib.MapSlice(
+			makerFillWithOrders,
+			func(mfwo types.MakerFillWithOrder) types.Order {
+				return mfwo.Order
+			},
+		)


nit: this can also be moved to GenerateOrderBookMatchFill

jonfung-dydx changed the base branch from main to jonfung/fillsProto April 10, 2024 21:31

github-actions bot added indexer protocol proto labels Apr 10, 2024

jonfung-dydx force-pushed the jonfung/fillsProto branch from e59e565 to daa3d81 Compare April 10, 2024 21:33

jonfung-dydx added 2 commits April 10, 2024 17:37

keeper methods

04e1c28

logic to send through optimistic matches

220e913

jonfung-dydx force-pushed the jonfung/streamOrderFills branch from e3ceeaa to 220e913 Compare April 10, 2024 21:37

github-actions bot removed indexer proto labels Apr 10, 2024

jonfung-dydx requested a review from jayy04 April 24, 2024 15:49

jayy04 reviewed Apr 24, 2024

View reviewed changes

jayy04 approved these changes Apr 24, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stream CheckTx Order Fills #1380

Stream CheckTx Order Fills #1380

jonfung-dydx commented Apr 10, 2024 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Apr 10, 2024 •

edited

Loading

Auto Review Skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (`.coderabbit.yaml`)

Documentation and Community

mergify bot commented Apr 10, 2024

mergify bot commented Apr 10, 2024

mergify bot commented Apr 10, 2024

jayy04 Apr 24, 2024

jayy04 Apr 24, 2024

Stream CheckTx Order Fills #1380

Are you sure you want to change the base?

Stream CheckTx Order Fills #1380

Conversation

jonfung-dydx commented Apr 10, 2024 • edited by coderabbitai bot Loading

Summary by CodeRabbit

coderabbitai bot commented Apr 10, 2024 • edited Loading

Auto Review Skipped

Walkthrough

Changes

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

Documentation and Community

mergify bot commented Apr 10, 2024

mergify bot commented Apr 10, 2024

mergify bot commented Apr 10, 2024

jayy04 Apr 24, 2024

Choose a reason for hiding this comment

jayy04 Apr 24, 2024

Choose a reason for hiding this comment

jonfung-dydx commented Apr 10, 2024 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Apr 10, 2024 •

edited

Loading

CodeRabbit Configration File (`.coderabbit.yaml`)