Skip to content

Introduce database import and export protocol messages #224

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 13 commits into
base: master
Choose a base branch
from

Conversation

farost
Copy link
Member

@farost farost commented May 8, 2025

Release notes: usage and product changes

Add migration protocol messages for usage in database import and database export operations:

  • a unidirectional database_export stream from a TypeDB server to export a specific database, similar to TypeDB 2.x;
  • a bidirectional databases_import stream between a client and a server to import an exported 2.x/3.x TypeDB database into a TypeDB 3.x server from a client.

The format of migration items used for these operations is an extended version of TypeDB 2.x's migration items, so it is backward compatible with 2.x database files. Important: it's not intended to import 3.x databases into 2.x servers.

Implementation

Add Migration { Item } message. The format is an extended version of the 2.x protocol, so it contains "outdated" fields for its compatibility with old databases.

Add Migration { Export } message. This operation consists of a single client Req { database } and multiple streamed server responses:

  1. An initial response with the schema.
  2. An unlimited number of migration items (multiple messages with multiple items in one message for potential optimizations)
  3. A Done message to signal that the server is ready to close the stream without errors. It can be substituted by a silent stream closure, but I preferred explicitness here.

Add Migration { Import } message. This operation consists of a stream of client requests:

  1. An initial request with the name of the database and its schema string.
  2. An unlimited number of migration items.
  3. A Done message to signal that the client is finished without errors, and the server can perform the final validation. This Done message is required and cannot be removed because the client has to check whether there were finalization errors or not.

and a stream of server responses (actually, there is either a single Done or a single error, but the stream is needed to return errors at any stage of the communication).

@farost farost changed the title Add database import and export protocol messages Introduce database import and export protocol messages May 27, 2025
message Res {
DatabaseReplicas database = 1;
}
}

message Import {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've placed all migration-related messages in a single file, so it's "encapsulated". However, this introduces an additional layer of optional client and server in Rust while unpacking this message, so it's a little irritating. Not sure if it's worth it or not. I like this design more, I guess.


// This is an emulation of the google ErrorDetails message. Generally, ErrorDetails are submitted via the GRPC error
// mechanism, but a manual error sending is required in streams
message Error {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was planning to reuse this error, but I understood that I don't need non-terminal errors in my protocol. However, I think that it's reasonable to generalize error messages for the whole TypeDB protocol.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant