Introduce database import and export protocol messages #224
Conversation
farost
changed the title
| message Res { | ||
| DatabaseReplicas database = 1; | ||
| } | ||
| } | ||
|
|
||
| message Import { |
There was a problem hiding this comment.
I've placed all migration-related messages in a single file, so it's "encapsulated". However, this introduces an additional layer of optional client and server in Rust while unpacking this message, so it's a little irritating. Not sure if it's worth it or not. I like this design more, I guess.
|
|
||
| // This is an emulation of the google ErrorDetails message. Generally, ErrorDetails are submitted via the GRPC error | ||
| // mechanism, but a manual error sending is required in streams | ||
| message Error { |
There was a problem hiding this comment.
I was planning to reuse this error, but I understood that I don't need non-terminal errors in my protocol. However, I think that it's reasonable to generalize error messages for the whole TypeDB protocol.
| UNSPECIFIED = 0; | ||
| VERSION = 6; | ||
| VERSION = 7; |
There was a problem hiding this comment.
@flyingsilverfin We need to discuss the new versioning approach. I may roll it back.
proto/migration.proto
Outdated
| message Migration { | ||
| message Export { | ||
| message Req { | ||
| string database = 1; |
There was a problem hiding this comment.
This and name on L45 should have the same name. Either name or database_name, I think.
There was a problem hiding this comment.
Fair. I thought there are more exceptions, but it's called string database only in the transaction opening request
proto/migration.proto
Outdated
| int64 relation_count = 3; | ||
| int64 role_count = 4; | ||
| int64 ownership_count = 5; | ||
| // 6 was deleted and cannot be used until a breaking change occurs |
There was a problem hiding this comment.
6 should be reserved then.
There was a problem hiding this comment.
This was copied from the 2.x implementation, but I didn't even know there was such a feature. Cool, done!
proto/migration.proto
Outdated
| bool boolean = 2; | ||
| int64 integer = 3; | ||
| double double = 4; | ||
| int64 datetime_millis = 5; // reserved for 2.x, time since epoch |
There was a problem hiding this comment.
| int64 datetime_millis = 5; // reserved for 2.x, time since epoch | |
| int64 datetime_millis = 5; // compatibility with 2.x, milliseconds since epoch |
Release notes: usage and product changes
Add
migrationprotocol messages for usage in database import and database export operations:database_exportstream from a TypeDB server to export a specific database, similar to TypeDB 2.x;databases_importstream between a client and a server to import an exported 2.x/3.x TypeDB database into a TypeDB 3.x server from a client.The format of migration items used for these operations is an extended version of TypeDB 2.x's migration items, so it is backward compatible with 2.x database files. Important: it's not intended to import 3.x databases into 2.x servers.
Implementation
Add
Migration { Item }message. The format is an extended version of the 2.x protocol, so it contains "outdated" fields for its compatibility with old databases.Add
Migration { Export }message. This operation consists of a single clientReq { database }and multiple streamed server responses:Donemessage to signal that the server is ready to close the stream without errors. It can be substituted by a silent stream closure, but I preferred explicitness here.Add
Migration { Import }message. This operation consists of a stream of client requests:Donemessage to signal that the client is finished without errors, and the server can perform the final validation. ThisDonemessage is required and cannot be removed because the client has to check whether there were finalization errors or not.and a stream of server responses (actually, there is either a single
Doneor a single error, but the stream is needed to return errors at any stage of the communication).