Introduce file-based database export and import #758

farost · 2025-05-28T08:14:27Z

Usage and product changes

Introduce interfaces to export databases into schema definition and data files and to import databases using these files. Database import supports files exported from both TypeDB 2.x and TypeDB 3.x.

Both operations are blocking and may take a significant amount of time to execute for large databases. Use parallel connections to continue operating with the server and its other databases.

Usage examples in Rust:

// export
let db = driver.databases().get(db_name).await.unwrap();
db.export_to_file(schema_file_path, data_file_path).await.unwrap();

// import
let schema = read_to_string(schema_file_path).unwrap();
driver.databases().import_from_file(db_name2, schema, data_file_path).await.unwrap();

Usage examples in Python:

# export
database = driver.databases.get(db_name)
database.export_to_file(schema_file_path, data_file_path)

# import
with open(schema_file_path, 'r', encoding='utf-8') as f:
    schema = f.read()
driver.databases.import_from_file(db_name2, schema, data_file_path)

Usage examples in Java:

// export
Database database = driver.databases().get(dbName);
database.exportToFile(schemaFilePath, dataFilePath);

// import
String schema = Files.readString(Path.of(schemaFilePath));
driver.databases().importFromFile(dbName2, schema, dataFilePath);

Implementation

Implemented the updated protocol.

As both operations work with streaming, the implementation is similar to transactions. The behavior is split into the file processing logic and networking (specialized for sync and async modes). The exposed interfaces present only the file-based versions, but additional interfaces for direct work with streams can be presented in future updates.

In Rust, paths are accepted as Rust Paths. In other languages working through C interfaces, it's pure strings for C layer transmission.

Database export

Implemented through the database interface and accepts two target files for export. Does not require a specific format of naming (so it's necessarily .typeql or .typedb. If any of the target files already exist, an error is returned.

The export operation consists of these steps:

prepare the output files
open a unidirectional GRPC stream from the server to the client
"block" on server response listening until an error or a "done" is received (blocking is implemented through a loop which resolves a listen promise presented by the network layer)
if there is a schema message, write it to the schema file and flush it right away
if there is a data items message, encode the items and write them to the data file
in case of an error, the output files are deleted (we own them as we create them at the beginning)

The network layer is basically just a task listening for the GRPC stream and transmitting the converted messages to the processing loop.

Database import

Implemented through the database_manager interface and accepts a database name, a schema definition query string (can be read from the exported file), and an exported data file. No naming requirements as well.

The import operation consists of these steps:

open the input file
open a bidirectional GRPC stream between the server and the client, send the initial request with the database's name and schema
eagerly start reading and decoding data items from the input file one by one, storing up to 250 items in the buffer (this number can be easily changed)
once the buffer is full, attempt items sending operation: this operation will check a potential early error signal from the server and then send the batch, returning to processing the rest of the file
once the file is read, send a "done" message and block until the server responds with either an error or its "done" message

The network layer consists of a blocking task for client-side requests and a listening task waiting for a one-shot signal from the server (either an error or a "done" message). Errors can be received at any time of processing, while "done" is expected only after a client-side "done" request. When a response is received, either an async or a sync sink receives this message, which should be checked before any client-side network operations to ensure proper interruption.

docs/modules/ROOT/partials/java/transaction/QueryOptions.adoc

farost · 2025-05-28T16:58:31Z

rust/src/common/error.rs

@@ -186,6 +194,20 @@ error_messages! { ConceptError
        2: "Cannot get concept from a concept row by index '{index}'.",
 }

+error_messages! { MigrationError


I understood that it's probably more correct to have a separate error. I may convert it to DatabaseError or smth. It's certainly not a connection error.

farost · 2025-05-28T16:59:48Z

rust/src/connection/database/export_stream.rs

+        Self { export_transmitter }
+    }
+
+    pub(crate) fn listen<'a>(&'a mut self) -> impl Promise<'a, Result<DatabaseExportAnswer>> + 'a {


Listen to exported items, similarly to how we listen for transaction responses.

listen normally refers to a loop. This is more like accept or even next.

next is good, thansk

farost · 2025-05-28T17:00:11Z

rust/src/connection/database/import_stream.rs

+        Self { import_transmitter }
+    }
+
+    pub(crate) fn items(&mut self, items: Vec<migration::Item>) -> Result {


Send a bucket of migration items.

send_items then. Or even just send, the items are the argument.

done is also "sent", so I don't like the send variant. Let it be send_items

rust/src/connection/database/import_stream.rs

rust/src/database/database.rs

farost · 2025-05-28T17:15:06Z

rust/src/database/database.rs

+            }
+        }
+
+        data_writer.flush()?;


I wonder if I need to flush the exported file from time to time hmmm

Is there a benefit? If the driver crashes you'd have to restart the export process anyway.

True. Yeah, I think it knows how to optimally write itself. I was more concerned about buffers for a moment.

farost · 2025-05-28T17:16:24Z

rust/src/database/database_manager.rs

+                let item = item?;
+                item_buffer.push(item);
+                if item_buffer.len() >= ITEM_BATCH_SIZE {
+                    import_stream.items(item_buffer.split_off(0))?;


split_off preserves the capacity of the buffer

Another option is mem::replace(&mut item_buffer, Vec::with_capacity(ITEM_BATCH_SIZE)).
Neither is particularly readable, but split_off is at least shorter.

I like split_off more, but I think it's a question of what you are used to. I used split_off in both the driver and the server, so I may get us more used to it =)

rust/src/database/database_manager.rs

farost · 2025-05-28T17:21:21Z

rust/src/database/migration.rs

+    _phantom_data: PhantomData<M>,
+}
+
+impl<M: Message + Default, R: BufRead> ProtoMessageIterator<M, R> {


Manual import data files' decoding. All the prost's helpers expect an impl of bytes::Buf, which requires an in-memory buffer and cannot be used on a file reader (e.g., writing a where I consume the whole file and then process it is 20-30 times smaller in code volume). At the same time, it's unclear how many bytes to read from the file without more manual operations.

The idea is the following: as the data is delimited by length, we need to read the length first, then read exactly length bytes to decode the item. Then, all the other bytes should be used for the next item, so they should be preserved in the buffer.

farost · 2025-05-29T09:46:42Z

rust/src/database/database_manager.rs

+        let schema_ref: &str = schema.as_ref();
+        let data_file_path = data_file_path.as_ref();
+        self.run_failsafe(name, |server_connection, name| async move {
+            let mut import_stream = server_connection.import_database(name, schema_ref.to_string()).await?;


We don't wait for a server response here and proceed to read the file and send its items. However, if an error is returned, we will stop on one of the .items(...) calls.

farost · 2025-05-29T10:25:37Z

rust/src/database/migration.rs

+    }
+}
+
+pub(crate) fn try_creating_export_file(path: impl AsRef<Path>) -> Result<File> {


No forced filename format.

farost · 2025-05-29T10:36:38Z

CI is red because there is no server artifact yet.

farost · 2025-06-03T15:54:59Z

rust/src/connection/network/transmitter/shutdown_guard.rs

+
+// Makes sure that the shutdown signal is sent when it's dropped. Can try sending redundant shutdown
+// signals after another shutdown source: the channel is expected to be closed (no-op).
+pub(crate) struct ShutdownGuard<T: Default> {


Previously, we always counted on the server to drop the stream. However, for import, we are the owners of the stream, and we should make sure to terminate it for both

GRPC connection dropping because of the server or the network

Our error leading to resource deallocation <- this struct helps specifically with this case

dmitrii-ubskii

Naming and style comments aside, looks good.

dmitrii-ubskii · 2025-06-06T16:21:52Z

rust/tests/behaviour/steps/lib.rs

+use crate::util::{create_temp_dir, TempDir};
+


Confused auto-import

dmitrii-ubskii · 2025-06-06T16:43:44Z

c/src/database_manager.rs

-use crate::{error::try_release_arc, iterator::iterator_arc_next};
+use crate::{error::try_release_arc, iterator::iterator_arc_next, memory::borrow};


dmitrii-ubskii · 2025-06-06T16:56:32Z

rust/src/connection/database/export_stream.rs

+        Self { export_transmitter }
+    }
+
+    pub(crate) fn listen<'a>(&'a mut self) -> impl Promise<'a, Result<DatabaseExportAnswer>> + 'a {


listen normally refers to a loop. This is more like accept or even next.

dmitrii-ubskii · 2025-06-06T16:58:13Z

rust/src/connection/database/import_stream.rs

+        Self { import_transmitter }
+    }
+
+    pub(crate) fn items(&mut self, items: Vec<migration::Item>) -> Result {


send_items then. Or even just send, the items are the argument.

dmitrii-ubskii · 2025-06-06T17:04:54Z

rust/src/connection/message.rs

+    DatabaseExportStream {
+        response_source: Streaming<database::export::Server>,
+    },
+    DatabaseExportAnswer {
+        response_source: Streaming<database::export::Server>,
    },


Is Response::DatabaseExportAnswer used? It seems to be a doublet of DatabaseExportStream.

Yep, it got redundant after a couple of protocol refactoring iterations. Good catch

dmitrii-ubskii · 2025-06-06T17:22:10Z

rust/src/connection/network/transmitter/import.rs

+        }
+    }
+
+    async fn listen(


Same as listen above, this should be renamed

dmitrii-ubskii · 2025-06-06T17:31:44Z

rust/src/database/database.rs

+            }
+        }
+
+        data_writer.flush()?;


Is there a benefit? If the driver crashes you'd have to restart the export process anyway.

dmitrii-ubskii · 2025-06-06T17:32:08Z

rust/src/database/database_manager.rs

+    #[cfg_attr(not(feature = "sync"), doc = "driver.databases().import_file(name, schema, data_path).await;")]
+    /// ```
+    #[cfg_attr(feature = "sync", maybe_async::must_be_sync)]
+    pub async fn import_file(


import_database_from_file

_database_ is clearly redundant, we don't call create create_databaseinstead.from` is fine, I will use it, alright. Although it feels redundant.

dmitrii-ubskii · 2025-06-06T17:34:38Z

rust/src/database/database_manager.rs

+                let item = item?;
+                item_buffer.push(item);
+                if item_buffer.len() >= ITEM_BATCH_SIZE {
+                    import_stream.items(item_buffer.split_off(0))?;


Another option is mem::replace(&mut item_buffer, Vec::with_capacity(ITEM_BATCH_SIZE)).
Neither is particularly readable, but split_off is at least shorter.

dmitrii-ubskii · 2025-06-06T17:35:49Z

rust/src/database/migration.rs

+    }
+}
+
+pub(crate) fn try_creating_export_file(path: impl AsRef<Path>) -> Result<File> {


try_create_... and so on, ing is against convention.

## Usage and product changes Add database export and database import operations. Database export saves the database information (its schema and data) on the client machine as two files at provided locations: ``` # database export <name> <schema file location> <data file location> database export my-database export/my-database.typeql export/my-database.typedb ``` Database import uses the exported files to create a new database with equivalent schema and data: ``` # database export <name> <schema file location> <data file location> database import their-database export/my-database.typeql export/my-database.typedb ``` ## Implementation Add new commands similar to other `database` interfaces. Prepare data to call the respective [typedb-driver APIs](typedb/typedb-driver#758).

farost added type: feature priority: high labels May 28, 2025

farost commented May 28, 2025

View reviewed changes

farost changed the title ~~Introduce file-based export and import~~ Introduce file-based database export and import May 29, 2025

farost commented May 29, 2025

View reviewed changes

farost marked this pull request as ready for review May 29, 2025 10:28

farost requested review from dmitrii-ubskii and flyingsilverfin as code owners May 29, 2025 10:28

farost mentioned this pull request May 29, 2025

Introduce database export and import typedb/typedb-console#292

Merged

farost commented Jun 3, 2025

View reviewed changes

dmitrii-ubskii approved these changes Jun 6, 2025

View reviewed changes

farost added 17 commits June 10, 2025 09:54

Trigger CI

df66376

A dumb try of file reading

eb3c733

Finish encoding and decoding of migration items

0a0ac2c

Update protocol ref

2a744dc

Add import and export endpoints

448f1a8

Implement the first ugly version of import and export

5670e06

Refactor import

6fa89cc

Introduce batched reading of files and sending for import

97e9dc4

Optimize network. Part 1

84b7c1f

Refactoring

bac04e0

Add Java and Python interfaces. Update docs

286368c

Intentionally break CI and test correct exit codes in docs checks

80addc0

Cleanup todo, fix docs

55c0fcc

Add a todo

f48b276

Update deps

72b5b9c

Reorder operations order

a8b9753

Implement export and import BDD steps in Rust

c38ded6

farost added 10 commits June 10, 2025 09:55

Fix migration items decoding and file handling

97eb325

Implement test steps for Rust and Java. Fix shutdown for import

3b85ab2

Add tests to Python

d50493e

Update docs for MigrationErrors

8a30ab2

Renames and cleanup

82441eb

Change methods signature to from and to. Update docs

5d113b2

Update deps

07f733e

Update Cargo files

631c1b1

Update bdd reef

270584d

Update typedb artifacts

b536c8b

farost force-pushed the 3.x-export-import branch from 3fe2be9 to b536c8b Compare June 10, 2025 14:43

farost merged commit 6db6f94 into typedb:master Jun 10, 2025
8 checks passed

farost deleted the 3.x-export-import branch June 10, 2025 15:07

		use crate::{error::try_release_arc, iterator::iterator_arc_next};
		use crate::{error::try_release_arc, iterator::iterator_arc_next, memory::borrow};

Introduce file-based database export and import #758

Introduce file-based database export and import #758

Uh oh!

Conversation

farost commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Usage and product changes

Implementation

Database export

Database import

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

farost commented May 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmitrii-ubskii left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

farost Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

farost commented May 28, 2025 •

edited

Loading

farost Jun 9, 2025 •

edited

Loading