Use `tar` archival format to transfer file tree structure and data between Polykey vaults and local file systems

[aryanjassal]

Specification

There are times when we need to transfer the secrets from a vault to either another vault or the user's file system. Sometimes, only one secret needs to be transferred. Other times, we need to transfer multiple file trees including their directory structure.

As all the vaults are stored on the same encrypted file system (efs), to transfer file trees between vaults, we only need to use regular file copying/moving operations on file systems; something along the lines of fs.promises.copy() should work well to transfer secrets between vaults.

However, doing this between the vaults and the user's file system is not as straightforward. To efficiently transmit the file tree, we will be using an archival format like tar. The tar archival format is inherently streamable, and can be used to zip the file tree into a single file, which can then be transmitted over a RPC call, then be unpacked on the client, effectively transferring the file structure to the user's file system. Of course, we can also compress the resulting file, but we won't get into that quite yet.

Additional context

• Pokykey#799 has seen discussion regarding streaming over file trees using RPC calls.
• gera2ld/tarjs can be used as a zero-dependency package to generate tar from a file system.
• matrixai/js-virtualtar can be looked into, as this was also attempting to make streamable tar bindings for JavaScript.

Tasks

1. Make an RPC handler responsible of copying/moving file tree.
2. To move file tree between vaults, just use the fs operations. Multiple locks might be required if transferring between multiple vaults.
3. To move file tree between vaults and file systems, make a tarball and stream it over RPC instead.

[linear]

ENG-412 Use `tar` archival format to transfer file tree structure and data between Polykey vaults and user file systems

[aryanjassal]

After taking a look at matrixai/js-virtualtar, I have realised that significant work still needs to go into the repo before it would be ready to be used within the Polykey ecosystem. This alone might take a cycle or two. I will need to play around with it before starting development on it.

[tegefaulkes]

What needs to be done before it's ready to be used?

[aryanjassal]

I haven't taken an in-depth dive into the requirements, just a brief skim-through over the code. As such, I'm not sure what exactly needs to be done for that. I'll need to see what it already has and what still needs to be done when I start working on this issue.

[aryanjassal]

The first iteration of the cp and mv command will only perform the operations between two vaults. Local file paths support is another topic, and will be implemented later on. As such, this issue is no longer blocking secrets cp and secrets mv.

[aryanjassal]

We should look into tar integration sooner than later. The current state of js-virtualtar is that it is using JavaScript and was last updated a 7 years ago. The repo it was forked from has a commit from this year. So, should I rebase and update our repo? Or use it as-is?

I'd say updating the js-virtualtar seems like our best bet. We will need to convert it to use TypeScripts and update the code to be more modern. This might take a cycle or two, but would be well worth it.

From what I could tell, our library does not include streaming support yet. This is something pretty important to our use case, so this would need to be implemented in whatever solution we go ahead with.

I am yet to look into the repo itself and play around with it to get a better idea of its functionality, so I will mention that later.

[CMCDragonkai]

You could start from scratch. The existing virtual tar codebase isn't up to our standards anymore. But you can use the same repo.

[CMCDragonkai]

There's an existing node version of far in npm and it's probably still up to date. You can use chatgpt to help generate the right protocol code to avoid any nodeisms.

[aryanjassal]

As we only need to use a subset of features for our tar implementation, it can be made much simpler.

The high-level structure is very easy, with header blocks being used to track directories and files. An example archive could look like this.

[Header for "dir1/"]  (512 bytes)
[Header for "dir1/file.txt"]  (512 bytes)
[File content for "file.txt"]  (padded to 512-byte boundary)
[Header for "dir2/"]  (512 bytes)
[Header for "dir2/image.png"]  (512 bytes)
[File content for "image.png"]  (padded to 512-byte boundary)
[End-of-archive marker] (1024 bytes of nulls)

Each 'block' is 512 bytes. Each block can be streamed as it is complete, making tar inherently compatible with streams. If the contents of a file is larger than 512 bytes, then multiple blocks are used until the entire file has been read. Then, the last block is padded to 512 bytes. Note that due to the block size, the tar files would have a larger overhead, especially for purposes like storing a couple bytes of a secret. This is a trade-off we just need to live with, as adding compression would be overkill for this application.

The format of a header block can look like this.

Offset	Size (bytes)	Field	Description
0	100	File name	Null-terminated ASCII/UTF-8 path, relative to the archive root.
100	8	File mode	Unix file permissions (octal, null-padded).
108	8	Owner UID	Numeric UID (octal, null-padded).
116	8	Owner GID	Numeric GID (octal, null-padded).
124	12	File size	Size of the file in bytes (octal, null-padded). If a directory, set to 0.
136	12	Modification time	Unix timestamp (octal, null-padded).
148	8	Checksum	Checksum of the header (filled with spaces during computation).
156	1	Type flag	'0' = regular file, '5' = directory.
157	100	File owner name	Null-terminated ASCII/UTF-8.
257	6	Magic	"ustar\0" (UStar magic).
263	2	Version	"00" (UStar format version).
265	32	Owner user name	Null-terminated ASCII/UTF-8.
297	32	Owner group name	Null-terminated ASCII/UTF-8.
329	8	Device major	Always 0 (unused in this minimal format).
337	8	Device minor	Always 0 (unused in this minimal format).
345	155	Padding (Prefix)	Used for long file names. Set to nulls if not used.
500	12	Unused	Zero-filled.

The checksum field (148–155) is calculated as the sum of all header bytes, treating them as unsigned integers, with the checksum field itself filled with spaces (ASCII 32).

Directories have no file data (size 0 in the header) and do not need padding.

To verify if a file is a valid tar file, the magic bytes at offset 257 can be read. If it doesn't match the string ustar\0, then it is not a valid tar archive.

This implementation is a simple implementation ignoring symbolic links, device, files, or extended attributes, assuming we are working with only regular files and directories. This implementation also provides options to include usernames, group names, permissions, etc. but these fields will be unused in the version required and used by Polykey, so they can be safely ignored and padded with zeros.

While this implementation is focusing on being simplistic, it should still work with general-purpose archival software which can work with tar files, like GNU tar, etc.

[CMCDragonkai]

It's also ideal that you can generate the tar archives deterministically. That may require a natural order of the files/nodes. Which should be possible when tarring from efs but from the real fs, then research "deterministic tar".