Discuss for .raw Type and Preservation of File Metadata

[chlins]

Background

The current model-spec (as outlined in docs/spec.md) supports bundling model artifacts in various formats, including .tar and potentially .raw. For .tar archives, file metadata such as file mode, ownership, and other attributes can be embedded in the tar header, which is useful for runtimes that rely on this information to properly handle the files (e.g., setting executable permissions or preserving ownership).

However, the .raw type, being a flat, uncompressed file format, does not inherently provide a mechanism to store such metadata. This poses a challenge for runtimes that may depend on these attributes to function correctly, especially in scenarios where permissions or other file properties are critical (e.g., executable scripts or binaries within a model artifact).

Problem

When using the .raw type, there is no clear way to preserve or convey file metadata that was previously available in .tar headers. This limitation could lead to:

• Loss of critical information (e.g., file modes like 755 for executables).
• Inconsistent behavior across runtimes that expect this metadata.
• Potential security or operational issues if the runtime cannot infer the intended file attributes.
  For example, a model artifact might include a script or binary that needs to be executable, but without metadata, the runtime would either need to assume defaults (potentially incorrect) or fail to execute it properly.

Questions for Discussion

• Should .raw artifacts be required to preserve the same metadata as .tar artifacts? If so, what’s the minimum set of attributes we need (e.g., mode, ownership, timestamps)?
• How should runtimes handle cases where metadata is missing for .raw files? Should there be a fallback mechanism?
• Are there existing standards or tools (e.g., OCI image spec) we can borrow from to address this?
  What would be the best way to store this metadata for .raw files while aligning with the goals of simplicity and portability in the model-spec?

[amisevsk]

I'm concerned this is heading in a bit of a strange direction -- do we have use-case examples for what the raw media type might be used for? I.e. cases where it makes sense to have an entirely generic single file, with preserved file metadata?

For example, I have been experimenting with storing tensorizer layers that are more-or-less equivalent to model-spec's raw in the KitOps project. However, even in this case, all we need to know is that it's a tensorizer file (I'm using a separate MediaType, but it could also be a well-known annotation, e.g. model-spec.file-type on a plain raw layer). That's enough context to know how to handle the file, without worrying about filesystem-specific metadata.

If we start to bring the whole filesystem's context into the discussion, we enter pretty weird places quickly -- are we saying we want the model-spec specification to enable distributing executable binaries in raw form? How do we address the security implications of this? I'd be reluctant to use any program that downloads arbitrary binaries from the internet and executes them non-interactively.

Some further questions for discussion:

1. What use cases are we targeting with such a wide-ranging, open feature? Can those use cases be addressed more directly (e.g. the type annotation above)
2. If we want to include file modes (including the executable flag), what are tools expected to do with that information? Should the execute the binary? What if it is malicious? If it's so that the file can be saved with appropriate flags, why is the normal tar format (or a seekable tar format) insufficient?
3. To me, the point of a raw format is that it should be for cases where regular filesystem metadata is irrelevant (e.g. tensorizer files). For these files, any filesystem metadata is meaningless and provides no information on how the file should be processed -- do we care that the tensorizer file has 644 or 640 permissions, when the intention is to load it into GPU memory?
4. If we're talking about single large files in raw format, and then working to preserve the metadata we lose by not using tar, why aren't we using tar for that file? I may be missing something here, but if a tar consists of a single file, all it is is a tar header + the file's contents. The raw case would effectively be "move the tar header into annotations", which sounds complicated for little gain. If a tarball contains a single file, we can already download it in parallel with multiple readers.

[amisevsk]

To add, we have discussed ways of mounting layers from OCI artifacts into normal OCI images (for tar-based layers, this is as simple as switching the mediatype to one the container runtime understands). If we're using raw layers, we're abandoning that (fairly large) benefit, as no runtime -- except for ones we explicitly adapt -- will be able to process them.

[aftersnow]

I'm concerned this is heading in a bit of a strange direction -- do we have use-case examples for what the raw media type might be used for? I.e. cases where it makes sense to have an entirely generic single file, with preserved file metadata?

For example, I have been experimenting with storing tensorizer layers that are more-or-less equivalent to model-spec's raw in the KitOps project. However, even in this case, all we need to know is that it's a tensorizer file (I'm using a separate MediaType, but it could also be a well-known annotation, e.g. model-spec.file-type on a plain raw layer). That's enough context to know how to handle the file, without worrying about filesystem-specific metadata.

If we start to bring the whole filesystem's context into the discussion, we enter pretty weird places quickly -- are we saying we want the model-spec specification to enable distributing executable binaries in raw form? How do we address the security implications of this? I'd be reluctant to use any program that downloads arbitrary binaries from the internet and executes them non-interactively.

Some further questions for discussion:

1. What use cases are we targeting with such a wide-ranging, open feature? Can those use cases be addressed more directly (e.g. the type annotation above)
2. If we want to include file modes (including the executable flag), what are tools expected to do with that information? Should the execute the binary? What if it is malicious? If it's so that the file can be saved with appropriate flags, why is the normal tar format (or a seekable tar format) insufficient?
3. To me, the point of a raw format is that it should be for cases where regular filesystem metadata is irrelevant (e.g. tensorizer files). For these files, any filesystem metadata is meaningless and provides no information on how the file should be processed -- do we care that the tensorizer file has 644 or 640 permissions, when the intention is to load it into GPU memory?
4. If we're talking about single large files in raw format, and then working to preserve the metadata we lose by not using tar, why aren't we using tar for that file? I may be missing something here, but if a tar consists of a single file, all it is is a tar header + the file's contents. The raw case would effectively be "move the tar header into annotations", which sounds complicated for little gain. If a tarball contains a single file, we can already download it in parallel with multiple readers.

Thank you @amisevsk , in my opinion:

1. The raw format is targeting to the big safetensors model weight files( ~10GB each, ~1TB in total), for example, deepseek-r1. If we use tar format, it wastes a lot of storages during tar and untar. Of course, we can work around this, but it's complicated. The totally ~1TB model weight files does not allow us to keep it with tar format and original file at the same time, in local storage.
2. We have met some scenarios that indicate tar is not suitable for LLM serving. For example, if the user changes the safetensors file mode or file create time, the tar file's sha256 result is changed, which leads to duplicated layer. But the file mode is meaningless for LLM serving, we hope we can have the same sha256 for the big safetensors files, regardless of the file meta modification. So raw format is better in this scenario.
3. Yes, I agree, the filesystem metadata is meaningless, so why we need tar? The tar is all about the header + content, the header is the meaningless file meta information, but sacrifices simplicity.
4. Yes, the purpose of raw is keeping the big safetensors file's sha256 unchanged, regardless the file meta, such as file mode. That makes the ~10GB safetensors layer duplicated, wastes a lot of storages, slows down the upload/download speed.

[chlins]

Following is a proposed solution which puts the file metadata in the layer annotations:

type Metadata struct {
    Name    string    `json:"name"`     // File name
    Mode    uint32    `json:"mode"`     // File permission mode (e.g., Unix permission bits)
    Uid     uint32    `json:"uid"`      // User ID (identifier of the file owner)
    Gid     uint32    `json:"gid"`      // Group ID (identifier of the file's group)
    Size    int64     `json:"size"`     // File size (in bytes)
    ModTime time.Time `json:"mtime"`    // File last modification time
    Typeflag byte     `json:"typeflag"` // File type flag (e.g., regular file, directory, etc.)
}

key: org.cnai.model.file.metadata
value:

{
    "name": "example.txt",
    "mode": 420, // JSON serialized as decimal representation, 420 represents 0644
    "uid": 1001,
    "gid": 1001,
    "size": 1024,
    "mtime": "2025-03-31T15:30:00Z",
    "typeflag": 0
}

Example manifest:

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "artifactType": "application/vnd.cnai.model.manifest.v1+json",
    "config": {
        "mediaType": "application/vnd.cnai.model.config.v1+json",
        "digest": "sha256:d5815835051dd97d800a03f641ed8162877920e734d3d705b698912602b8c763",
        "size": 301
    },
    "layers": [
        {
            "mediaType": "application/vnd.cnai.model.doc.v1.tar",
            "digest": "sha256:3f907c1a03bf20f20355fe449e18ff3f9de2e49570ffb536f1a32f20c7179808",
            "size":1024
            "annotations": {
                  "org.cnai.model.filepath": "example.txt",
                  "org.cnai.model.file.metadata": "{\"name\": \"example.txt\", \"mode\": 420, \"uid\": 1001, \"gid\": 1001, \"size\": 1024, \"mtime\": \"2025-03-31T15:30:00Z\", \"typeflag\": 0}"
            }
        }
    ]
}

[gaius-qi]

Following is a proposed solution which puts the file metadata in the layer annotations:

key: org.cnai.model.file.metadata value:

{
"name": "foo.txt",
"size": 123456,
"isDir": false,
"mode": "0644",
"modTime": "2025-03-31T15:30:00+08:00"
}
Example manifest:

{
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"artifactType": "application/vnd.cnai.model.manifest.v1+json",
"config": {
"mediaType": "application/vnd.cnai.model.config.v1+json",
"digest": "sha256:d5815835051dd97d800a03f641ed8162877920e734d3d705b698912602b8c763",
"size": 301
},
"layers": [
{
"mediaType": "application/vnd.cnai.model.weight.v1.tar",
"digest": "sha256:3f907c1a03bf20f20355fe449e18ff3f9de2e49570ffb536f1a32f20c7179808",
"size": 30327160
"annotations": {
"org.cnai.model.filepath": "foo.txt",
"org.cnai.model.file.metadata": "{"name":"foo.txt","size":123456,"isDir":false,"mode":"0644","modTime":"2025-03-31T15:30:00+08:00"}"
}
}
]
}

According to the definition, uid and gid should be added, refer to https://github.com/CloudNativeAI/model-spec/blob/main/docs/spec.md#file-attributes.

Please refer to the tar format naming, refer to https://www.gnu.org/software/tar/manual/html_node/Standard.html.

[chlins]

Following is a proposed solution which puts the file metadata in the layer annotations:
key: org.cnai.model.file.metadata value:
{
"name": "foo.txt",
"size": 123456,
"isDir": false,
"mode": "0644",
"modTime": "2025-03-31T15:30:00+08:00"
}
Example manifest:
{
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"artifactType": "application/vnd.cnai.model.manifest.v1+json",
"config": {
"mediaType": "application/vnd.cnai.model.config.v1+json",
"digest": "sha256:d5815835051dd97d800a03f641ed8162877920e734d3d705b698912602b8c763",
"size": 301
},
"layers": [
{
"mediaType": "application/vnd.cnai.model.weight.v1.tar",
"digest": "sha256:3f907c1a03bf20f20355fe449e18ff3f9de2e49570ffb536f1a32f20c7179808",
"size": 30327160
"annotations": {
"org.cnai.model.filepath": "foo.txt",
"org.cnai.model.file.metadata": "{"name":"foo.txt","size":123456,"isDir":false,"mode":"0644","modTime":"2025-03-31T15:30:00+08:00"}"
}
}
]
}

According to the definition, uid and gid should be added, refer to https://github.com/CloudNativeAI/model-spec/blob/main/docs/spec.md#file-attributes.

Please refer to the tar format naming, refer to https://www.gnu.org/software/tar/manual/html_node/Standard.html.

@gaius-qi Good catch, updated in #44 (comment).

[amisevsk]

While I can see the argument for including information such as file name, I still question the reasoning behind the currently proposed set of metadata to be included here. I missed the model-spec call today, so this may have already been discussed (in which case I apologize for reiterating)

type Metadata struct {
    Name    string    `json:"name"`     // File name
    Mode    uint32    `json:"mode"`     // File permission mode (e.g., Unix permission bits)
    Uid     uint32    `json:"uid"`      // User ID (identifier of the file owner)
    Gid     uint32    `json:"gid"`      // Group ID (identifier of the file's group)
    Size    int64     `json:"size"`     // File size (in bytes)
    ModTime time.Time `json:"mtime"`    // File last modification time
    Typeflag byte     `json:"typeflag"` // File type flag (e.g., regular file, directory, etc.)
}

• Why are we duplicating a size field in this metadata struct, when it is a part of the OCI descriptor spec? What is the expected behavior when these two values do not agree?
• What are type flags used for here? What does it mean to have a symlink or directory as a raw layer?
• Do we really need UID/GID, or does it make sense to just use the current user's UID/GID or some default value (thinking e.g. in a container runtime environment). Similarly, do we need a full set of mode bits? I've expressed my concerns with using this for executable blobs, but I'm also not sure if/when we would ever want to e.g. not let a user access the file.
• What's the purpose of including file last modification time? Since the point of these raw layers is to not change digests when file metadata is changed, how can we rely on mtime carrying information we care about?

I feel like most of these fields can be set to reasonable defaults without affecting behavior/usage (with the added benefit of being slightly more readable):

type Metadata struct {
    Path       string  `json:"name"`       // File relative path (with parent directories created if necessary)
    Executable bool    `json:"executable"` // Whether file is executable
}

I'm taking a lot of this from documentation about reproducible tars (e.g. here)

• Set mode to some reasonable default (e.g. 664) -- maybe include an executable flag if we want to support executable blobs (sets mode to 775 instead)
• UID/GID are defaulted to current user's UID/GID or e.g. 1000/0
• Drop size in favor of the size on the descriptor
• Modtime can be set to the current time, or whatever is convenient
• Assume any file in a raw layer is a regular file and set type flags accordingly