Metadata size in tables

[hyanwong]

With tree sequences from real data, some of the metadata columns are likely to take up quite a lot of space (e.g. most of the storage). It would be helpful to know this when looking at a tree sequence e.g. with tskit info.

At the moment, we just report "Has Metadata: No/Yes" for each table. Might it be more informative to give "Metadata: None/XXX bytes" instead? In the html output we could simply put some sort of null marker (e.g. ∅, or just left blank) versus the byte size, rather than ✅.

We could either report the size of the metadata column, or the size of the metadata + metadata_offset columns. I guess the first makes a little more sense.

[jeromekelleher]

Good idea, and exactly what I'm seeing in terms of storage use. Another column "Metadata size" could be good - I interpret the "Has Metadata" column as "does this have a schema"?

So, perhaps we could have two columns "Metadata Schema" and "Metadata Size" (or a single column "Metadata" split into two, "Schema" and "Size"). Ideally, the "Schema" column would be clickable to see a representation of the schema.

No need for anything that fancy in a first pass though.

[hyanwong]

As an alternative, we could quote the percentage of the table bytes devoted to metadata?

I interpret the "Has Metadata" column as "does this have a schema"

Is that true, though? I assumed that a table with metadata contents but no schema (which we do get quite a lot!) would still have a green tick.

[jeromekelleher]

No, it probably isn't correct - would be a useful thing to indicate though.

Quoting size as a percentage of table size would be useful I think.

[hyanwong]

No, it probably isn't correct - would be a useful thing to indicate though.

Quoting size as a percentage of table size would be useful I think.

So two columns replace the "has metadata" column, named something like "has schema" and "% metadata"?

[jeromekelleher]

I think that'd work well, yeah.

[hyanwong]

Instead of a tick to indicate presence/absence of a schema, we could simply print the codec (or blank for a schema with no codec)? Then we can see if it's a struct or json (or anything else we come up with)