bundle: filesystem metadata format

[philips]

@stevvooe and I caught-up in person about our digest discussion and the need for serialize file-system metadata. If you want to read my attempt it is found here: #5 (comment)

Problem: a rootfs for a container bundle sitting on-disk may not reflect the exact intended state of the bundle when it was copied to its current location. Possible causes might include: running on filesystems with varying levels of metadata support (nfs w/o xattrs), accidental property changes (chown -R), or purposeful changes (xattrs added to enforce local policies).

Obviously the files contents will be identical so that isn't a concern.

Solution: If we hope to create a stable digest of the bundle in the face of these likely scenarios we should store the intended filesystem metadata into a file itself. This can be done in a variety of ways and this issue is a place to discuss pros/cons. As a piece of prior-art @vbatts has implemented https://github.com/vbatts/tar-split and we have the linux package managers with tools to verify and restore filesystem metadata from a database with rpm -a --setperms and rpm -V.

[stevvooe]

After some thought, we need a format that does the following:

1. A manifest enumerates files in a bundle.
   1. Provides a content hash.
   2. Provides a path.
   3. Provides a file type.
   4. Provides standard file mode.
   5. Provides xattr.
   6. Provides an extension mechanism.
      1. Geared towards multiple OS support.
      2. Not infinitely extendable but should be easy to add new field.
2. Bundle contents attributes can be reset to contents of file manifest.
   1. Bundle is scanned and any differences from manifest are rectified.
   2. Unames/Gnames/Uid/Gid can be mapped during "reset".
3. Bundle contents can be verified against manifest.
   1. Content hash can be checked.
   2. Attributes can be checked.
      1. Certain attributes can be checked against machine-local mapping (uid/gid, etc.).
   3. Manifest can optionally be signed.

Meeting the above three use cases with this format puts us above the test for Cost problems with the ideas from #5. Requirement 2 above has a common use case, by avoiding placing unreasonable requirements on transports. Requirement 3 above gives us the functionality of #5 with the extra benefits of Requirement 2.

@shykes @philips @crosbymichael

[philips]

@stevvooe I am having a hard time parsing this sentence: "Meeting the above three use cases with this format puts us above the test for Cost problems with the ideas from #5."

I agree with all three needs overall despite my confusion above.

[stevvooe]

@philips That is a poor sentence where I've shoved in a lot of meaning.

What I'm saying is that, given the goals of #5 (cryptographic verification), the cost of scanning a bundle is not warranted. Given these new goals, the cost is warranted and it makes bundles portable over different transports. Basically, we have a solid reason for filesystem scanning that could also be used as a signable target.

[philips]

@stevvooe Ack. So the next step is a .proto file?

[stevvooe]

No better way to get started than with a straw man:

syntax = "proto3";

package ocf.bundle;

// BundleManifest specifies the entries in a container bundle, keyed and
// sorted by path.
message BundleManifest {

    message Entry {
        // path specifies the path from the bundle root
        string path = 1;

        // NOTE(stevvooe): Need to define clear precedence for user/group/uid/gid precedence.

        string user = 2;
        string group = 3;

        uint32 uid = 4;
        uint32 gid = 5;

        // mode defines the file mode and permissions. We've used the same
        // bit-packing from Go's os package,
        // http://golang.org/pkg/os/#FileMode, since they've done the work of
        // creating a cross-platform layout.
        uint32 mode = 6;

        // NOTE(stevvooe): Beyond here, we start defining type specific fields.

        // digest specifies the content digest of the target file. Only valid for
        // regular files. The strings are formatted as <alg>:<digest hex bytes>.
        // The digests are added in order of precedence favored by the 
        // generating party.
        repeated string digest = 7;

        // target defines the target of a hard or soft link, relative to the
        // bundle root.
        string target = 8;

        // specifies major and minor device numbers for charactor and block devices.
        string major = 9;
        string minor = 10;

        message XAttr {
            string name = 1;
            string value = 2;
        }

        // xattr provides storage for extended attributes for the target resource.
        repeated XAttr xattr = 11;

        // AlternateDataStream represents NTFS Alternate Data Streams for 
        // the targeted resource.
        message AlternateDataStream {
            string name = 1;
            bytes value = 2;
        }

        // ads stores one or more alternate data streams for the given resource.
        repeated AlternateDataStream ads = 12;
    }

    repeated Entry entries = 1;
}

Changes:

• digest field is now repeated
• Added description for AlternateDataStream type, formerly ADS

[philips]

@stevvooe looks pretty good. Two questions:

• What is an ADS?
• Should digest be repeated so we can deprecate old hashes and upgrade to new ones over time?

[stevvooe]

What is an ADS?

This is NTFS equivalent of extended attributes (sort of), known as "Alternate Data Streams". The semantics are slightly different, so I've pulled it out into a separate type. Notice the use of type bytes for the value, instead of string. I'd like to get some feedback from a Windows expert to see if this is sufficient.

Should digest be repeated so we can deprecate old hashes and upgrade to new ones over time?

In this case, I don't see why not.

I've updated the comment in-line.

[stevvooe]

@philips We may also want to define an exclusion operator to the manifest specification, since it operates at the bundle level.

enum Op {
    // EXCLUDE specifies that the matched files should be explicitly exlcuded
    // from the manifest. They may be still part of the bundle.
    EXCLUDE = 0;

    // INCLUDE specifies that the resource should be include in the manifest.
    // This has the act of "pinning" the resource. For example, if the resouce
    // is later matched by an exclude statement, it will still be included.
    INCLUDE = 1;
}

message PathSpec {
    Op operation = 1 [default=EXCLUDE];

    // path specifies a path relative to the bundle root. If the path is a
    // directory, the entire tree will be excluded.
    string path = 2;

    // pattern specifies a glob to match resources and apply the operation.
    string pattern = 3;
}

// path specifies the bundle paths covered by the manifest. Specifications are
// order by precedence. For a given path, only the first matching
// specification applies. During processing, it is important to fully process
// all resources, even if a directory is excluded, since child resources may
// first match an inclusion.
repeated PathSpec pathSpec;

Benefits:

• Very explicit for recalculation
• Allows us to catch files that may have showed up and don't belong

Cost:

• More processing time
• Subtle include/exclusion behavior to minimally specify file sets
• Can contribute to instability of file -- equivalent manifests may have different specs

Another possibility is to allow this to be specified on the command line when first building the manifest. That doesn't allow us to catch "extra" files, but that may not be that important and likely doesn't warrant the extra complexity.

[bitshark]

Okay, so after reading this, I think this makes a bit more sense to me . . . I think the area where I'm having trouble is understanding the scope vis a vis the goals...

I think what I've read so far is well thought out and reasoned, so props to everyone , heh. Forgive me ahead of time, but I wanted to share some thoughts. Discard if you don't think they are useful.

There seems a basic consensus -- in the limited info I've read on the container crypto goals --that everyone would probably be on board with the following ideas /in principle/ , certainly as options... These are labeled as the 'Basic Themes' below .. areas where there's agreement:

Basic Themes

• Containers may need to be secured against A-MITM in-transit (integrity)
• Containers may need to be secured against A-MITM during distribution (integrity)
• We need a way of validating and identifying who is the author of a container (authentication, non-repudiation, integrity)
• Containers do NOT need their own transport layer, since we are assuming TLS / OpenSSH will take care of that .
• <--- Additional reasons go here --->

Just as general thought, first I'm going to put these here just as problems I have run into myself with design of cryptography... You guys may know this already but writing this down helps me organize my thoughts.

I'll be back with specifics tomorrow or Saturday.

Statements in general I've found true in crypto engineering. May be useful in this context.

• Never reinvent the wheel in crypto -- this is the 'once upon a time' for systems that get owned
• If the full description & implementation of yr system can withstand the scrutiny of a 16-year old Norwegian hacker named Jon , such that it cannot be cracked in 8 lines of Perl, then you are off to a decent start. (unlike DeCSS)
• Always aim for zero-knowledge even though it's near impossible to do right... The line of thinking provides insight regardless.
• Never put new / novel cryptography and/or standards directly into production -- if necessary , only do so if it's backed by scientific research and a rock-solid implementation
• Don't do an implementation from scratch -- always use an open-source well-tested library like NaCl, PyCrypto, etc.
• Side channel attacks are not theoretical (Heartbleed, WEP, Lucky13, etc). Leaky system borders are where side channels live.
• Padding oracle attacks like Heartbleed from crappy solutions in SSL/TLS RFCs for how to authenticate packets... Do not repeat, heh.
• Work at the highest level possible, in primitives/concepts rather than in methods
• Use nonces, timestamps, & digital signatures wherever possible to avoid replay attacks
• Correct forward operation is Compress, Encrypt, then MAC/Authenticate (never the opposite)... The correct reverse operation is MAC/Authenticate, Decrypt, then Decompress. ("Cryptographic Doom Principle" for side channels)
  http://www.thoughtcrime.org/blog/the-cryptographic-doom-principle/
• Study existing protocols... always adapt an existing format or protocol instead of writing one from scratch
• Scope creep in crypto can be brutal, it goes on forever

- Define our threat model before we do anything else. Who are we trying to protect against?

What are we trying to protect (data, secrets, access, etc)? What is our tolerance in terms of a threat vs it's complexity/likelyhood?

I've got a series of specific questions in regards to the proposed standards here (which I think are pretty awesome )... but I'm tired at the moment -- I'll post the specific questions / comments tomorrow: I hope this helps in the time being . . . This is my take on how to think about crypto engineering.

General Questions:

• What is our threat model? Who is the attacker? What are their capabilities/
• What does a passive attack look like (if one is possible here)?
• What form is an active attack? What is the weakest link in the existing thinking?
• Do we need Digests? What threat do they address and how?
• Do we need Digital Sigs / MACs? How are Digital Sigs keyed? What threat do they address?
• Do we have a line of thinking for the key distribution model? WoT / CA etc. Here it's often there's no choice but that we must piggyback on an existing implementation like OpenPGP (like Ubuntu PPA)
• Based on the threat model, what are the appropriate countermeasures (aka cryptogrpahic primitives)?
• Do we need to support encryption as a use-case, or can we cut it with just hashes / digital sigs?

Specific Questions (assuming key distribution is solved, heh... like with OpenPGP):

• Should we deploy Digital Signatures to secure 'single-file' compressed container images (like rkt does now)?
• Do we know if containers will always (or at least almost all of the time) be distributed on the Internets as some sort of compressed single-file archive ( exact format is irrelevant)?
• If containers are distributed as a single-file archive, and these are secured properly with digital sigs and/or GPG etc, why is it necessary (or beneficial) to secure the data-at-rest in an uncompressed container?
• Has a container format been finalized or standardized beyond the json manifest? What is certain about the container archive format, and what is up in the air? What will be up to the implementation, and what is defined by the spec?
• What are the advantages of securing data-at-rest inside the container vs simply securing a container archive image file? Do we gain security here? If so, what specific attack does this prevent?
• What are the pitfalls in scope creep and/or other impacts of securing data beyond a single-file container archive?
• Are there any compromise solutions that lie between these two ideas?
• What is the threat model we are considering?
• Is a threat where someone tricks me into downloading a malicious container?
• Or does someone poison the ARP table at a starbucks while I am downloading the container image, such that I download their container image?
• Are we protecting against a 'runaway app' inside a container? A malicious actor on the host system which has done privilege escalation? Industrial espionage by ceiling cat?
• What sort of primitives do we need to achieve stated level of security?
• What can we re-use to minimize the effort and make it easy (or certainly at least straightforward) for others to implement this standard?
• What can we re-use to avoid implementation , testing, and possible exploit headaches?

BTW these questions arent meant to all have answers obviously. They are more like engineering food .

Anyway -- thanks, good work, and good luck gentleman. Looking forward to writing about the details tomorrow if time permits, as you all have some really good ideas here, and I'm sure you'll sort this all out.

[vbatts]

this fileset would be a binary packed format, like CrAU?