Provide a TAP device to enclave application

.github/workflows/golangci-lint.yml

@@ -18,7 +18,7 @@ jobs:
     steps:
       - uses: actions/setup-go@v3
         with:
-          go-version: 1.17
+          go-version: 1.19
       - uses: actions/checkout@v3
       - name: golangci-lint
         uses: golangci/golangci-lint-action@v3

.gitignore

@@ -0,0 +1 @@
+cmd/nitriding

Makefile

@@ -1,11 +1,18 @@
-.PHONY: all test lint
+.PHONY: all test lint clean
 
-godeps = *.go go.mod go.sum
+binary = cmd/nitriding
+godeps = *.go go.mod go.sum cmd/*.go
 
-all: test lint
+all: test lint $(binary)
 
 lint:
 	golangci-lint run
 
 test: $(godeps)
 	@go test -cover ./...
+
+$(binary): $(godeps)
+	make -C cmd/
+
+clean:
+	make -C cmd/ clean

README.md

@@ -2,95 +2,82 @@
 
 [![GoDoc](https://pkg.go.dev/badge/github.com/brave/nitriding?utm_source=godoc)](https://pkg.go.dev/github.com/brave/nitriding)
 
-This package helps with building Go-based Web applications on top of AWS Nitro
-Enclaves.  The package provides the following features:
-
-1. Automatically obtains an HTTPS certificate (either self-signed or via [Let's
-   Encrypt](https://letsencrypt.org)) for clients to securely connect to your
-   enclave over the Internet.
-
-2. Automatically exposes an HTTPS endpoint for remote attestation.  After
-   having audited your enclave's source code, your users can conveniently
-   verify the enclave by using a tool like
-   [verify-enclave](https://github.com/brave-experiments/verify-enclave)
-   and running:
+This Go tool kit makes it possible to run your application inside an
+[AWS Nitro Enclave](https://aws.amazon.com/ec2/nitro/nitro-enclaves/).
+Let's assume that you built a Web service in Rust.  You can now use nitriding to
+move your Rust code into a secure enclave, making it possible for your users to
+remotely verify that you are in fact running the code that you claim to run.
+Nitriding provides the following features:
+
+* Automatically obtains an HTTPS certificate (either self-signed or via
+  [Let's Encrypt](https://letsencrypt.org))
+  for clients to securely connect to your enclave over the Internet.  Nitriding
+  can act as a TLS-terminating reverse HTTP proxy for your application, so your
+  application does not have to deal with obtaining certificates.
+
+* Automatically exposes an HTTPS endpoint for remote attestation.  After having
+  audited your enclave's source code, your users can conveniently verify the
+  enclave's image by using a tool like
+  [verify-enclave](https://github.com/brave-experiments/verify-enclave)
+  and running:
 
    ```
-   make verify CODE=/path/to/code/ ENCLAVE=https://example.com/attest
+   make verify CODE=/path/to/code/ ENCLAVE=https://enclave.com/enclave/attestation
    ```
 
-3. Provides an API for the enclave application to securely share confidential
-   key material with an identical, remote enclave.
+* Are you building an application that uses a protocol other than HTTP?  If so,
+  nitriding makes it possible to register a hash over your application's public
+  key material which is subsequently included in the
+  [attestation document](https://docs.aws.amazon.com/enclaves/latest/user/nitro-enclave-concepts.html#term-attestdoc).
+  This allows your users to verify that their connection is securely terminated
+  inside the enclave, regardless of the protocol that you are using.
 
-4. Starts a proxy component that transparently translates between IP and VSOCK,
-   so you can write IP-based networking code without having to worry about
-   the enclave's constrained VSOCK interface.
+* Provides an API to scale enclave applications horizontally while synchronizing
+  state between enclaves.
 
-5. Automatically initializes the enclave's entropy pool using the Nitro
-   hypervisor.
+* AWS Nitro Enclaves only provide a highly constrained
+  [VSOCK channel](https://docs.aws.amazon.com/enclaves/latest/user/nitro-enclave-concepts.html#term-socket)
+  between the enclave and its host.  Nitriding creates TAP interface inside the
+  enclave, allowing your application to transparently access the Internet
+  without having to worry about VSOCK, port forwarding, or tunneling.
+
+* Automatically initializes the enclave's entropy pool using the Nitro
+  hypervisor.
 
 To learn more about nitriding's trust assumptions, architecture, and build
 system, take a look at our [research paper](https://arxiv.org/abs/2206.04123).
 
-## Configuration
-
-Nitriding's
-[configuration object](https://pkg.go.dev/github.com/brave-experiments/nitriding#Config)
-contains comments that explain the purpose of each variable.
+## Usage
 
-## Example
+To use nitriding, the following steps are necessary:
 
-Use the following "hello world" example to get started.  The program
-instantiates a new Web server that's listening on port 8443, for the domain
-example.com.  It also registers an HTTP handler for the path `/hello-world`
-which, when accessed, simply responds with the string "hello world".
+1. Make sure that your enclave application supports
+   [reproducible builds](https://reproducible-builds.org);
+   otherwise, users won't be able to verify your enclave image.
 
-Note that in order for this example to work, you need to set up two programs on
-the parent EC2 instance:
+2. Set up
+   [this proxy application](https://github.com/containers/gvisor-tap-vsock/tree/main/cmd/gvproxy)
+   on the EC2 host.
 
-1. [viproxy](https://github.com/brave/viproxy) by running:
+3. Bundle your application and nitriding together in a Dockerfile.  The
+   nitriding stand-alone executable must be invoked first, followed by your
+   application.  To build the nitriding executable, run `make cmd/nitriding`.
+   (Then, run `./cmd/nitriding -help` to see a list of command line options.)
+   For reproducible Docker images, we recommend
+   [kaniko](https://github.com/GoogleContainerTools/kaniko)
+   and
+   [ko](https://github.com/ko-build/ko) (for Go applications only).
 
-   ```bash
-   export CID=5 # The CID you assigned when running "nitro-cli run-enclave --enclave-cid X ...".
-   export IN_ADDRS=":8443,:80,3:80"
-   export OUT_ADDRS="${CID}:8443,${CID}:80,127.0.0.1:1080"
-   viproxy
-   ```
+4. Once your application is done bootstrapping, it must let nitriding know, so
+   it can start the Internet-facing Web server that handles remote attestation
+   and other tasks.  To do so, the application must issue an HTTP GET request to
+   `http://127.0.0.1:8080/enclave/ready`.  The handler ignores URL parameters
+   and responds with a status code 200 if the request succeeded.  Note that the
+   port in this example, 8080, is controlled by nitriding's `-intport` command
+   line flag.
 
-2. A SOCKS proxy, e.g.
-   [this one](https://github.com/brave-intl/bat-go/tree/nitro-utils/nitro-shim/tools/socksproxy).
-
-That said, here is the enclave application:
-
-```golang
-package main
-
-import (
-	"fmt"
-	"log"
-	"net/http"
-
-	"github.com/brave/nitriding"
-)
-
-func helloWorldHandler(w http.ResponseWriter, r *http.Request) {
-	fmt.Fprintln(w, "hello world")
-}
-
-func main() {
-	enclave := nitriding.NewEnclave(
-		&nitriding.Config{
-			FQDN:    "example.com",
-			Port:    8443,
-			UseACME: true,
-		},
-	)
-	enclave.AddRoute(http.MethodGet, "/hello-world", helloWorldHandler)
-	if err := enclave.Start(); err != nil {
-		log.Fatalf("Enclave terminated: %v", err)
-	}
-}
-```
+Take a look at [this example application](example/) to learn how nitriding works
+in practice.
 
 ## Development
 
@@ -99,3 +86,8 @@ To test and lint the code, run:
 ```
 make
 ```
+
+## More documentation
+
+* [System architecture](doc/architecture.md)
+* [Example application](example/)

attestation.go

@@ -2,6 +2,7 @@ package nitriding
 
 import (
 	"bytes"
+	"crypto/sha256"
 	"encoding/base64"
 	"encoding/hex"
 	"errors"
@@ -18,6 +19,8 @@ const (
 	nonceLen       = 20           // The size of a nonce in bytes.
 	nonceNumDigits = nonceLen * 2 // The number of hex digits in a nonce.
 	maxAttDocLen   = 5000         // A (reasonable?) upper limit for attestation doc lengths.
+	hashPrefix     = "sha256:"
+	hashSeparator  = ";"
 )
 
 var (
@@ -34,13 +37,32 @@ var (
 	getPCRValues = func() (map[uint][]byte, error) { return _getPCRValues() }
 )
 
-// getAttestationHandler takes as input a SHA-256 hash over an HTTPS
-// certificate and returns a HandlerFunc.  This HandlerFunc expects a nonce in
-// the URL query parameters and subsequently asks its hypervisor for an
-// attestation document that contains both the nonce and the certificate hash.
-// The resulting Base64-encoded attestation document is then returned to the
-// requester.
-func getAttestationHandler(certHash *[32]byte) http.HandlerFunc {
+// AttestationHashes contains hashes over public key material which we embed in
+// the enclave's attestation document for clients to verify.
+type AttestationHashes struct {
+	tlsKeyHash [sha256.Size]byte // Always set.
+	appKeyHash [sha256.Size]byte // Sometimes set, depending on application.
+}
+
+// Serialize returns a byte slice that contains our concatenated hashes.  Note
+// that all hashes are always present.  If a hash was not initialized, it's set
+// to 0-bytes.
+func (a *AttestationHashes) Serialize() []byte {
+	str := fmt.Sprintf("%s%s%s%s%s",
+		hashPrefix,
+		a.tlsKeyHash,
+		hashSeparator,
+		hashPrefix,
+		a.appKeyHash)
+	return []byte(str)
+}
+
+// attestationHandler takes as input an AttestationHashes struct and returns a
+// HandlerFunc.  This HandlerFunc expects a nonce in the URL query parameters
+// and subsequently asks its hypervisor for an attestation document that
+// contains both the nonce and the hashes in the given struct.  The resulting
+// Base64-encoded attestation document is then returned to the requester.
+func attestationHandler(hashes *AttestationHashes) http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
 		if r.Method != http.MethodGet {
 			http.Error(w, errMethodNotGET, http.StatusMethodNotAllowed)
@@ -67,7 +89,7 @@ func getAttestationHandler(certHash *[32]byte) http.HandlerFunc {
 			return
 		}
 
-		rawDoc, err := attest(rawNonce, certHash[:], nil)
+		rawDoc, err := attest(rawNonce, hashes.Serialize(), nil)
 		if err != nil {
 			http.Error(w, errFailedAttestation, http.StatusInternalServerError)
 			return

attestation_test.go

@@ -1,6 +1,9 @@
 package nitriding
 
 import (
+	"bytes"
+	"crypto/sha256"
+	"encoding/base64"
 	"io"
 	"net/http"
 	"net/http/httptest"
@@ -31,7 +34,7 @@ func expect(t *testing.T, resp *http.Response, statusCode int, errMsg string) {
 }
 
 func testReq(t *testing.T, req *http.Request, statusCode int, errMsg string) {
-	attestationHandler := getAttestationHandler(&[32]byte{})
+	attestationHandler := attestationHandler(&AttestationHashes{})
 	rec := httptest.NewRecorder()
 	attestationHandler(rec, req)
 	expect(t, rec.Result(), statusCode, errMsg)
@@ -84,3 +87,44 @@ func TestArePCRsIdentical(t *testing.T) {
 		t.Fatal("Failed to recognize different PCRs as such.")
 	}
 }
+
+func TestAttestationHashes(t *testing.T) {
+	e := createEnclave()
+	appKeyHash := [sha256.Size]byte{1, 2, 3, 4, 5}
+
+	// Start the enclave.  This is going to initialize the hash over the HTTPS
+	// certificate.
+	if err := e.Start(); err != nil {
+		t.Fatal(err)
+	}
+	defer e.Stop() //nolint:errcheck
+	signalReady(t, e)
+
+	// Register dummy key material for the other hash to be initialized.
+	rec := httptest.NewRecorder()
+	buf := bytes.NewBufferString(base64.StdEncoding.EncodeToString(appKeyHash[:]))
+	req := httptest.NewRequest(http.MethodPost, pathHash, buf)
+	e.privSrv.Handler.ServeHTTP(rec, req)
+
+	s := e.hashes.Serialize()
+	expectedLen := sha256.Size*2 + len(hashPrefix)*2 + len(hashSeparator)
+	if len(s) != expectedLen {
+		t.Fatalf("Expected serialized hashes to be of length %d but got %d.",
+			expectedLen, len(s))
+	}
+
+	// Make sure that the serialized slice starts with "sha256:".
+	prefix := []byte(hashPrefix)
+	if !bytes.Equal(s[:len(prefix)], prefix) {
+		t.Fatalf("Expected prefix %s but got %s.", prefix, s[:len(prefix)])
+	}
+
+	// Make sure that our previously-set hash is as expected.
+	expected := []byte(hashSeparator)
+	expected = append(expected, []byte(hashPrefix)...)
+	expected = append(expected, appKeyHash[:]...)
+	offset := len(hashPrefix) + sha256.Size
+	if !bytes.Equal(s[offset:], expected) {
+		t.Fatalf("Expected application key hash of %x but got %x.", expected, s[offset:])
+	}
+}

certcache.go

@@ -2,19 +2,11 @@ package nitriding
 
 import (
 	"context"
-	"errors"
-	"net"
-	"net/http"
 	"sync"
 
-	"github.com/mdlayher/vsock"
 	"golang.org/x/crypto/acme/autocert"
 )
 
-var (
-	errHTTP01Failed = errors.New("failed to listen for HTTP-01 challenge")
-)
-
 // certCache implements the autocert.Cache interface.
 type certCache struct {
 	sync.RWMutex
@@ -53,31 +45,3 @@ func (c *certCache) Delete(ctx context.Context, key string) error {
 	delete(c.cache, key)
 	return nil
 }
-
-func listenHTTP01(errChan chan error, mgr *autocert.Manager) {
-	// Let's Encrypt's HTTP-01 challenge requires a listener on port 80:
-	// https://letsencrypt.org/docs/challenge-types/#http-01-challenge
-	var l net.Listener
-	var err error
-
-	if inEnclave {
-		l, err = vsock.Listen(uint32(80), nil)
-		if err != nil {
-			errChan <- errHTTP01Failed
-			return
-		}
-		defer func() {
-			_ = l.Close()
-		}()
-	} else {
-		l, err = net.Listen("tcp", ":80")
-		if err != nil {
-			errChan <- errHTTP01Failed
-			return
-		}
-	}
-
-	elog.Print("Starting autocert listener.")
-	errChan <- nil
-	_ = http.Serve(l, mgr.HTTPHandler(nil))
-}

cmd/Makefile

@@ -0,0 +1,15 @@
+.PHONY: all lint clean
+
+binary = nitriding
+godeps = *.go ../*.go ../go.mod ../go.sum
+
+all: lint $(binary)
+
+lint:
+	golangci-lint run
+
+$(binary): $(godeps)
+	go build -o $(binary)
+
+clean:
+	rm -f $(binary)