immutability: redesign api to improve verbosity (#40)

* testing: simplify command

Go 1.9 no longer includes the vendor directory in wildcard commands, so we can remove the verbosity.

* docs: expand contributing guide

* docs: rewrite features list

* docs: remove example

The example for the New function is pretty much redundant.

* memcall: remove execute permissions

Related to #37. More refactoring needed.

* dependencies: update

* immutability: improve verbosity

This change changes the nomenclature and API used to improve verbosity and encourage better programming style.

* docs: improve docs summary

* api: rename EqualTo => EqualBytes

The new name is more clear about its intentions.

* patch: race condition

Use internal API to prevent mutex conflicts.

* docs: improve wording

* docs: fix outdated wording

* canary: minor optimisations

* destroy: refactor and optimise

.travis.yml

@@ -6,8 +6,7 @@ os:
 
 script:
     - go build -race -v .
-    - go test -race -v ./memcall/...
-    - go test -race -v ./
+    - go test -race -v ./...
 
 notifications:
   email: false

CONTRIBUTING.md

@@ -3,5 +3,8 @@
 We accept contributions from anyone. Here is the general process:
 
 1. Make a fork of the repository.
-2. Make your changes. If you're a first-time contributor, also add yourself to the [`AUTHORS`](/AUTHORS) file (in alphabetical order).
-3. Open a PR against the `development` branch.
+2. Make your changes, preferably to the `development` branch.
+3. Add your contribution to the [`AUTHORS`](/AUTHORS) file. If you're a first-time contributor, first add your name and email (in alphabetical order).
+4. Open a PR against the `development` branch.
+
+Changes are squashed and merged into `development`, and then `development` is merged conventionally into `master`.

README.md

@@ -1,7 +1,7 @@
 <p align="center">
   <img src="https://cdn.rawgit.com/awnumar/memguard/master/logo.svg" height="140" />
   <h3 align="center">MemGuard</h3>
-  <p align="center">Pure go library for secure handling of sensitive memory.</p>
+  <p align="center">Easy and secure handling of sensitive memory, in pure Go.</p>
   <p align="center">
     <a href="https://travis-ci.org/awnumar/memguard"><img src="https://travis-ci.org/awnumar/memguard.svg?branch=master"></a>
     <a href="https://ci.appveyor.com/project/awnumar/memguard/branch/master"><img src="https://ci.appveyor.com/api/projects/status/nrtqmdolndm0pcac/branch/master?svg=true"></a>
@@ -16,13 +16,14 @@ This is a thread-safe package, designed to allow you to easily handle sensitive
 
 ## Features
 
-* Memory is allocated using system calls, thereby bypassing the Go runtime and preventing the GC from messing with it.
-* To prevent buffer overflows and underflows, the secure buffer is sandwiched between two protected guard pages. If these pages are accessed, a SIGSEGV violation is triggered.
-* The secure buffer is prepended with a random canary. If this value changes, the process will panic. This is designed to prevent buffer underflows.
-* All pages between the two guards are locked to stop them from being swapped to disk.
-* The secure buffer can be made read-only so that any other action triggers a SIGSEGV violation.
-* When freeing, all secure memory is wiped.
-* The API also includes functions for time-constant copying and comparison, disabling system core dumps, and catching signals.
+* Interference from the garbage-collector is blocked by using system-calls to manually allocate memory ourselves.
+* It is very difficult for another process to find or access sensitive memory as the data is sandwiched between guard-pages. This feature also acts as an immediate access alarm in case of buffer overflows.
+* Buffer overflows are further protected against using a random canary value. If this value changes, the process will panic.
+* We try our best to prevent the system from writing anything sensitive to the disk. The data is locked to prevent swapping, system core dumps can be disabled, and the kernel is advised (where possible) to never include the secure memory in dumps.
+* True kernel-level immutability is implemented. That means that if _anything_ attempts to modify an immutable container, the kernel will throw an access violation and the process will terminate.
+* All sensitive data is wiped before the associated memory is released back to the operating system.
+* Side-channel attacks are mitigated against by making sure that the copying and comparison of data is done in constant-time.
+* Accidental memory leaks are mitigated against by harnessing Go's own garbage-collector to automatically destroy containers that have run out of scope.
 
 Some of these features were inspired by [libsodium](https://github.com/jedisct1/libsodium), so credits to them.
 

appveyor.yml

@@ -15,8 +15,7 @@ build_script:
 - go build -race -v .
 
 test_script:
-    - go test -race -v ./memcall/...
-    - go test -race -v ./
+- go test -race -v ./...
 
 notifications:
 - provider: Slack

container.go

@@ -0,0 +1,92 @@
+package memguard
+
+import (
+	"crypto/subtle"
+	"runtime"
+	"sync"
+	"unsafe"
+
+	"github.com/awnumar/memguard/memcall"
+)
+
+/*
+LockedBuffer is a structure that holds secure values.
+
+The protected memory itself can be accessed with the Buffer() method. The various states can be accessed with the IsDestroyed() and IsMutable() methods, both of which are pretty self-explanatory.
+
+The number of LockedBuffers that you are able to create is limited by how much memory your system kernel allows each process to mlock/VirtualLock. Therefore you should call Destroy on LockedBuffers that you no longer need, or simply defer a Destroy call after creating a new LockedBuffer.
+
+The entire memguard API handles and passes around pointers to LockedBuffers, and so, for both security and convenience, you should refrain from dereferencing a LockedBuffer.
+
+If an API function that needs to edit a LockedBuffer is given one that is immutable, the call will return an ErrImmutable. Similarly, if a function is given a LockedBuffer that has been destroyed, the call will return an ErrDestroyed.
+*/
+type LockedBuffer struct {
+	*container  // Import all the container fields.
+	*littleBird // Monitor this for auto-destruction.
+}
+
+// container implements the actual data container.
+type container struct {
+	sync.Mutex // Local mutex lock.
+
+	buffer  []byte // Slice that references the protected memory.
+	mutable bool   // Is this LockedBuffer mutable?
+}
+
+// littleBird is a value that we monitor instead of the LockedBuffer
+// itself. It allows us to tell the GC to auto-destroy LockedBuffers.
+type littleBird [16]byte
+
+// Global internal function used to create new secure containers.
+func newContainer(size int, mutable bool) (*LockedBuffer, error) {
+	// Return an error if length < 1.
+	if size < 1 {
+		return nil, ErrInvalidLength
+	}
+
+	// Allocate a new LockedBuffer.
+	ib := new(container)
+	b := &LockedBuffer{ib, new(littleBird)}
+
+	// Round length + 32 bytes for the canary to a multiple of the page size..
+	roundedLength := roundToPageSize(size + 32)
+
+	// Calculate the total size of memory including the guard pages.
+	totalSize := (2 * pageSize) + roundedLength
+
+	// Allocate it all.
+	memory := memcall.Alloc(totalSize)
+
+	// Make the guard pages inaccessible.
+	memcall.Protect(memory[:pageSize], false, false)
+	memcall.Protect(memory[pageSize+roundedLength:], false, false)
+
+	// Lock the pages that will hold the sensitive data.
+	memcall.Lock(memory[pageSize : pageSize+roundedLength])
+
+	// Set the canary.
+	subtle.ConstantTimeCopy(1, memory[pageSize+roundedLength-size-32:pageSize+roundedLength-size], canary)
+
+	// Set Buffer to a byte slice that describes the reigon of memory that is protected.
+	b.buffer = getBytes(uintptr(unsafe.Pointer(&memory[pageSize+roundedLength-size])), size)
+
+	// Set appropriate mutability state.
+	b.mutable = true
+	if !mutable {
+		b.MakeImmutable()
+	}
+
+	// Use a finalizer to make sure the buffer gets destroyed if forgotten.
+	runtime.SetFinalizer(b.littleBird, func(_ *littleBird) {
+		go ib.Destroy()
+	})
+
+	// Append the container to allLockedBuffers. We have to add container
+	// instead of LockedBuffer so that littleBird can become unreachable.
+	allLockedBuffersMutex.Lock()
+	allLockedBuffers = append(allLockedBuffers, ib)
+	allLockedBuffersMutex.Unlock()
+
+	// Return a pointer to the LockedBuffer.
+	return b, nil
+}

docs.go

@@ -1,29 +1,50 @@
 /*
 Package memguard lets you easily handle sensitive values in memory.
 
-The general working cycle is as follows:
+    package main
 
-    // Create a new writable LockedBuffer of length 16.
-    encryptionKey, err := memguard.New(16, false)
-    if err != nil {
-        panic(err)
+    import (
+        "fmt"
+
+        "github.com/awnumar/memguard"
+    )
+
+    func main() {
+        // Tell memguard to listen out for interrupts, and cleanup in case of one.
+        memguard.CatchInterrupt(func() {
+            fmt.Println("Interrupt signal received. Exiting...")
+        })
+        // Make sure to destroy all LockedBuffers when returning.
+        defer memguard.DestroyAll()
+
+        // Normal code continues from here.
+        foo()
     }
-    defer encryptionKey.Destroy()
 
-    // Move bytes into the buffer.
-    // Do not append or assign, use API call.
-    encryptionKey.Move([]byte("yellow submarine"))
+    func foo() {
+        // Create a 32 byte, immutable, random key.
+        key, err := memguard.NewImmutableRandom(32)
+        if err != nil {
+            // Oh no, an error. Safely exit.
+            fmt.Println(err)
+            memguard.SafeExit(1)
+        }
+        // Remember to destroy this key when the function returns.
+        defer key.Destroy()
 
-    // Use the buffer wherever you need it.
-    Encrypt(encryptionKey.Buffer(), plaintext)
+        // Do something with the key.
+        fmt.Printf("This is a %d byte key.\n", key.Size())
+        fmt.Printf("This key starts with %x\n", key.Buffer()[0])
+    }
 
 The number of LockedBuffers that you are able to create is limited by how much memory your system kernel allows each process to mlock/VirtualLock. Therefore you should call Destroy on LockedBuffers that you no longer need, or simply defer a Destroy call after creating a new LockedBuffer.
 
 If a function that you're using requires an array, you can cast the buffer to an array and then pass around a pointer. Make sure that you do not dereference the pointer and pass around the resulting value, as this will leave copies all over the place.
 
-    key, err := memguard.NewRandom(16, false)
+    key, err := memguard.NewImmutableRandom(16)
     if err != nil {
-        panic(err)
+        fmt.Println(err)
+        memguard.SafeExit(1)
     }
     defer key.Destroy()
 
@@ -45,6 +66,6 @@ When terminating your application, care should be taken to securely cleanup ever
     defer memguard.DestroyAll()
 
     // Use memguard.SafeExit() instead of os.Exit().
-    memguard.SafeExit(0) // 0 is the status code.
+    memguard.SafeExit(1) // 1 is the exit code.
 */
 package memguard

docs_test.go

@@ -2,15 +2,6 @@ package memguard
 
 import "fmt"
 
-func ExampleNew() {
-	key, err := New(32, false)
-	if err != nil {
-		fmt.Println(err)
-		SafeExit(1)
-	}
-	defer key.Destroy()
-}
-
 func ExampleCatchInterrupt() {
 	CatchInterrupt(func() {
 		fmt.Println("Exiting...")

errors.go

@@ -5,9 +5,9 @@ import "errors"
 // ErrDestroyed is returned when a function is called on a destroyed LockedBuffer.
 var ErrDestroyed = errors.New("memguard.ErrDestroyed: buffer is destroyed")
 
+// ErrImmutable is returned when a function that needs to modify a LockedBuffer
+// is given a LockedBuffer that is immutable.
+var ErrImmutable = errors.New("memguard.ErrImmutable: cannot modify immutable buffer")
+
 // ErrInvalidLength is returned when a LockedBuffer of smaller than one byte is requested.
 var ErrInvalidLength = errors.New("memguard.ErrInvalidLength: length of buffer must be greater than zero")
-
-// ErrReadOnly is returned when a function that needs to modify a LockedBuffer
-// is given a LockedBuffer that is marked as being read-only.
-var ErrReadOnly = errors.New("memguard.ErrReadOnly: buffer is marked read-only")

internals.go

@@ -10,31 +10,20 @@ import (
 )
 
 var (
-	// Grab the system page size.
+	// Ascertain and store the system memory page size.
 	pageSize = os.Getpagesize()
 
-	// Once object to ensure CatchInterrupt is only executed once.
+	// Canary value that acts as an alarm in case of disallowed memory access.
+	canary = createCanary()
+
+	// Create a dedicated sync object for the CatchInterrupt function.
 	catchInterruptOnce sync.Once
 
-	// Store pointers to all of the LockedBuffers.
+	// Array of all active containers, and associated mutex.
 	allLockedBuffers      []*container
 	allLockedBuffersMutex = &sync.Mutex{}
 )
 
-// container implements the actual data container.
-type container struct {
-	sync.Mutex // Local mutex lock.
-
-	buffer []byte // Slice that references protected memory.
-
-	readOnly  bool // Is this memory read-only?
-	destroyed bool // Is this LockedBuffer destroyed?
-}
-
-// littleBird is a value that we monitor instead of the LockedBuffer
-// itself. It allows us to tell the GC to auto-destroy LockedBuffers.
-type littleBird [16]byte
-
 // Create and allocate a canary value. Return to caller.
 func createCanary() []byte {
 	// Canary length rounded to page size.
@@ -46,18 +35,18 @@ func createCanary() []byte {
 	// Allocate it.
 	memory := memcall.Alloc(totalLen)
 
-	// Lock the pages that will hold the canary.
-	memcall.Lock(memory[pageSize : pageSize+roundedLen])
-
 	// Make the guard pages inaccessible.
 	memcall.Protect(memory[:pageSize], false, false)
 	memcall.Protect(memory[pageSize+roundedLen:], false, false)
 
+	// Lock the pages that will hold the canary.
+	memcall.Lock(memory[pageSize : pageSize+roundedLen])
+
 	// Fill the memory with cryptographically-secure random bytes (the canary value).
 	c := getBytes(uintptr(unsafe.Pointer(&memory[pageSize+roundedLen-32])), 32)
 	fillRandBytes(c)
 
-	// Mark the middle page as read-only.
+	// Tell the kernel that the canary value should be immutable.
 	memcall.Protect(memory[pageSize:pageSize+roundedLen], true, false)
 
 	// Return a slice that describes the correct portion of memory.
@@ -71,11 +60,11 @@ func roundToPageSize(length int) int {
 
 // Get a slice that describes all memory related to a LockedBuffer.
 func getAllMemory(b *container) []byte {
-	// Calculate the length of the buffer and the associated rounded value.
-	bufLen, roundedBufLen := len(b.buffer), roundToPageSize(len(b.buffer)+32)
+	// Calculate the size of the entire container's memory.
+	roundedBufLen := roundToPageSize(len(b.buffer) + 32)
 
 	// Calculate the address of the start of the memory.
-	memAddr := uintptr(unsafe.Pointer(&b.buffer[0])) - uintptr((roundedBufLen-bufLen)+pageSize)
+	memAddr := uintptr(unsafe.Pointer(&b.buffer[0])) - uintptr((roundedBufLen-len(b.buffer))+pageSize)
 
 	// Calculate the size of the entire memory.
 	memLen := (pageSize * 2) + roundedBufLen

memcall/memcall_osx.go

@@ -25,7 +25,7 @@ func Unlock(b []byte) {
 // Alloc allocates a byte slice of length n and returns it.
 func Alloc(n int) []byte {
 	// Allocate the memory.
-	b, err := unix.Mmap(-1, 0, n, unix.PROT_READ|unix.PROT_WRITE|unix.PROT_EXEC, unix.MAP_PRIVATE|unix.MAP_ANON)
+	b, err := unix.Mmap(-1, 0, n, unix.PROT_READ|unix.PROT_WRITE, unix.MAP_PRIVATE|unix.MAP_ANON)
 	if err != nil {
 		panic(fmt.Sprintf("memguard.memcall.Alloc(): could not allocate [Err: %s]", err))
 	}
@@ -46,11 +46,11 @@ func Protect(b []byte, read, write bool) {
 	// Ascertain protection value from arguments.
 	var prot int
 	if read && write {
-		prot = unix.PROT_READ | unix.PROT_WRITE | unix.PROT_EXEC
+		prot = unix.PROT_READ | unix.PROT_WRITE
 	} else if read {
-		prot = unix.PROT_READ | unix.PROT_EXEC
+		prot = unix.PROT_READ
 	} else if write {
-		prot = unix.PROT_WRITE | unix.PROT_EXEC
+		prot = unix.PROT_WRITE
 	} else {
 		prot = unix.PROT_NONE
 	}

memcall/memcall_unix.go

@@ -29,7 +29,7 @@ func Unlock(b []byte) {
 // Alloc allocates a byte slice of length n and returns it.
 func Alloc(n int) []byte {
 	// Allocate the memory.
-	b, err := unix.Mmap(-1, 0, n, unix.PROT_READ|unix.PROT_WRITE|unix.PROT_EXEC, unix.MAP_SHARED|unix.MAP_ANONYMOUS|0x00020000)
+	b, err := unix.Mmap(-1, 0, n, unix.PROT_READ|unix.PROT_WRITE, unix.MAP_SHARED|unix.MAP_ANONYMOUS|0x00020000)
 	if err != nil {
 		panic(fmt.Sprintf("memguard.memcall.Alloc(): could not allocate [Err: %s]", err))
 	}
@@ -50,11 +50,11 @@ func Protect(b []byte, read, write bool) {
 	// Ascertain protection value from arguments.
 	var prot int
 	if read && write {
-		prot = unix.PROT_READ | unix.PROT_WRITE | unix.PROT_EXEC
+		prot = unix.PROT_READ | unix.PROT_WRITE
 	} else if read {
-		prot = unix.PROT_READ | unix.PROT_EXEC
+		prot = unix.PROT_READ
 	} else if write {
-		prot = unix.PROT_WRITE | unix.PROT_EXEC
+		prot = unix.PROT_WRITE
 	} else {
 		prot = unix.PROT_NONE
 	}