Add "keyring remove" command

Adds a CLI and RESTful API operation for "keyring remove", with simple
test cases.  Added the corresponding Java API operation.  Updated the
API documentation.

API change: for consistency with RESTful API design, the GET
/restful/keyring/add operation now returns "201 Created" not "200 OK" if
successful.

doc/REST-API-Keyring.md

@@ -1,6 +1,6 @@
 Keyring REST API
 ================
-[Serval Project][], February 2016
+[Serval Project][], November 2016
 
 Introduction
 ------------
@@ -10,24 +10,55 @@ easily be created by any node at any time.  Each [Serval DNA][] daemon that
 runs on a node in the network stores its own identities in the [Keyring][], an
 encrypted store protected by passwords, and gives applications access to the
 Keyring via the **Keyring REST API** described in this document.  Using this
-API, client applications can query, unlock, lock, create, and modify identities
-in the keyring.
+API, client applications can add, remove, unlock, lock, query, and modify
+identities in the keyring.
 
-### Basic concepts
+Basic concepts
+--------------
 
-#### Serval ID
+### Serval ID
 
 Every identity in the [Serval mesh network][] is represented by its **Serval
 ID**, (usually abbreviated to [SID][], and formerly known as “Subscriber ID”),
-which is a unique 256-bit public key in the [Curve25519][] key space that is
-generated from the random *Serval ID secret* when the identity is created.  The
-SID is used:
+which is a unique 256-bit public key in the [Curve25519][] *crypto-box* key
+space that is generated from the random *Serval ID secret* when the identity is
+created.  The SID is used:
 
 *  as the network address in the [Serval Mesh network][]
+*  to encrypt [MDP][] messages
 *  to identify the senders, recipients and authors of [Rhizome bundles][]
 *  to identify the parties in a [MeshMS conversation][]
 
-#### Rhizome Secret
+### Serval Signing ID
+
+Every identity in the [Serval mesh network][] has a **Serval Signing ID**,
+which is a unique 256-bit public key in the [Curve25519][] *crypto-sign* key
+space that is generated at the same time as the [Serval ID](#serval-id) when
+the identity is created.  The Signing ID is used:
+
+*  to prevent forgery of [Serval Mesh network][] routing messages
+*  to authenticate non-encrypted [MDP][] messages
+
+### DID
+
+The **DID** ([Dialled Identity][]) is a telephone number, represented as a
+string of five or more digits from the set `123456789#0*`.  It is used by the
+[DNA][] protocol to allow [Serval mesh network][] users to discover each other
+by telephone number; the first step in establishing a mesh voice call.
+
+### Name
+
+The **Name** is a short, non-blank, non-empty, unstructured string assigned by
+a human user to an identity.  It is used to represent the identity to human
+users, as it is more recognisable than a hexadecimal [SID](#serval-id) or a
+[DID](#did) (telephone number).
+
+The name is encoded using [UTF-8][].  Since it is intended for human
+consumption, it may be constrained to contain only printable characters and no
+carriage-motion characters (eg, TAB U+0009 or LF U+0010), and to not start or
+end with white space.
+
+### Rhizome Secret
 
 The *Rhizome Secret* is a secret key, separate from the [SID](#serval-id)
 secret, that is generated randomly for each new identity, and stored in the
@@ -36,7 +67,7 @@ the [Bundle Secret][] of a bundle into its [manifest][], in the form of the
 [Bundle Key][], thus relieving [Rhizome][] applications of the burden of having
 to store and protect Bundle Secrets themselves.
 
-#### PIN
+### PIN
 
 When an identity is created, it can optionally be given a PIN (passphrase).  If
 the PIN is *empty* then the identity is permanently unlocked (visible).
@@ -50,13 +81,44 @@ identities.
 If a PIN is lost and forgotten, then the identity (identities) it unlocks will
 remain locked and unusable forever.  There is no “master PIN” or back-door.
 
-#### Identity unlocking
+### Identity unlocking
 
-All Keyring API requests can supply a passphrase using the optional **pin**
+All Keyring requests can supply a passphrase using the optional **pin**
 parameter, which unlocks all keyring identities protected by that password,
-prior to performing the request.  Serval DNA caches every password it receives
-until the password is revoked using the *lock* request, so once an identity is
-unlocked, it remains visible until explicitly locked.
+prior to performing the request.  Serval DNA caches every PIN it receives until
+the PIN is revoked using the [lock request](#get-restful-keyring-lock), so once
+an identity is unlocked, it remains visible until explicitly locked.
+
+Keyring REST API common features
+--------------------------------
+
+### Keyring JSON result
+
+All Keyring requests relating to a single identity that do not produce a
+special response content for the outcome, return the following augmented [JSON
+result][] object as the HTTP response content:
+
+    {
+        "http_status_code": ...,
+        "http_status_message": "...",
+        "identity": {
+            "sid": "<hex64>",
+            "identity": "<hex64>",
+            "did": "...",
+            "name": "..."
+        }
+    }
+
+*  the `sid` field is the [SID](#serval-id); a string containing 64 uppercase
+   hexadecimal digits
+*  the `identity` field is the [Signing Id](#serval-signing-id); a string
+   containing 64 uppercase hexadecimal digits
+*  the `did` field is the string [DID](#did); omitted if the identity has no DID
+*  the `name` field is the string [Name](#name); omitted if the identity has no
+   name
+
+Keyring REST API operations
+---------------------------
 
 ### GET /restful/keyring/identities.json
 
@@ -65,10 +127,12 @@ The table columns are:
 
 *   **sid**: the [SID](#serval-id) of the identity, a string of 64 uppercase
     hex digits
-*   **did**: the optional [DID][] (telephone number) of the identity, either
-    *null* or a string of five or more digits from the set `123456789#0*`
-*   **name**: the optional name of the identity, either *null* or a non-empty
-    string of [UTF-8] characters
+*   **identity**: the [Signing ID](#serval-signing-id) of the identity, a
+    string of 64 uppercase hex digits
+*   **did**: the optional [DID](#did) (telephone number) of the identity;
+    `null` if none is assigned
+*   **name**: the optional string [Name](#name) of the identity; `null` if none
+    is assigned
 
 ### GET /restful/keyring/add
 
@@ -77,9 +141,20 @@ parameter is supplied, then the new identity will be protected by that
 password, and the password will be cached by Serval DNA so that the new
 identity is unlocked.
 
+Returns [201 Created][201] if an identity is created; the [JSON
+result](#keyring-json-result) describes the identity that was created.
+
+### GET /restful/keyring/SID/remove
+
+Removes an existing identity with a given [SID](#serval-id).
+
+If there is no unlocked identity with the given SID, this request returns [404
+Not Found][404].  Otherwise it returns [200 OK][200] and the [JSON
+result](#keyring-json-result) describes the identity that was removed.
+
 ### GET /restful/keyring/SID/set
 
-Sets the [DID][] and/or name of the unlocked identity that has the given
+Sets the [DID](#did) and/or name of the unlocked identity that has the given
 [SID](#serval-id).  The following parameters are recognised:
 
 *   **did**: sets the DID (phone number); must be a string of five or more
@@ -89,9 +164,18 @@ Sets the [DID][] and/or name of the unlocked identity that has the given
 If there is no unlocked identity with the given SID, this request returns [404
 Not Found][404].
 
+### GET /restful/keyring/SID/lock
+
+Locks an existing identity with a given [SID](#serval-id).
+
+If there is no unlocked identity with the given SID, this request returns [404
+Not Found][404].  Otherwise it returns [200 OK][200] and the [JSON
+result](#keyring-json-result) describes the identity that was locked.
+
 
 -----
 **Copyright 2015 Serval Project Inc.**  
+**Copyright 2016 Flinders University**  
 ![CC-BY-4.0](./cc-by-4.0.png)
 Available under the [Creative Commons Attribution 4.0 International licence][CC BY 4.0].
 
@@ -105,14 +189,18 @@ Available under the [Creative Commons Attribution 4.0 International licence][CC
 [cryptographic identities]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:security_framework
 [Curve25519]: https://en.wikipedia.org/wiki/Curve25519
 [SID]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:sid
-[DID]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:did
+[Dialled Identity]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:did
+[DNA]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:dna
+[MDP]: ./Mesh-Datagram-Protocol.md
 [Rhizome]: ./REST-API-Rhizome.md
 [Rhizome bundles]: ./REST-API-Rhizome.md#bundle
 [manifest]: ./REST-API-Rhizome.md#manifest
 [Bundle Secret]: ./REST-API-Rhizome.md#bundle-secret
 [Bundle Key]: ./REST-API-Rhizome.md#bundle-key
 [MeshMS conversation]: ./REST-API-MeshMS.md#conversation
+[JSON result]: ./REST-API.md#json-result
 [JSON table]: ./REST-API.md#json-table
+[UTF-8]: https://en.wikipedia.org/wiki/UTF-8
 [200]: ./REST-API.md#200-ok
 [201]: ./REST-API.md#201-created
 [202]: ./REST-API.md#202-accepted

doc/REST-API-Rhizome.md

@@ -399,15 +399,15 @@ remove them.  They reveal internal details of the storage of the bundle:
 
 All Rhizome requests to fetch or insert a single bundle that do not produce a
 special response content for the outcome, return the following augmented [JSON
-result](#json-result) object as the HTTP response content:
+result][] object as the HTTP response content:
 
     {
-     "http_status_code": ...,
-     "http_status_message": "...",
-     "rhizome_bundle_status_code": ...,
-     "rhizome_bundle_status_message": "...",
-     "rhizome_payload_status_code": ...,
-     "rhizome_payload_status_message": "..."
+        "http_status_code": ...,
+        "http_status_message": "...",
+        "rhizome_bundle_status_code": ...,
+        "rhizome_bundle_status_message": "...",
+        "rhizome_payload_status_code": ...,
+        "rhizome_payload_status_message": "..."
     }
 
 *  the `rhizome_bundle_status_code` field is the integer [bundle status code](#bundle-status-code)
@@ -946,6 +946,7 @@ Available under the [Creative Commons Attribution 4.0 International licence][CC
 [Serval Mesh network]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:mesh_network
 [Serval DNA]: ../README.md
 [REST-API]: ./REST-API.md
+[JSON result]: ./REST-API.md#json-result
 [store and forward]: https://en.wikipedia.org/wiki/Store_and_forward
 [SID]: ./REST-API-Keyring.md#serval-id
 [Keyring]: ./REST-API-Keyring.md

doc/REST-API.md

@@ -442,7 +442,7 @@ for example, some *404* responses from Rhizome have phrases like, “Bundle not
 found”, “Payload not found”, etc.
 
 Some responses augment the *JSON result* object with extra fields; for example,
-[Rhizome JSON result](#rhizome-json-result).
+[Rhizome JSON result][] and [Keyring JSON result][].
 
 ### JSON table
 
@@ -521,6 +521,8 @@ Available under the [Creative Commons Attribution 4.0 International licence][CC
 [Internet Media Type]: https://www.iana.org/assignments/media-types/media-types.xhtml
 [Rhizome bundle]: ./REST-API-Rhizome.md#bundle
 [Rhizome manifest]: ./REST-API-Rhizome.md#manifest
+[Rhizome JSON result]: ./REST-API-Rhizome.md#rhizome-json-result
+[Keyring JSON result]: ./REST-API-Keyring.md#keyring-json-result
 [bundle secret]: ./REST-API-Rhizome.md#bundle-secret
 [text+binarysig format]: ./REST-API-Rhizome.md#textbinarysig-manifest-format
 [JSON]: https://en.wikipedia.org/wiki/JSON

java/org/servalproject/servaldna/ServalDClient.java

@@ -91,6 +91,11 @@ public KeyringIdentity keyringAdd(String did, String name, String pin) throws Se
 		return KeyringCommon.addIdentity(this, did, name, pin);
 	}
 
+	public KeyringIdentity keyringRemove(SubscriberId sid, String pin) throws ServalDInterfaceException, IOException
+	{
+		return KeyringCommon.removeIdentity(this, sid, pin);
+	}
+
 	public RhizomeBundleList rhizomeListBundles() throws ServalDInterfaceException, IOException
 	{
 		RhizomeBundleList list = new RhizomeBundleList(this);

java/org/servalproject/servaldna/keyring/KeyringCommon.java

@@ -238,13 +238,34 @@ public static KeyringIdentity addIdentity(ServalDHttpConnectionFactory connector
 			query_params.add(new ServalDHttpConnectionFactory.QueryParam("pin", pin));
 		HttpURLConnection conn = connector.newServalDHttpConnection("/restful/keyring/add", query_params);
 		conn.connect();
-		Status status = receiveRestfulResponse(conn, HttpURLConnection.HTTP_OK);
+		Status status = receiveRestfulResponse(conn, HttpURLConnection.HTTP_CREATED);
 		try {
 			decodeRestfulStatus(status);
 			dumpStatus(status, System.err);
 			if (status.identity == null)
 				throw new ServalDInterfaceException("invalid JSON response; missing identity");
+			return status.identity;
+		}
+		finally {
+			if (status.input_stream != null)
+				status.input_stream.close();
+		}
+	}
 
+	public static KeyringIdentity removeIdentity(ServalDHttpConnectionFactory connector, SubscriberId sid, String pin)
+		throws IOException, ServalDInterfaceException
+	{
+		Vector<ServalDHttpConnectionFactory.QueryParam> query_params = new Vector<ServalDHttpConnectionFactory.QueryParam>();
+		if (pin != null)
+			query_params.add(new ServalDHttpConnectionFactory.QueryParam("pin", pin));
+		HttpURLConnection conn = connector.newServalDHttpConnection("/restful/keyring/" + sid.toHex() + "/remove", query_params);
+		conn.connect();
+		Status status = receiveRestfulResponse(conn, HttpURLConnection.HTTP_OK);
+		try {
+			decodeRestfulStatus(status);
+			dumpStatus(status, System.err);
+			if (status.identity == null)
+				throw new ServalDInterfaceException("invalid JSON response; missing identity");
 			return status.identity;
 		}
 		finally {

javatest/org/servalproject/test/Keyring.java

@@ -81,6 +81,17 @@ static void add(String did, String name, String pin) throws ServalDInterfaceExce
 		System.exit(0);
 	}
 
+	static void remove(SubscriberId sid, String pin) throws ServalDInterfaceException, IOException, InterruptedException
+	{
+		ServalDClient client = new ServerControl().getRestfulClient();
+		KeyringIdentity id = client.keyringRemove(sid, pin);
+		System.out.println("sid=" + id.sid +
+						   ", did=" + id.did +
+						   ", name=" + id.name
+			);
+		System.exit(0);
+	}
+
 	public static void main(String... args)
 	{
 		if (args.length < 1)
@@ -93,6 +104,8 @@ else if (methodName.equals("set"))
 				set(new SubscriberId(args[1]), args[2], args[3], args.length >= 5 ? args[4] : null);
 			else if (methodName.equals("add"))
 				add(args[1], args[2], args.length >= 4 ? args[3] : null);
+			else if (methodName.equals("remove"))
+				remove(new SubscriberId(args[1]), args.length >= 3 ? args[2] : null);
 		} catch (Exception e) {
 			e.printStackTrace();
 			System.exit(1);

keyring.c

@@ -368,7 +368,6 @@ void keyring_free_identity(keyring_identity *id)
     link_stop_routing(id->subscriber);
   bzero(id,sizeof(keyring_identity));
   free(id);
-  return;
 }
 
 /*
@@ -1423,7 +1422,7 @@ keyring_identity *keyring_create_identity(keyring_file *k, const char *pin)
 {
   DEBUGF(keyring, "k=%p", k);
   /* Check obvious abort conditions early */
-  if (!k->bam) { WHY("keyring lacks BAM (not to be confused with KAPOW)"); return NULL; }
+  if (!k->bam) { WHY("keyring lacks BAM"); return NULL; }
 
   if (!pin) pin="";
 
@@ -1479,6 +1478,35 @@ static int write_random_slot(keyring_file *k, unsigned slot)
   return 0;
 }
 
+/* Remove the given identity from the keyring by overwriting it's slot in the keyring file with
+ * random data, and unlinking it from the in-memory cache list.  Does NOT call
+ * keyring_free_identity(id), so the identity's contents remain intact; the caller must free the
+ * identity if desired.
+ */
+void keyring_destroy_identity(keyring_file *k, keyring_identity *id)
+{
+  DEBUGF(keyring, "k=%p, id=%p", k, id);
+  if (!k->bam) {
+    WHY("keyring lacks BAM");
+    return;
+  }
+  assert(id->slot != 0);
+  DEBUGF(keyring, "Destroy identity in slot %u", id->slot);
+
+  // Mark the slot as unused in the BAM.
+  set_slot(k, id->slot, 0);
+
+  // Fill the slot in the file with random bytes.
+  write_random_slot(k, id->slot);
+
+  // Unlink the identity from the in-memory cache.
+  keyring_identity **i = &k->identities;
+  while (*i && *i != id)
+    i = &(*i)->next;
+  if (*i == id)
+    *i = id->next;
+}
+
 int keyring_commit(keyring_file *k)
 {
   DEBUGF(keyring, "k=%p", k);

keyring.h

@@ -132,6 +132,7 @@ int keyring_commit(keyring_file *k);
 keyring_identity *keyring_inmemory_identity();
 void keyring_free_identity(keyring_identity *id);
 keyring_identity *keyring_create_identity(keyring_file *k, const char *pin);
+void keyring_destroy_identity(keyring_file *k, keyring_identity *id);
 void keyring_identity_extract(const keyring_identity *id, const char **didp, const char **namep);
 int keyring_load_from_dump(keyring_file *k, unsigned entry_pinc, const char **entry_pinv, FILE *input);
 int keyring_dump(keyring_file *k, XPRINTF xpf, int include_secret);