Documentation overhaul. (#215)

This refactors the `docs/` directory into `docs/` and `docs-dev/` for
user documentation and developer documentation,
with the following overall motivations:

- Remove developer documentation from HTML documentation so that users do not
  land on it. Ideally this would be something like `docs/user/` and `docs/dev/`
  but GitHub does not allow using anything except `docs/` for the root.

- Change PKCS#11 library documentation to focus exclusively on the case of
  users installing it in production. This means replacing mentions of
  "the user you will run aziot-keyd as" with "aziotks", etc.

- Don't link to HTTP API docs from the index since they're only useful
  for host module authors. They're linked from the host-module doc instead.

  Also, the TPM service is exclusively an internal service, so move it to
  developer docs.

- Move `packaging.md` to developer docs since the user does not need
  the level of detail it provides about what the package contains.
  The user docs only cover the super-config and a new page about
  the `aziotctl` binary.

- Hard-code the nav bar instead of letting Jekyll generate it, so that
  it preserves the order of pages rather than sort them alphabetically.

README.md

@@ -5,9 +5,7 @@ This repository contains the code of the Azure IoT Identity Service and related
 
 The contents of the `docs/` directory are served at <https://azure.github.io/iot-identity-service/>
 
-Go to <https://azure.github.io/iot-identity-service/> for an overview of the services in this repository.
-
-Go to <https://azure.github.io/iot-identity-service/dev/> for developer documentation, and see [`CONTRIBUTING`](CONTRIBUTING.md) for contribution guidelines.
+See [`docs-dev/README.md`](docs-dev/README.md) for developer documentation, and [`CONTRIBUTING`](CONTRIBUTING.md) for contribution guidelines.
 
 
 # License

docs-dev/README.md

@@ -0,0 +1,19 @@
+# Developer documentation
+
+These docs are meant for developers working on this repository who need to build the services from source, deploy in testing environments, etc.
+
+If you are not a developer working on this repository, please go to user docs at <https://azure.github.io/iot-identity-service/> instead.
+
+- [Building the services](building.md)
+
+- [Running the services locally](running/index.md)
+
+- [Building the packages](packages.md)
+
+- [End-to-end tests](e2e-tests.md)
+
+- [Provisioning using EST, with certificates issued by EST (On-prem PKI)](est-ca.md)
+
+- [TPM Service API docs](tpm-service.md)
+
+- [OpenSSL engine internals](openssl-engine-internals.md)

docs-dev/est-ca.md

@@ -0,0 +1,91 @@
+# Provisioning using EST, with certificates issued by EST (private PKI)
+
+---
+
+![Cloud Setup](img/est-ca-cloud-setup.svg)
+
+---
+
+![Device Setup](img/est-ca-device-setup.svg)
+
+---
+
+![Provisioning](img/est-ca-provisioning-simple.svg)
+
+[(Click here for detailed version)](img/est-ca-provisioning-detailed.svg)
+
+---
+
+![Runtime operation](img/est-ca-runtime-operation-simple.svg)
+
+[(Click here for detailed version)](img/est-ca-runtime-operation-detailed.svg)
+
+---
+
+1. KS configuration
+
+    ```toml
+    [pkcs11]
+    "lib_path" = "/usr/lib/hardware-specific-pkcs11-library.so"
+    "base_slot" = "pkcs11:slot-id=0?pin-value=1234"
+
+    [preloaded_keys]
+    "bootstrap" = "pkcs11:slot-id=0;object=device%20id?pin-value=1234"
+    ```
+
+1. CS configuration
+
+    ```toml
+    "homedir_path" = "/var/lib/iotedge/cs"
+
+    [cert_issuance]
+    device-ca = "est" # Only needed if one of the others has the value "device-ca"
+    device-id = "est" # or "device-ca" for minting locally
+    module-id = "est" # or "device-ca" for minting locally
+    module-server = "est" # or "device-ca" for minting locally
+
+    [cert_issuance.est]
+    "method" = "x509"
+    "identity_cert" = "est-id"
+    "identity_pk" = "est-id"
+    "bootstrap_identity_cert" = "bootstrap" #optional
+    "bootstrap_identity_pk" = "bootstrap" #optional
+
+    [cert_issuance.est.urls]
+    # Default EST endpoint URL
+    default = "https://estendpoint.com/.well-known/est/simpleenroll"
+
+    # EST endpoint URL specifically for the est-id cert
+    "est-id" = "https://estendpoint.com/.well-known/est/est-id/simpleenroll"
+
+    # EST endpoint URL specifically for the device-id cert
+    "device-id" = "https://estendpoint.com/.well-known/est/device-id/simpleenroll"
+
+    # EST endpoint URL specifically for the device-ca cert
+    "device-ca" = "https://estendpoint.com/.well-known/est/device-ca/simpleenroll"
+
+    [preloaded_certs]
+    "bootstrap" = "/var/secrets/bootstrap.cer"
+    "est-ca" = "/var/secrets/est-ca.cer"
+    "trust-bundle" = [
+        "est-ca",
+    ]
+    ```
+
+1. IS configuration
+
+    ```toml
+    [provisioning]
+    "source" = "dps"
+    "scope_id" = "<ADD dps SCOPE ID HERE>"
+
+    [provisioning.attestation]
+    "method" = "x509"
+    "identity_cert" = "device-id"
+    "identity_pk" = "device-id"
+
+    ```
+
+The elements in the `trust-bundle` array and the `provisioning.attestation.identity_cert` value are certificate IDs defined in the CS's `preloaded_certs` table.
+
+The `provisioning.attestation.identity_pk` value is a key ID defined in the KS's `preloaded_keys` table.

docs/openssl-engine-internals.md → docs-dev/openssl-engine-internals.md

@@ -1,4 +1,4 @@
-# Openssl engine internals
+# OpenSSL engine internals
 
 
 This document talks about some basic crypto concepts, how the openssl API works, how openssl engines work, and how KS, libaziot-keys, and the two openssl engines used by it are structured.

docs/dev/packages.md → docs-dev/packages.md

@@ -66,12 +66,12 @@ For an example to put it all together, let's say you want to build the CentOS 7
 
 ```sh
 docker run -it --rm \
-	-v "$(realpath ~/src/iot-identity-service):/src" \
-	-e 'ARCH=amd64' \
-	-e 'PACKAGE_VERSION=1.2.0' \
-	-e 'PACKAGE_RELEASE=0' \
-	centos:7 \
-	'/src/ci/package.sh'
+    -v "$(realpath ~/src/iot-identity-service):/src" \
+    -e 'ARCH=amd64' \
+    -e 'PACKAGE_VERSION=1.2.0' \
+    -e 'PACKAGE_RELEASE=0' \
+    centos:7 \
+    '/src/ci/package.sh'
 ```
 
 and at the end you would have these files under `~/src/iot-identity-service/packages`:
@@ -93,12 +93,12 @@ For another example, let's say you want to build the Debian 10 package for ARM32
 
 ```sh
 docker run -it --rm \
-	-v "$(realpath ~/src/iot-identity-service):/src" \
-	-e 'ARCH=arm32v7' \
-	-e 'PACKAGE_VERSION=1.2.0' \
-	-e 'PACKAGE_RELEASE=0' \
-	debian:10-slim \
-	'/src/ci/package.sh'
+    -v "$(realpath ~/src/iot-identity-service):/src" \
+    -e 'ARCH=arm32v7' \
+    -e 'PACKAGE_VERSION=1.2.0' \
+    -e 'PACKAGE_RELEASE=0' \
+    debian:10-slim \
+    '/src/ci/package.sh'
 ```
 
 and at the end you would have these files under `~/src/iot-identity-service/packages`:

docs/packaging.md → docs-dev/packaging.md

@@ -1,11 +1,4 @@
-# Packaging
-
-The Identity Service (IS), Keys Service (KS), Certificates Service (CS) and TPM Service (TPMS) are stand-alone components that can be used on Linux-based Azure IoT devices to ease the process of provisioning the device with Azure, to allow modules on the device to securely and cooperatively connect to IoT Hub using module identities, and to provide common cryptographic services to modules like HSM-backed keys and CA-issued certificates.
-
-On Azure IoT Edge devices, these services form the backbone of the Azure IoT Edge service - the Azure IoT Edge service relies on the Identity Service to provision the Edge device identity and Edge module identities in Azure, to provide the CA certificate used for module server certificates like Edge Hub's, and so on. For more in-depth discussion of the IoT Edge package, refer to its documentation on ["Packaging".](https://github.com/Azure/iotedge/blob/master/doc/packaging.md)
-
-These components ship in the `aziot-identity-service` package. This document describes the contents of this package and how to install and configure it.
-
+# Package contents
 
 ## Binaries and libraries
 
@@ -33,7 +26,7 @@ differently-named symlinks to this `aziotd` binary. This saves space since a lot
 
 `/usr/lib/libaziot_keys.so` is a library used by the KS for pluggable cryptographic key storage. It exposes a simple C API and is designed to be replaceable by the user if they need to. However the implementation shipped by default is already quite versatile, since it supports both filesystem keys as well as all HSMs with PKCS#11 libraries, so we don't expect users to need to replace it.
 
-`/usr/lib/engines-1.1/aziot_keys.so` is an OpenSSL engine that can be used for mTLS when the client certificate's private key is backed by the KS. See ["Openssl engine internals"](openssl-engine-internals.md) for details.
+`/usr/lib/engines-1.1/aziot_keys.so` is an OpenSSL engine that can be used for mTLS when the client certificate's private key is backed by the KS. See ["OpenSSL engine internals"](openssl-engine-internals.md) for details.
 
 
 ## Config files
@@ -64,9 +57,7 @@ All files found in this way are unioned. The `config.toml` is considered first,
 
 Every service has a corresponding `config.toml.default` file that indicates the default settings for that service if no config file is provided.
 
-However, we expect the user to primarily configure the services via the "super-config" file. A template of this file is installed at `/etc/aziot/config.toml.template`. The easiest way for the user to configure all services is to copy this template to a new file, say `/etc/aziot/config.toml`, edit it, then "apply" the contents of the file to all the services with `sudo aziotctl config apply`.
-
-This is explained in more detail in the ["Installing and configuring the package"](#installing-and-configuring-the-package) section below.
+However, we expect the user to primarily configure the services via the "super-config" file.
 
 
 ## Service endpoint socket files
@@ -132,31 +123,3 @@ For example, if you have a module process, the process needs to interact with th
 ## Package dependencies
 
 The binaries and libraries shipped in this package only depend on OpenSSL (v1.0 on CentOS 7, v1.1 on other supported distributions).
-
-If you plan to use a PKCS#11 library with KS to store your cryptographic keys in an HSM or TPM, you will need to install this library and its dependencies in accordance with your HSM manufacturer.
-
-
-# Installing and configuring the package
-
-As mentioned above, the `aziot-identity-service` package ships with the template of a "super-config", at `/etc/aziot/config.toml/template`. The easiest way for the user to configure all services is to copy this template to a new file, say `/etc/aziot/config.toml`, edit it, then "apply" the contents of the file to all the services with `sudo aziotctl config apply`.
-
-This file is called the "super-config" because it is an amalgamation of all the configurations of the individual services, with some simplifications. Since most use cases would require the user to keep all the service configs in sync with each other, this approach of having a single "super-config" helps make it easier and reduce errors.
-
-For example, if the user wanted to configure their device to use DPS provisioning with X.509 attestation, they would have to configure the certificate's private key as a preloaded key file URI in the KS config, the certificate as a preloaded cert file URI in the CS config, and the provisioning method in IS to use those private key and certificate via their IDs. With the super-config, they are instead able to configure provisioning to use those key and cert file URIs directly. `aziotctl config apply` does the job of converting the "super-config" into the individual services' configs, which includes synthesizing IDs for the key and cert and hooking them all up correctly.
-
-Therefore, the easiest way to install and configure the package is:
-
-```sh
-# Install the package
-sudo apt install aziot-identity-service
-
-# Copy the config template into a new empty config.
-sudo cp /etc/aziot/config.toml.template /etc/aziot/config.toml
-
-# Edit the new empty config and fill in your provisioning information,
-# plus anything else you want to customize. See comments in the file for details.
-sudo $EDITOR /etc/aziot/config.toml
-
-# Apply the new config to all the services and restart them as necessary.
-sudo aziotctl config apply
-```

docs-dev/running/pkcs11.md

@@ -0,0 +1,9 @@
+# Setting up your PKCS#11 library
+
+Follow the user steps at <https://azure.github.io/iot-identity-service/pkcs11/>
+
+If you want to use tpm2-pkcs11 with the TPM simulator, see [`ibmswtpm2`](ibmswtpm2.md)
+
+To verify your HSM and its PKCS#11 library with aziot-keyd, see [Testing the openssl engine.](aziot-keyd.md#testing-the-openssl-engine)
+
+If the script mentioned there completes successfully, then the hardware and PKCS#11 library ought to be suitable for `aziot-keyd` to use. If there are errors such as crashes or signature verification failures, then it might be a problem with the hardware configuration, a bug in the PKCS#11 library, or a bug in the PKCS#11-related code in this repository.

docs/_includes/nav.html

@@ -1,6 +1,3 @@
-{% assign items = site.pages | sort: 'url' %}
-{% assign urlbase = site.url | append: "/iot-identity-service" %}
-
 <style>
     #nav-sidebar {
         margin: 0;
@@ -16,38 +13,28 @@
 <div id="nav-sidebar">
     <h3>Contents</h3>
     <ul>
-        {% assign level = 1 %}
-        {% for i in items %}
-            {% unless i.title %}
-                {% continue %}
-            {% endunless %}
+        <li><a href="/iot-identity-service/">Overview</a></li>
+
+        <li><a href="/iot-identity-service/installation.html">Installing the Azure IoT Identity Service</a></li>
+
+        <li><a href="/iot-identity-service/configuration.html">Configuring the Azure IoT Identity Service</a></li>
 
-            {% assign path = i.url | split: "/" %}
-            {% assign path_level = path.size | minus: 1 | at_least: 1 %}
+        <li><a href="/iot-identity-service/aziotctl.html">Managing the Azure Iot Identity Service</a></li>
 
-            {% if path_level > level %}
-                {% assign limit = path_level | minus: 1 %}
-                {% for l in (level..limit) %}
-                    <ul style="margin-bottom:0;">
-                {% endfor %}
-                {% assign level = path_level %}
-            {% elsif path_level < level %}
-                {% assign limit = level | minus: 1 %}
-                {% for l in (path_level..limit) %}
-                    </ul>
-                {% endfor %}
-                {% assign level = path_level %}
-            {% endif %}
+        <li><a href="/iot-identity-service/pkcs11/">Protecting keys via an HSM or TPM</a></li>
+        <ul style="margin-bottom: 0">
+            <li><a href="/iot-identity-service/pkcs11/tpm2-pkcs11.html">tpm2-pkcs11</a></li>
+            <li><a href="/iot-identity-service/pkcs11/cryptoauthlib.html">cryptoauthlib</a></li>
+            <li><a href="/iot-identity-service/pkcs11/softhsm.html">softhsm</a></li>
+        </ul>
 
-            <li>
-            {% if i.url == page.url %}
-                <a href="{{ urlbase }}{{ i.url }}" style="font-weight: bold;">
-            {% else %}
-                <a href="{{ urlbase }}{{ i.url }}">
-            {% endif %}
+        <li><a href="/iot-identity-service/host-module.html">Writing a host module</a></li>
 
-            {{ i.title }}
-            </a></li>
-        {% endfor %}
+        <li><a href="/iot-identity-service/api/">HTTP API of the Services</a></li>
+        <ul style="margin-bottom: 0">
+            <li><a href="/iot-identity-service/api/identity-service.html">Identity Service</a></li>
+            <li><a href="/iot-identity-service/api/keys-service.html">Keys Service</a></li>
+            <li><a href="/iot-identity-service/api/certificates-service.html">Certificates Service</a></li>
+        </ul>
     </ul>
 </div>

docs/identity-service.md → docs/api/identity-service.md

@@ -14,7 +14,7 @@ The returned `auth.keyHandle` value is meant to be used with the [Keys Service](
 
 - If `auth.type` is `"x509"`, `keyHandle` is a handle to the identity X.509 certificate's private key. The `auth.certId` contains the certificate ID of the identity X.509 certificate, which can be retrieved used using the [Certificates Service.](certificates-service.md) Together this private key and certificate should be used for IoT Hub authentication via client certificate.
 
-  For convenience, the Identity Service package ships with an openssl engine that can load the handles of private keys. If the caller of this API uses openssl for TLS, it can use this engine. More details can be found on the [Openssl engine internals](openssl-engine-internals.md) page.
+  For convenience, the Identity Service package ships with an openssl engine that can load the handles of private keys. If the caller of this API uses openssl for TLS, it can use this engine. More details can be found on the [OpenSSL engine internals](openssl-engine-internals.md) page.
 
 #### Response for principals associated to device identities (SAS case)
 
@@ -469,7 +469,7 @@ IoT Hub module identities will get generated by Identity service (IS) for contai
 
 2. OS configuration
 
-    The host process package (using elevated admin privileges) needs to add OS userid, used by the host process to call IS APIs, in the `aziotid` group. See [Packaging](packaging.md) for more details on the `aziot-identity-service` package.
+    The host process package (using elevated admin privileges) needs to add OS userid, used by the host process to call IS APIs, in the `aziotid` group.
 
 
 ### Host process package configuration responsibilities

docs/api/index.md

@@ -0,0 +1,9 @@
+# HTTP API of the services
+
+These pages document the HTTP API of the services, for use by host module authors.
+
+- [Identity Service](identity-service.md)
+
+- [Keys Service](keys-service.md)
+
+- [Certificates Service](certificates-service.md)

docs/aziotctl.md

@@ -0,0 +1,32 @@
+# Managing the Azure Iot Identity Service
+
+The `aziotctl` CLI tool is used to manage and interact with the services.
+
+See `aziotctl --help` for the full help. A few notable commands are listed below.
+
+Note that most `aziotctl` commands need to be run as root.
+
+
+## Applying changes in the super-config to the services
+
+```sh
+sudo aziotctl config apply
+```
+
+By default, this command expects the super-config to be at `/etc/aziot/config.toml`. To use a different path, specify the `-c / --config` parameter.
+
+
+## Stop all services
+
+```sh
+sudo aziotctl system stop
+```
+
+
+## Restart all services
+
+```sh
+sudo aziotctl system restart
+```
+
+Note: Since the services use systemd socket activation, this command starts all the socket units but does not start any service except the Identity Service (`aziot-identityd.service`). The other services (`aziot-keyd.service`, `aziot-certd.service`, etc) will automatically start if they need to.

docs/configuration.md

@@ -0,0 +1,19 @@
+# Configuring the Azure IoT Identity Service
+
+The Azure IoT Identity Service package installs four different services - the Identity Service, Keys Service, Certificates Service, and TPM Service. All of these services can be configured from a single configuration file, called the "super-config".
+
+A template of this file is installed at `/etc/aziot/config.toml.template`. Copy it to a new file `/etc/aziot/config.toml`, and edit the file with the details of your device, IoT Hub, etc, based on the comments in the file. Then, run the command `sudo aziotctl config apply` to apply the configuration to the services.
+
+```sh
+# Copy the template to a new file
+sudo cp /etc/aziot/config.toml.template /etc/aziot/config.toml
+
+# Edit the new empty config and fill in your provisioning information,
+# plus anything else you want to customize. See the comments in the file for details.
+sudo $EDITOR /etc/aziot/config.toml
+
+# Apply the configuration to the services
+sudo aziotctl config apply
+```
+
+If you need to make any changes to the `/etc/aziot/config.toml` file, do so, then re-run `sudo aziotctl config apply` to re-apply those changes to the services.