Documentation rework & mu.semte.ch import

README.md

@@ -8,27 +8,29 @@ If you do not know where to get started, check out any/all of the following
 ## Reference
 For technical information in semantic.works, you can see the following references:
 - [Naming conventions](docs/references/naming-conventions.md)
-- [Documentation](docs/references/documentation.md)
-- [Represting logged in users](docs/references/representing-logged-in-users.md)
+- [Representing logged in users](docs/references/representing-logged-in-users.md)
 
 ## Discussions
 If you want more information behind the design of semantic.works, you can read the following discussions:
 - **Why semantic...**
     - [... technology?](docs/discussions/why-semantic-tech.md)
     - [... microservices?](docs/discussions/why-semantic-microservices.md)
 - [mu.semte.ch primer](docs/discussions/mu-semtech-primer.md)
+- [semantic.works' documentation structure](docs/discussions/documentation-structure.md)
 - **Experimentation**
     - [All or nothing fails to innovate](docs/discussions/experimentation.md#all-or-nothing-fails-to-innovate)
     - [mu.semte.ch as the ideal playground](docs/discussions/experimentation.md#musemtech-as-the-ideal-playground)
-- [Reactive programming](docs/discussions/reactive-programming.md)
+- **The benefits of microservices through...**
+    - [Reactive programming](docs/discussions/reactive-programming.md)
+    - [Smaller & readable code](docs/discussions/smaller-readable-code.md)
 - [Sharing authorization](docs/discussions/sharing-authorization.md)
-- [Smaller & readable code](docs/discussions/smaller-readable-code.md)
 - [Docker multi-stage builds benefits](docs/discussions/docker-multi-stage-builds.md)
 - [Supporting MacOS](docs/discussions/supporting-mac-os.md)
 
 ## How-to
-- [Develop with your local IDE and tools inside a Docker container](docs/how-to/developing-inside-containers.md)
-
+- [Documentation quickstart](docs/how-tos/documentation-quickstart.md)
+- [Develop with your local IDE and tools inside a Docker container](docs/how-tos/developing-inside-containers.md)
+- [Troubleshooting - Slow starting containers using 100% CPU](docs/how-tos/troubleshooting---slow-starting-containers.md)
 
 ## Writeups
 Our retrospectives on...

docs/discussions/documentation-structure.md

@@ -0,0 +1,73 @@
+# Semantic.works Documentation
+
+Documentation is hard, but it is instrumental to helping your code be useful to people that aren't you, and that's a worthwhile pursuit. Even if you're making things just for yourself, the aforementioned "people that aren't you" descriptor also includes you in a few years, months or weeks. To have our documentation be as useful and in-tune as possible with the semantic technologies we craete, the following principles were outlined.
+
+## Design principles
+1. [*Don't Repeat Yourself.*](#1-dont-repeat-yourself)
+2. [*All-encompassing.*](#2-all-encompassing)
+3. [*Ease of use: writing.*](#3-ease-of-use-writing)
+4. [*Ease of use: reading.*](#4-ease-of-use-reading)
+5. [*Micro-first.*](#5-micro-first)
+6. [*Non-proprietary.*](#6-non-proprietary)
+
+
+### 1. Don't Repeat Yourself
+If we're copying documentation/text across places, we're doing something wrong:
+- This may cause inconsistencies and out of date versions, breaching [(4) ease of use: readers](#4-ease-of-use-readers), as well as [(2) all-encompassing](#2-all-encompassing)
+- This requires extra upkeep/work, breaching [(3) ease of use: writers](#3-ease-of-use-writers)
+
+### 2. All-encompassing
+Many documentations are incredibly lacking, as you've probably experienced over your time as a developer. The divio documentation standard, which you can view on [documentation.divio.com](https://documentation.divio.com/), attempts to clearly outline *what* should be documented, and *where* you can organise it. While I recommend viewing the talk and/or text on their website, a [quick start is included in this repository's how-tos](../how-tos/documentation-quickstart.md).
+
+### 3. Ease of use: writing
+Minimising maintenance for the people who make the code is essential. Lots of people do not have the time, resources, training, or quite simply the desire to write down docs. This is one of the reasons we stay away from external documentation sites like gitbook: having them sign in to a whole different platform to write the documentation for their code would be way too much friction. This would also make us reliant on the external documentation provider, breaching [(6) Non-proprietary](#6-non-proprietary), and would require navigating to an external location for readers and writers alike, thus additionally breaching [(4) Ease of use: reading](#4-ease-of-use-reading).
+
+### 4. Ease of use: reading
+- We have a bunch of projects going on, written in Ruby, Lisp, Elixer, JavaScript, Python... The documentations should nevertheless resemble eachother and be consistent. This is additionally made possible thanks to the the divio documentation standard, as seen in [(2) all-encompassing](#2-all-encompassing).
+- End-users should not have to jump through hoops to read the documentation. If you have a local clone of a repository, you should not be required to have an active internet connection to read its documentation.
+
+### 5. Micro-first
+Semantic.works is about microservices, so that means a bunch of documentation will be **small**. This means it is important to adhere to the structures of big projects involuntarily. An entire documentation folder for something that can be explained in one README is a no-go. Start from a single (README) file, and expand only if necessary.
+
+### 6. Non-proprietary
+Our repositores and projects are made to work anywhere, so this should be the case for our documentation as well. Websites, services and hosts shutting down\* or changing their licenses\*\*  is not unheard of. No editor lock-in, no host lock-in.
+
+- \*[Freshcode/freshmeat](https://jeffcovey.net/2014/06/19/freshmeat-net-1997-2014/), [Microsoft CodePlex](https://devblogs.microsoft.com/bharry/shutting-down-codeplex/), [Google Code](https://code.google.com/archive/)
+- \*\*[Terraform/Hashicorp products](https://blog.gruntwork.io/the-future-of-terraform-must-be-open-ab0b9ba65bca), [Red Hat Enterprise Linux](https://thenewstack.io/how-red-hats-license-change-is-reinvigorating-enterprise-linux-distros/)
+
+
+## Implementation
+To achieve all of the above goals, we are putting the following structure(s) in place:
+
+### Write the documentation
+All documentation ends up in one of the following places
+- If it's about the semantic works stack in its entirety, it should go in the `project` repository (see: this document)
+- If it's something useful for people getting started with the semantic works stack as a whole, it should go in the `mu-project` repository. 
+- If it's about a specific microservice or project, it should go inside of that projects repository
+
+The documentation should ideally be written as presented in [project/how-tos/documentation-quickstart](../how-tos/documentation-quickstart.md). This also covers how to tag new revisions.
+
+For microservice developers and most people working on semantic.works, this is all you have to do! This will ensure that most design principles are met. However, to have a more readily available & traditionally structured documentation, the following steps get executed.
+
+### Semantic.works
+[View the fork on GitHub here](https://github.com/Denperidge-Redpencil/semantic.works)
+
+Semantic.works was (re-)developed to contain all necessary links & information about our repositories. This includes links & documentations to current & previous revisions. However, we ensured that *none* of this would have to be hardcoded into the site, thanks to...
+
+### App-mu-info
+[View the fork on GitHub here](https://github.com/Denperidge-Redpencil/app-mu-info-rework)
+
+Which is a service that holds a linked-data database and an endpoint about repositories and their revisions. This includes links to the repos and images, as well as the contents of the documentation, both parsed and unparsed. And this service fetches its data dynamically thanks to...
+
+### Repo-harvester
+[View the repo on GitHub here](https://github.com/mu-semtech/repo-harvester)
+
+Which is a microservice that - after some basic configuration - will collect & clone all repositories and images from specified accounts & hosts. It collects all the information app-mu-info's data model needs, including all documentation aggregated & parsed into the divio sections. The documentation stuff is thanks to...
+
+### Divio-docs-parser
+[View the repo on GitHub here](https://github.com/Denperidge-Redpencil/divio-docs-parser)
+
+Which is a Python package that, when given a directory, will handle everything in terms of finding & parsing documentationo. It returns a class filled with fully parsed & easily useable Python dictionaries.
+
+
+*This document has been loosely adapted from Denperidge's learning writeup. You can view it [here](https://github.com/Denperidge-Redpencil/Learning.md/blob/main/Notes/docs.md)*.

docs/references/documentation.md → docs/how-tos/documentation-quickstart.md

@@ -46,3 +46,14 @@ Checklist:
 - [ ] Explains design decisions
 - [ ] Considers alternatives
 - [ ] Helps the reader make sense of things
+
+
+## Releasing a new version
+Once a new version of a semantic.works repository is made it should...
+- Be tagged using `git tag`
+- Have a DockerHub image with the same tag (NOTE: this happens automatically if your repository has a `.woodpecker/.tag.yml` configuration)
+
+*This will ensure that the documentations in app-mu-info get updated. See [project/discussions/how-tos/documentation-structure](../discussions/documentation-structure.md) for more info*
+
+
+

docs/how-tos/troubleshooting---slow-starting-containers.md

@@ -0,0 +1,25 @@
+# Troubleshooting - Slow starting containers using 100% CPU
+
+Some docker images used in Semantic Works stacks, notably those based on sbcl (common lisp) and elixir images, are very slow and CPU intensive to start if the limits of open file descriptors are very high for the container. This leads to a process using 100% of a CPU for some time before that container becomes usable. This can be worked around by setting the defaults for new containers in the docker daemon config (/etc/docker/daemon.json (create it if it doesn't exist)):
+
+```json
+{
+  "default-ulimits": {
+    "nofile": {
+      "Hard": 104583,
+      "Name": "nofile",
+      "Soft": 104583
+    }
+  }
+}
+```
+
+Or, if you want these high defaults for some reason, you can set per-container limits in a docker-compose file for each of the mu-project services:
+
+```yml
+    ulimits:
+      nofile:
+        soft: 104583
+        hard: 104583
+```
+

docs/references/template-requirements.md

@@ -0,0 +1,141 @@
+# Template requirements
+
+## PoC (proof of concept) requirements
+-   Respond to HTTP requests
+    -   Parse JSON & JSONAPI payloads
+    -   Support implementation of GET PUT PATCH POST DELETE
+    -   On port 80 during production
+
+-   Send out SPARQL queries & updates over HTTP
+    -   Perform JSON request to SPARQL HTTP endpoint
+    -   Parse JSON query response
+    -   Support Virtuoso Open Source [[1]](#1)
+
+-   Generate HTTP response
+    -   Build JSON & JSONAPI body
+    -   Set outgoing HTTP headers
+
+-   User-code
+    -   Load user-code from `/app`
+    -   Download & install dependencies on startup
+
+## Full template requirements
+
+-   [**All requirements for the PoC**](#poc-proof-of-concept-requirements)
+
+-   Configuration
+    -   Access environment variables [[2]](#notes)
+    -   Support live-reload for development [[3]](#notes)
+
+-   Respond to HTTP requests
+    -   Parse incoming HTTP headers
+    -   Destructure request URL
+    -   Parse query parameters [[4]](#notes)
+
+-   Send out SPARQL queries & updates over HTTP
+    -   Set headers of the SPARQL HTTP request [[5]](#notes)
+    -   Provide escaping methods for SPARQL input
+
+-   User-code
+    -   Load user configuration code from /config
+    -   Use ONBUILD to automate template extension [[6]](#notes)
+    -   Support the generation of UUIDs, namely uuid-v1
+
+-   Enhancing workflow
+    -   Use /template-name/setup.sh for downloading all dependencies 
+        into the image [[7]](#notes)
+    -   Use /template-name/startup.sh for launching the webserver.
+    -   Use the ENVIRONMENT variable to enable development mode &
+        live-reload [[8]](#notes)
+        -   Debugging console (if possible)
+        -   Enable debugging symbols (if possible)
+        -   Enable error output (if configurable)
+
+## Helper functions
+Ideally, your template should expose some helper functions, including
+but not limited to the following. Extra parameters can be added as 
+needed or wanted.
+
+
+| Function name      | Parameters            | Description |
+| ------------------ | --------------------- | ----------- |
+| error              | title: `string`, http_error_code: `int` | Returns a JSONAPI compliant error response with the given status code (default: 400). |
+| query              | query: `string`       | Executes the given SPARQL select/ask/construct query. [This function should automatically pass the correct headers](#passing-headers).  Also see `update`. |
+| session_id_header  | request: `object`     | Get the session id from the HTTP request headers. |
+| sparql_escape      | value: `any`          | Detects the value type and runs the appropiate [sparql escape function](#sparql_escape). |
+| update             | query: `string`       | Executes the given SPARQL update query. [This function should automatically pass the correct headers](#passing-headers). Also see `query`. |
+| uuid               | None                  | Generates an uuid. Meant for use in linked data (`mu:uuid`). | 
+
+*Note: camelcase is used in this documentation. However, when writing the helpers in the language of your choice, use the notation that is in line with the language in question.*
+
+### sparql_escape
+The following helper functions all have the same description/funcionality, but for different types. As such, the description column is omitted.
+
+**Description:** Converts the given object to a SPARQL-safe RDF object string with the right RDF-datatype.
+This functions should be used especially when inserting user-input to avoid SPARQL-injection.
+
+| Function name          | Parameters        |
+| ---------------------- | ----------------- |
+| sparql_escape_string   | value: `string`   |
+| sparql_escape_uri      | value: `url`      |
+| sparql_escape_decimal  | value: `decimal`  |
+| sparql_escape_int      | value: `int`      |
+| sparql_escape_float    | value: `float`    |
+| sparql_escape_date     | value: `date`     |
+| sparql_escape_datetime | value: `datetime` |
+| sparql_escape_bool     | value: `bool`     |
+
+### Passing headers
+The following headers should be passed when making queries:
+
+| Header name            | Type     | Description             |
+| ---------------------- | -------- | ----------------------- |
+| MU-SESSION-ID          |          |                         |
+| MU-CALL-ID             |          |                         |
+| MU-AUTH-ALLOWED-GROUPS |          | (bidirectional)         |
+| MU-AUTH-USED-GROUPS    |          |                         |
+
+
+## Notes
+
+1.  This constraint should be dropped in the distant future, but it is
+    our default choice at the time. If you don\'t do anything special,
+    this will be ok.
+
+2.  Docker makes it easy to set POSIX environment variables. These are
+    the variables you\'d be able to access in a shell (`$PATH` is a
+    good example). These should be accessible so services can override
+    your service.
+
+3.  Live-reload does not exist on all systems, but you can almost always
+    fake it. Some languages need a restart (like nodejs and ruby), for
+    some others you should connect to the service and reload the code
+    through your editor (like Common Lisp with slime). This reloading
+    does not need to auto-reload the dependencies, as these rarely
+    change and thus warrant restarting the service.
+
+4.  Query parameters is the content that goes behind the question mark
+    (?) in the URL. It constitutes of key-value pairs in which the
+    keys needn\'t be unique.
+
+5.  This requirement comes forth from the work Jonathan Langens is
+    performing on Sharing Knowledge (link should be added to basic
+    ground work in the field).
+
+6.  The ONBUILD statement of Docker allows you to execute commands when
+    someone extends your image. This means you can automatically copy
+    the relevant contents of the repository in your /app and your
+    /config. This further reduces the overhead of implementing a new
+    template.
+
+7.  This construction makes it predictable how extensions of this
+    template should behave. If someone builds a template on top of
+    your template (inception-style), then it\'s easily understandable
+    what needs to be called.
+
+8.  It may be that development mode can not be enabled using an
+    environment variable as it requires too many extra dependencies.
+    This is strongly advised, but it is an option. In that case, the
+    development image should be maintained in the same repository as
+    the template repository. The name of the development should be
+    `semtech/<template-name>-dev`