Docs: getting started section

docker/pdns/run.sh

@@ -32,6 +32,9 @@ webserver-allow-from=0.0.0.0/0
 api=yes
 api-key=changeme
 
+zone-cache-refresh-interval=0
+zone-metadata-cache-ttl=0
+
 load-modules=liblmdbbackend.so
 launch=lmdb
 lmdb-filename=/lmdb/instance-$instance/db

docs/getting-started.md

@@ -1,15 +1,100 @@
 # Getting started
 
-This section explains how to get started with PowerDNS Authoritative version 4.8+ and Lightning Stream.
+The easiest way try out Lightning Stream with the PowerDNS Authoritative server is through the Docker Compose demo
+in the [Lightning Stream Github repository](https://github.com/PowerDNS/lightningstream/).
 
+For manual installation instructions, please check [this section](pdns-auth-installation.md).
+
+
+## Docker Compose demo
+
+The Lightning Stream repository contains a Docker Compose demo of Lightning Stream running alongside
+the PowerDNS Authoritative server, to sync the LMDB backend data.
 
 !!! warning
 
-    This page assumes the use of PowerDNS Authoritative version 4.8 and higher, which uses [native Lightning Stream value headers](schema.md).
+    This demo does NOT handle [schema migrations](schema-migration.md) between different versions of PowerDNS Authoritative server.
+    It is NOT suitable for production use!
+
+The compose setup runs two read-write DNS servers, each with their own syncer, syncing to a bucket in a MinIO server.
+Additionally, a third server has Lightning Stream running in receive-only mode.
+
+The Lightning Stream config used can be found in `docker/pdns/lightningstream.yaml`. Note that the
+config file contents can reference environment variables.
+
+To get it up and running:
+
+    docker-compose up -d
+
+You may need to rerun this command once, because of a race condition when creating the LMDBs.
+
+To see the services:
+
+    docker-compose ps
+
+This should show output like:
+
+```
+NAME                      IMAGE                   SERVICE   PORTS
+lightningstream-auth1-1   powerdns/pdns-auth-48   auth1     127.0.0.1:4751->53/tcp, 127.0.0.1:4751->53/udp, 127.0.0.1:4781->8081/tcp
+lightningstream-auth2-1   powerdns/pdns-auth-48   auth2     127.0.0.1:4752->53/tcp, 127.0.0.1:4752->53/udp, 127.0.0.1:4782->8081/tcp
+lightningstream-auth3-1   powerdns/pdns-auth-48   auth3     127.0.0.1:4753->53/tcp, 127.0.0.1:4753->53/udp, 127.0.0.1:4783->8081/tcp
+lightningstream-minio-1   minio/minio             minio     127.0.0.1:4730->9000/tcp, 127.0.0.1:4731->9001/tcp
+lightningstream-sync1-1   lightningstream-sync1   sync1     127.0.0.1:4791->8500/tcp
+lightningstream-sync2-1   lightningstream-sync2   sync2     127.0.0.1:4792->8500/tcp
+lightningstream-sync3-1   lightningstream-sync3   sync3     127.0.0.1:4793->8500/tcp
+```
+
+Open a new terminal to see all the logs, to get a feeling of how and when Lightning Stream syncs data:
+
+    docker-compose logs
+
+Then, in another terminal, call these convenience scripts, with a delay between them to allow for syncing between instances:
+
+    docker/pdns/pdnsutil -i 1 create-zone example.org
+    docker/pdns/pdnsutil -i 1 secure-zone example.org
+    docker/pdns/pdnsutil -i 1 set-meta example.org foo bar
+    docker/pdns/pdnsutil -i 2 generate-tsig-key example123
+    docker/pdns/pdnsutil -i 1 add-record example.org www A 60 192.0.2.3
+
+    sleep 2
+
+    docker/pdns/curl-api -i 2 /api/v1/servers/localhost/zones/example.org
+    docker/pdns/curl-api -i 2 /api/v1/servers/localhost/zones/example.org/metadata
+    docker/pdns/curl-api -i 1 /api/v1/servers/localhost/tsigkeys
+
+These scripts execute `pdnsutil` and `curl` inside of the Docker containers. The `-i` flag specifies the instance number
+to operate on.
+
+For example, you can check if the records were correctly synced to instance 2 with this command:
+
+    $ docker/pdns/pdnsutil -i 2 list-zone example.org
+    $ORIGIN .
+    example.org	3600	IN	SOA	a.misconfigured.dns.server.invalid hostmaster.example.org 0 10800 3600 604800 3600
+    www.example.org	60	IN	A	192.0.2.3
+
+The same should be true for instance 3, which is in receive-only mode:
+
+    $ docker/pdns/pdnsutil -i 3 list-zone example.org
+    $ORIGIN .
+    example.org	3600	IN	SOA	a.misconfigured.dns.server.invalid hostmaster.example.org 0 10800 3600 604800 3600
+    www.example.org	60	IN	A	192.0.2.3
+
+You can use `dig` against the DNS servers to verify that the new `A` record works:
+
+    $ dig +short -p 4753 @127.0.0.1 www.example.org
+    192.0.2.3
+
+To list all the generated snapshots, you can use this script:
+
+    docker/pdns/lightningstream -i 1 snapshots list -l
+
+To view a dump of the LMDB contents, you can use the `dump-lmdb` command:
 
-    TODO: explain how to use with 4.7. Can it work with 4.6?
+    docker/pdns/dump-lmdb -i 1
+    docker/pdns/dump-lmdb -i 2
 
+You can also browse the snapshots in MinIO at <http://localhost:4731/buckets/lightningstream/browse>
+(login with minioadmin / minioadmin).
 
-TODO:
 
-- Docker demo (no schemaversion handling!)

docs/pdns-auth-installation.md

@@ -0,0 +1,154 @@
+# Lightning Stream with PowerDNS Authoritative server
+
+This section explains how to install and run Lightning Stream with the PowerDNS Authoritative
+server the old fashioned way.
+
+!!! TODO
+
+    Explain how to handle [schema migrations](schema-migration.md) in this setup.
+
+## Building Lightning Stream 
+
+At the moment of writing, this project requires Go 1.19. Please check the `go.mod` file for the current
+Go version expected.
+
+To install the binary in a given location, simply run:
+
+    GOBIN=$HOME/bin go install ./cmd/lightningstream/
+
+Or run `./build.sh` to install it in a `bin/` subdirectory of this repo. 
+
+Easy cross compiling is unfortunately not supported, because the LMDB bindings require CGo.
+
+
+## Configuring PowerDNS Authoritative server 4.8+
+
+To install PowerDNS Authoritative server, please read [its installation instructions](https://doc.powerdns.com/authoritative/installation.html).
+Make sure to install version 4.8 or higher for Lightning Stream. Also install the `lmdb` backend for PowerDNS Authoritative server, if packaged
+separately.
+
+!!! warning
+
+    This section assumes the use of PowerDNS Authoratitive version 4.8 and higher, which uses [native Lightning Stream value headers](schema.md).
+    As of March 2023, an alpha release of 4.8 is available for testing. 
+
+    These instructions will NOT work with earlier versions, but scroll down if you insist on trying it out with 4.7.
+
+Lightning Stream requires the following PowerDNS Authoritative server settings:
+
+```
+# Lightning Stream uses the LMDB backend 
+load-modules=liblmdbbackend.so
+launch=lmdb
+
+# Path to the directory where the LMDB databases for this instance will be stored.
+# This MUST be unique per instance, if you are running more than one on the same server.
+lmdb-filename=/path/to/lmdb
+
+# Run it with a single shard, to simplify management and configuration.
+# Note that this cannot safely be changed later!
+lmdb-shards=1
+
+# This MUST be enabled to safely handle multiple writers
+lmdb-random-ids=yes
+
+# This MUST be enabled to track and propagate deletes
+lmdb-flag-deleted=yes
+
+# You may want a lower number than 16000 MB, which is the default on 64 bit systems.
+lmdb-map-size=1000
+
+# You may want to reduce the cache interval to 1 second, or disable it
+# altogether with 0, to quickly see your changes. The default is 300 seconds.
+# An interval of 1 second will likely provide you with most of the benefits of caching,
+# with a barely noticeable delay.
+zone-cache-refresh-interval=0
+zone-metadata-cache-ttl=0
+```
+
+
+## Configuring Lightning Stream
+
+A basic Lightning Stream configuration for PowerDNS Authoritative looks like this:
+
+```yaml
+instance: unique-instance-name  # IMPORTANT: change this
+lmdbs:
+  main:
+    # Auth 'lmdb-filename' plus '/db'
+    path: /path/to/lmdb/db
+    schema_tracks_changes: true
+    options:
+      no_subdir: true
+      create: true      # optional for 'main', as auth will create it on startup, if needed
+      map_size: 1000MB  # for create=true, make sure to match auth's lmdb-map-size
+  shard:
+    # Auth 'lmdb-filename' plus '/db-0'
+    path: /path/to/lmdb/db-0
+    schema_tracks_changes: true
+    options:
+      no_subdir: true
+      create: true      # strongly recommended for shards
+      map_size: 1000MB  # for create=true, make sure to match auth's lmdb-map-size
+
+storage:
+  #type: fs
+  type: s3
+  options:
+    #root_path: /tmp/snapshots
+    access_key: minioadmin
+    secret_key: minioadmin
+    region: us-east-1
+    bucket: lightningstream-auth48 # use a different bucket or prefix for each auth version
+    create_bucket: true
+    endpoint_url: http://localhost:9000
+
+http:
+  address: ":8500"  # for status and metrics
+```
+
+Please check [the configuration section](configuration.md) for details and other options.
+
+Lightning Stream can be run in the foreground as follows:
+
+    $ lightningstream --config=/path/to/config.yaml sync 
+
+Ensure that both PowerDNS Authoritative and Lightning Stream have write access to the LMDBs,
+for example by running them under the same system user.
+
+
+
+## Running it with an older Authoritative server
+
+While it is possible to run Lightning Stream with PowerDNS Authoritative 4.7 in [non-native mode](schema-shadow.md),
+this is NOT recommended due to the [many downsides](schema-shadow.md#caveats) of running it in non-native mode.
+
+!!! warning
+
+    This is NOT recommended, other than for testing and experimenting with non-native shadow mode.
+    Please use PowerDNS Authoritative 4.8+ instead.
+
+To run it with PowerDNS Authoritative 4.7 in non-native mode, change the above Lightning Stream configuration as follows:
+
+```yaml
+# WARNING: NOT RECOMMENDED, non-native mode for PowerDNS Authoritative server 4.7
+lmdbs:
+  main:
+    ...
+    schema_tracks_changes: false
+    dupsort_hack: true
+  shard:
+    ...
+    schema_tracks_changes: false
+
+storage:
+  type: s3
+  options:
+    ...
+    bucket: lightningstream-auth47 # use a different bucket or prefix for each auth version
+```
+
+And remove the `lmdb-flag-deletes` option from the Auth config. The Lightning Stream shadow DBIs will be
+tracking these deletes instead.
+
+

mkdocs.yml

@@ -17,6 +17,7 @@ nav:
   - 'Commands': commands.md
   - 'PowerDNS Integration':
     - 'Getting Started': getting-started.md
+    - 'Traditional installation': pdns-auth-installation.md
   - 'Schema':
     - 'General considerations': schema.md
     - 'Native header schema': schema-native.md