v0.5.0

New feature: tombsweeper

The main new feature in the release is the 'tombsweeper'. When a record is deleted in a Lightning Stream installation, we do not actually remove the record, but instead write a deletion marker. Up until now these deletion markers were never deleted, leading to unbounded database growth in installations that see delete-heavy workloads. The new tombsweeper can clean these deletion markers after a configured interval.

WARNING: This is DISABLED by default, because enabling this functionality requires careful consideration. When enabled, you MUST make sure that no instance that has been offline for longer than the sweeper retention_days will ever reconnect. If this does happen, old entries that have been deleted may be resurrected causing anything from undesired results to database state corruption. Be especially careful about development or testing systems that only come online occasionally.

For small personal installations with infrequent changes, it is probably better NOT to enable this functionality, and accept that deleted entries accumulate over time. Only consider this you have frequent updates and deletion markers would actually become a problem over time. And in that case, be very conservative with the retention_days setting, the longer the better.

What's Changed

• Tombsweeper: clean stale delete markers by @wojas in #89
• Rename module to github.com/PowerDNS/lightningstream/lmdbenv by @wojas in #76
• GitHub Actions workflow for building and pushing multi-platform Docker images to Docker Hub by @joel-ling in #78
• Upgrade to Go 1.24 & CI fixes by @wojas in #87
• Add hooks by @wojas in #88
• Module upgrades, including lmdb-go to 1.9.3 by @wojas in #91
• Add Minio to github actions to use in test suite by @verscheures in #77

New Contributors

• @joel-ling made their first contribution in #78
• @verscheures made their first contribution in #77

Full Changelog: v0.4.3...v0.5.0

v0.4.4

Very minor update that only updates the Docker image base distribution to Debian Bookworm.

What's Changed

• Backport to 0.4: use debian bookworm for docker image by @wojas in #92

Full Changelog: v0.4.3...v0.4.4

v0.4.3

What's Changed

• Downloader: call Close on updates that are overwritten in (*Downlader).LoadOnce by @ahouene in #70
• S3 snapshot metrics by @nvaatstra in #74

New Contributors

• @nvaatstra made their first contribution in #74

Full Changelog: v0.4.2...v0.4.3

v0.4.2

What's Changed

• pdns.conf fix, plus a doc nit by @Habbie in #60
• upgrade simpleblob to v0.2.5, as well as compress and logrus by @ahouene in #62

New Contributors

• @Habbie made their first contribution in #60
• @ahouene made their first contribution in #62

Full Changelog: v0.4.1...v0.4.2

v0.4.1

This fixes a resource leak that can cause growing memory and CPU usage over time (#57) for all S3 backend users.

It also fixes the update marker not working when a global S3 prefix is used, which causes the backend to use more expensive LIST API calls, leading to increased S3 usage costs. (#55)
This only affects you if you are using the opt-in upgrade marker.

What's Changed

• Upgrade Simpleblob to v0.2.4 by @franklouwers in #58

New Contributors

• @franklouwers made their first contribution in #58, thank you!

Full Changelog: v0.4.0...v0.4.1

v0.4.0

What's Changed

This release introduces a new snapshot loader and serialiser that significantly reduces the memory use of Lightning Stream, by keeping the items in protobuf format and deserialising on the fly, and several other improvements to reduce peak memory usage.

We now limit the number of snapshots that can concurrently be held in memory using the new memory_downloaded_snapshots and memory_decompressed_snapshots configuration options. In the past this number could grow up to the number of unique instance names.

• Reduce heap memory usage by @wojas in #31
• Upgrade simpleblob to 0.2.3 by @wojas in #51
• Experimental pdns-v5-fix-duplicate-domains command by @wojas in #50

Improvements to documentation, tests and CI

• Minor readme fix by @Bodenhaltung in #48
• Check spelling updates by @jsoref in #47
• Update docs for 0.4.0 by @wojas in #47

New Contributors

• @Bodenhaltung made their first contribution in #48

Full Changelog: v0.3.1...v0.4.0

v0.3.1

What's Changed

• Reduce memory used for when loading a snapshot by @wojas in #30
• New experimental migrate-timestamps command by @wojas in #28
• Upgrade docker compose to Auth 4.8 with native schema by @wojas in #34
• Add MIT license by @wojas in #35

Security content

• Bump golang.org/x/net from 0.1.0 to 0.7.0 by @dependabot in #45

Improvements to documentation, tests and CI

• Go tests: run in Github Actions, and make the sync tests reliable by @wojas in #32
• Add documentation by @wojas in #29
• Reword test feature message by @jsoref in #38
• Spelling by @jsoref in #39
• Fix two typos by @horazont in #40
• Docs: getting started section by @wojas in #42
• Fix race in syncer test by @wojas in #43
• Add check-spelling by @jsoref in #41
• Spelling by @wojas in #46
• CI: build binaries for linux and macOS by @wojas in #44

New Contributors

• @jsoref made their first contribution in #38
• @horazont made their first contribution in #40
• @dependabot made their first contribution in #45

Full Changelog: v0.3.0...v0.3.1

v0.3.0

Released on 1 March 2023.

BREAKING CHANGES

This one will not affect any actual users, as we know there were no native schema applications before v0.3.0.

The header format changed from an 8 byte timestamp to a new 24+ byte header.
Non-native schemas will have new shadow tables automatically created in the new format, but may lose some timestamp accuracy for changes around the migration.

Highlights

• New header schema #10
• Automatic cleanup of old snapshots #9
• Single-run support with --only-once #11 and #13
• Add receive command for receive-only operation #24
• Various snapshots subcommand for debugging and managing snapshots #26

Other changes

• Fix incorrect DBI creation causing corrupt dupsort data #20
• Healthz monitoring endpoint #7
• Various fixes for native schema applications #14 and #17
• Go 1.19 build fixes #6
• Performance improvements #15
• Improved status page #18 #25
• Other fixes and improvements #19
• Wait for marker file flag #21
• Allow zero timestamps #22
• Add override_create_flags option #23

Full Changelog: v0.2.0...v0.3.0

v0.2.0

Released on 11 November 2022.

Initial external release.