feat: integrated sqlite db for metadata #21543
Conversation
williamhbaker
force-pushed
the
wb-sqlite-metadb-21260
branch
from
47a64fb to
dfe5fa1
Compare
go.mod
Outdated
| @@ -58,6 +58,7 @@ require ( | |||
| github.com/kevinburke/go-bindata v3.11.0+incompatible | |||
| github.com/lib/pq v1.2.0 // indirect | |||
| github.com/mattn/go-isatty v0.0.12 | |||
| github.com/mattn/go-sqlite3 v1.11.0 | |||
There was a problem hiding this comment.
Curious, why v1.11.0 instead of the lastest v1.14.7?
There was a problem hiding this comment.
No particular reason - v1.11.0 was actually in the go.sum (has been for a while it looks like...?) so I guess that's why it ended up being v1.11.0 in go.mod. That's pretty old and there's no specific reason for us to not use the latest version so I updated to v1.14.7.
There was a problem hiding this comment.
Needs some CHANGELOGing as well! For something this big I'd probably add a dedicated section at the top about adding sqlite in general (see previous releases where we had blurbs about adding ARM, or about breaking APIs). Then there could be 2 lines in the "new features" section for:
- Adding the
--sqlite-pathconfig option - Adding the
diskoption to--store
Oh yeah definitely...I updated the change log, let me know if it makes sense. |
Closes #21260
Closes #21261
This sets up the basic functionality of a sqlite metadata store in the application. There is a wrapper around the
*sql.DBwhich provides a few utility methods including flushing the database tables of data for end-to-end testing and applying database migrations upon startup ofinfluxd.The migration system uses
go-bindatawhich is pretty clunky compared to the native embedding that is part ofgo1.16, and I hope to refactor the migration code when we upgrade.It is also worth pointing out that this includes changes to the flags used to start
influxd, including:--storeflag used to work with string values ofboltormemory- it will now work withdiskormemory, and alsoboltfor backwards compatibility. Using eitherdiskorboltutilizes on-disk storage for metadata, whereasmemoryuses temporary in-memory databases for testing purposes.--sqlite-pathflag for specifying the path to the sqlite database! This is comparable to the--bolt-pathand--engine-pathflags. If a--bolt-pathis specified but not a--sqlite-path, the--sqlite-pathwill be set to store the sqlite database with the default name in the same directory as specified by the--bolt-path.