Skip to content

tapdb+sqlc: sqlc: add script to merge SQL migrations into consolidated schemas #1387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Feb 20, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -52,3 +52,4 @@ vendor/

# Release builds
/taproot-assets-*
.aider*
14 changes: 12 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -249,10 +249,20 @@ gen: rpc sqlc
sqlc:
@$(call print, "Generating sql models and queries in Go")
./scripts/gen_sqlc_docker.sh
@$(call print, "Merging SQL migrations into consolidated schemas")
go run ./cmd/merge-sql-schemas/main.go

sqlc-check: sqlc
sqlc-check:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you remove the dependency on the sqlc target? Now it won't actually generate the schema anymore to see if there's a diff.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad, missed this. I removed it to verify that the bash fragment I had worked (so I could remove the file, then run the check w/o it adding the file again).

@$(call print, "Verifying sql code generation.")
if test -n "$$(git status --porcelain '*.go')"; then echo "SQL models not properly generated!"; git status --porcelain '*.go'; exit 1; fi
@if [ ! -f tapdb/sqlc/schemas/generated_schema.sql ]; then \
echo "Missing file: tapdb/sqlc/schemas/generated_schema.sql"; \
exit 1; \
fi
@if test -n "$$(git status --porcelain '*.go')"; then \
echo "SQL models not properly generated!"; \
git status --porcelain '*.go'; \
exit 1; \
fi

rpc:
@$(call print, "Compiling protos.")
Expand Down
106 changes: 106 additions & 0 deletions cmd/merge-sql-schemas/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
package main

import (
"database/sql"
"log"
"os"
"path/filepath"
"regexp"
"sort"

_ "modernc.org/sqlite" // Register the pure-Go SQLite driver.
)

func main() {
// Open an in-memory SQLite database.
db, err := sql.Open("sqlite", ":memory:")
if err != nil {
log.Fatalf("failed to open in-memory db: %v", err)
}
defer db.Close()

migrationDir := "tapdb/sqlc/migrations"
files, err := os.ReadDir(migrationDir)
if err != nil {
log.Fatalf("failed to read migration dir: %v", err)
}

var upFiles []string
upRegex := regexp.MustCompile(`\.up\.sql$`)
for _, f := range files {
if !f.IsDir() && upRegex.MatchString(f.Name()) {
upFiles = append(upFiles, f.Name())
}
}
sort.Strings(upFiles)

// Execute each up migration in order.
for _, fname := range upFiles {
path := filepath.Join(migrationDir, fname)
data, err := os.ReadFile(path)
if err != nil {
log.Fatalf("failed to read file %s: %v", fname, err)
}
_, err = db.Exec(string(data))
if err != nil {
log.Fatalf("error executing migration %s: %v", fname,
err)
}
}

// ---------------------------------------------------------------------
// Retrieve final database schema from sqlite_master.
//
// SQLite automatically maintains a special table called sqlite_master,
// which holds metadata about all objects inside the database, such as
// tables, views, indexes, and triggers. Each row in this table
// represents an object, with columns such as "type" (the kind of
// object), "name" (the object's name), and "sql" (the SQL DDL statement
// that created it).
//
// In our case, after running all the migration files on an in‑memory
// database, we execute the following query to extract only the schema
// definitions for tables and views. Ordering by name ensures the output
// is stable across runs.
//
// This way, we can consolidate and export the complete database schema
// as it stands after all migrations have been applied.
// ---------------------------------------------------------------------
rows, err := db.Query(`
SELECT type, name, sql FROM sqlite_master
WHERE type IN ('table','view') ORDER BY name`,
)
if err != nil {
log.Fatalf("failed to query schema: %v", err)
}
defer rows.Close()

var generatedSchema string
for rows.Next() {
var typ, name, sqlDef string
if err := rows.Scan(&typ, &name, &sqlDef); err != nil {
log.Fatalf("error scanning row: %v", err)
}

// Append the retrieved CREATE statement. We add a semicolon and
// a couple of line breaks to clearly separate each object's
// definition.
generatedSchema += sqlDef + ";\n\n"
}
if err := rows.Err(); err != nil {
log.Fatalf("error iterating rows: %v", err)
}

// Finally, we'll write out the new schema, taking care to ensure that
// that output dir exists.
outDir := "tapdb/sqlc/schemas"
if err = os.MkdirAll(outDir, 0755); err != nil {
log.Fatalf("failed to create schema output dir: %v", err)
}
outFile := filepath.Join(outDir, "generated_schema.sql")
err = os.WriteFile(outFile, []byte(generatedSchema), 0644)
if err != nil {
log.Fatalf("failed to write final schema file: %v", err)
}
log.Printf("Final consolidated schema written to %s", outFile)
}
Loading
Loading