[feat] add support for outside-of-transaction migrations

.github/copilot-instructions.md

@@ -0,0 +1,56 @@
+# GitHub Copilot Contribution Rules
+
+One page. No ambiguity. Follow or reject the change.
+
+## Good engineering
+- always prefer to remove code than to add it
+- always look for ways to simplify
+
+## Core
+- Remove things completely. No placeholders, stubs, TODO, commentary about removal. History lives only in git.
+- No historical or refactor narration comments. Comments only explain current behavior, invariants, limits, dialect nuances, or (released) API deprecations.
+- ASCII only. No Unicode of any kind.
+
+## Public API
+- Public = exported + existed in a tagged release + intentionally documented.
+- Changing/removing: update doc comment only if callers must adapt.
+- Shim only if break is not trivially fixable. Shim must be minimal, delegated, marked `// Deprecated:` with planned removal version, and tested.
+- No shims for anything else.
+
+## Testing
+- Coverage target 95-100% for unit-testable code. 
+- No mocks.
+- Use testify.
+- Deterministic, table-driven tests. Assert meaningful outcomes only.
+- Disallow: mocking frameworks, log-text assertions (unless log text is contract), sleeps for timing.
+- Prefer errors.Is() over checking error text
+
+## Style & Refactors
+- Prefer elimination over abstraction. No commented-out code. No speculative feature flags.
+- Inline vs extract based strictly on clarity or real reuse.
+- Silent refactors do not add commentary. Behavioral changes only update directly affected docs.
+
+## Errors
+- Use memsql/errors only.
+- Sentinel errors with errors.String type, augmented with .Errorf
+- Wrap once at subsystem boundaries with context. Do not re-wrap in tight loops. Avoid repeating obvious parameter values.
+- Don't capitalize the first letter
+
+## Temporary changes for debugging
+- Mark temporary code with "XXX"
+- Add freely
+
+## Checklist (all must be true)
+- [ ] No history / refactor narration comments.
+- [ ] No placeholders or commented-out code left behind.
+- [ ] ASCII only.
+- [ ] Public API changes reviewed; shims only when allowed & tested.
+- [ ] Coverage for unit-testable code >=95% 
+- [ ] No mocks.
+- [ ] Error wrapping matches boundary rule.
+- [ ] Tests use testify and do not use t.Errorf() or t.FailNow()
+- [ ] Sentinel errors use errors.String
+- [ ] No temporary (XXX) code remaining for commits
+
+## Enforcement
+Non-compliant changes are rejected regardless of technical merit.

.github/workflows/codeql-analysis.yml

@@ -64,10 +64,10 @@ jobs:
     - name: Autobuild
       uses: github/codeql-action/autobuild@f443b600d91635bebf5b0d9ebc620189c0d6fba5
 
-    # ℹ️ Command-line programs to run using the OS shell.
-    # 📚 https://git.io/JvXDl
+  # Info: Command-line programs to run using the OS shell.
+  # Docs: https://git.io/JvXDl
 
-    # ✏️ If the Autobuild fails above, remove it and uncomment the following three lines
+  # If the Autobuild fails above, remove it and uncomment the following three lines
     #    and modify them (or add more) to build your code if your project
     #    uses a compiled language
 

README.md

@@ -149,6 +149,28 @@ complete.  If the migration is revised, then the later parts can
 be re-tried as long as the earlier parts are not modified.  This
 does not apply to `Compute()`ed migrations.
 
+### PostgreSQL non-transactional (idempotent) DDL
+
+Some PostgreSQL statements (e.g. `CREATE INDEX CONCURRENTLY`, `DROP
+INDEX CONCURRENTLY`, `REFRESH MATERIALIZED VIEW CONCURRENTLY`)
+cannot run inside an explicit transaction block. Libschema auto-detects
+common patterns in `Script(...)` and will execute those outside a
+transaction; you can also opt in explicitly by using a generic type
+that is *not* a `*sql.Tx` (e.g. `Generate[*sql.DB]`). Non-transactional
+migrations must be idempotent: safe to retry after partial failure.
+A fuller rationale, supported patterns, and enum/value guidance
+live in `lspostgres/NON_TRANSACTIONAL.md`.
+
+### RepeatUntilNoOp (brief)
+
+`RepeatUntilNoOp()` re-runs a single DML statement until
+`RowsAffected()==0`. Use only for pure data updates where the driver
+reports accurate counts. Avoid DDL, guarded `CREATE ... IF NOT
+EXISTS`, concurrent index / materialized view commands, or
+multi-statement scripts. For anything more complex, prefer a
+`Computed` migration and loop explicitly. See the Go doc comment
+on `RepeatUntilNoOp` for detailed guidance.
+
 ## Command line
 
 The `OverrideOptions` can be added as command line flags that 
@@ -291,9 +313,11 @@ will be clearly documented and will fail in a way that does not cause hidden pro
 For example, switching from using "flag" to using OverrideOptions will trigger
 an obvious breakage if you try to use a flag that no longer works.
 
-Anticpated changes for the future:
+Anticipated changes:
 
 - API tweaks
 - Support for additional databases
 - Support for additional logging APIs
 - Support for tracing spans (per migration)
+
+

api.go

@@ -4,9 +4,27 @@ import (
 	"context"
 	"database/sql"
 
+	"github.com/memsql/errors"
+
 	"github.com/muir/libschema/internal"
+)
+
+const (
+	// ErrDataAndDDL indicates a single migration mixes schema changes (DDL) and data
+	// manipulation (DML) where that combination is disallowed.
+	ErrDataAndDDL errors.String = "migration combines DDL (schema changes) and data manipulation"
+
+	// ErrNonIdempotentDDL indicates a non-transactional migration contains easily
+	// guardable DDL lacking an IF (NOT) EXISTS clause.
+	ErrNonIdempotentDDL errors.String = "unconditional migration has non-idempotent DDL"
 
-	"github.com/pkg/errors"
+	// ErrNonTxMultipleStatements indicates a non-transactional (idempotent) script migration
+	// attempted to execute multiple SQL statements when only one is allowed.
+	ErrNonTxMultipleStatements errors.String = "non-transactional migration has multiple statements"
+
+	// ErrNonIdempotentNonTx indicates a required-to-be-idempotent non-transactional migration
+	// failed idempotency validation heuristics.
+	ErrNonIdempotentNonTx errors.String = "non-idempotent non-transactional migration"
 )
 
 const DefaultTrackingTable = "libschema.migration_status"
@@ -46,14 +64,16 @@ type MigrationOption func(Migration)
 
 // Migration defines a single database defintion update.
 type MigrationBase struct {
-	Name            MigrationName
-	async           bool
-	rawAfter        []MigrationName
-	order           int // overall desired ordring across all libraries, ignores runAfter
-	status          MigrationStatus
-	skipIf          func() (bool, error)
-	skipRemainingIf func() (bool, error)
-	repeatUntilNoOp bool
+	Name             MigrationName
+	async            bool
+	rawAfter         []MigrationName
+	order            int // overall desired ordring across all libraries, ignores runAfter
+	status           MigrationStatus
+	skipIf           func() (bool, error)
+	skipRemainingIf  func() (bool, error)
+	repeatUntilNoOp  bool
+	nonTransactional bool  // set automatically or by ForceNonTransactional / inference
+	forcedTx         *bool // if not nil, explicitly chosen transactional mode (true=transactional, false=non-transactional)
 }
 
 func (m MigrationBase) Copy() MigrationBase {
@@ -190,16 +210,26 @@ func Asynchronous() MigrationOption {
 	}
 }
 
-// RepeatUntilNoOp marks a migration as potentially being needed to run multiple times.
-// It will run over and over until the database reports that the migration
-// modified no rows.  This can useuflly be combined with Asychnronous.
+// RepeatUntilNoOp marks a migration (Script or Generate) to be re-executed until
+// the underlying driver reports zero rows affected. Use it for single, pure DML
+// statements that progressively transform data (e.g. UPDATE batches that move a
+// limited subset each run). It is NOT a generic looping primitive.
+//
+// Safety / correctness guidelines:
+//   - Single statement only - multi-statement scripts can give a meaningless final RowsAffected().
+//   - DML only - avoid DDL (CREATE/ALTER/DROP/REFRESH) or utility commands; most return 0 and will
+//     terminate immediately or loop uselessly.
+//   - Idempotent per batch - repeating the same statement after partial success must not corrupt data.
+//   - Do NOT mix with non-transactional Postgres DDL (concurrent index builds, REFRESH MATERIALIZED VIEW CONCURRENTLY).
+//   - If logic needs conditionals or multiple statements, write a Computed migration and loop explicitly.
 //
-// This marking only applies to Script() and Generated() migrations.  The migration
-// must be a single statement.
+// Prefer a Computed migration when:
+//   - You need to run multiple statements per batch
+//   - You must inspect progress with custom queries
+//   - RowsAffected() is unreliable or driver-dependent
 //
-// For Computed() migrations, do not use RepeatUntilNoOp.  Instead simply write the
-// migration use Driver.DB() to get a database handle and use it to do many little
-// transactions, each one modifying a few rows until there is no more work to do.
+// Future note: libschema may emit debug warnings for obviously unreliable usages
+// (e.g. DDL + RepeatUntilNoOp). Treat the above bullets as normative behavior now.
 func RepeatUntilNoOp() MigrationOption {
 	return func(m Migration) {
 		m.Base().repeatUntilNoOp = true
@@ -247,6 +277,58 @@ func SkipRemainingIf(pred func() (bool, error)) MigrationOption {
 	}
 }
 
+// ForceNonTransactional forces a migration (Script, Generate, or Computed) to run without
+// a wrapping transaction. By using this you assert the migration is idempotent (safe to retry).
+// Overrides any automatic inference the driver would normally perform.
+//
+// Generate note: For drivers where generation always occurs inside a transaction context (e.g. Postgres
+// Generate with *sql.Tx generator) forcing non-transactional only affects execution of the
+// produced SQL, not the generator callback itself. Drivers will reject impossible combinations (e.g.
+// attempting to force non-transactional on a generator that fundamentally requires *sql.Tx if they cannot
+// safely downgrade).
+func ForceNonTransactional() MigrationOption {
+	return func(m Migration) {
+		b := m.Base()
+		v := false // false means non-transactional
+		b.forcedTx = &v
+	}
+}
+
+// ForceTransactional forces a migration to run inside a transaction even if automatic inference
+// would choose non-transactional execution.
+//
+// Generate note: For generator-based migrations whose callback already receives *sql.Tx (Generate)
+// this is effectively a no-op (they are already transactional). For generators that inherently execute
+// outside a transaction (none exist in current public API) forcing transactional would be rejected.
+//
+// WARNING (foot-gun): On MySQL / SingleStore this DOES NOT make DDL atomic. Those engines
+// autocommit each DDL statement regardless of any BEGIN you issue. By forcing transactional mode:
+//   - Earlier DML in the same migration may roll back while preceding DDL remains applied.
+//   - The bookkeeping UPDATE that records migration completion can fail while schema changes
+//     have already been partially or fully applied (leaving the migration marked incomplete and
+//     retried later against an already-changed schema).
+//   - Mixed DDL + DML ordering inside a forced transactional migration can produce inconsistent,
+//     misleading results during retries or partial failures.
+//
+// Use this only if you fully understand these semantics and prefer to retain a transactional wrapper
+// for non-DDL statements. If you simply need safe DDL, prefer writing idempotent statements and let
+// the driver downgrade automatically.
+func ForceTransactional() MigrationOption {
+	return func(m Migration) {
+		b := m.Base()
+		v := true // true means transactional
+		b.forcedTx = &v
+	}
+}
+
+// ApplyForceOverride overrides transactionality for any prior force call (ForceTransactional
+// or ForceNonTransactional)
+func (m *MigrationBase) ApplyForceOverride() {
+	if m.forcedTx != nil { // forced override present
+		m.SetNonTransactional(!*m.forcedTx)
+	}
+}
+
 func (d *Database) DB() *sql.DB {
 	return d.db
 }
@@ -290,6 +372,26 @@ func (m *MigrationBase) HasSkipIf() bool {
 	return m.skipIf != nil
 }
 
+// NonTransactional reports if the migration must not be wrapped in a transaction.
+// This is automatically set for drivers (e.g. Postgres) that provide generic
+// migration helpers which infer non-transactional status from the connection
+// type used (e.g. *sql.DB vs *sql.Tx). A migration marked nonTransactional will
+// be executed without an encompassing BEGIN/COMMIT; status recording still
+// occurs within its own small transaction when supported.
+func (m *MigrationBase) NonTransactional() bool {
+	return m.nonTransactional
+}
+
+func (m *MigrationBase) SetNonTransactional(v bool) {
+	m.nonTransactional = v
+}
+
+// ForcedTransactional reports if ForceTransactional() was explicitly called.
+func (m *MigrationBase) ForcedTransactional() bool { return m.forcedTx != nil && *m.forcedTx }
+
+// ForcedNonTransactional reports if ForceNonTransactional() was explicitly called.
+func (m *MigrationBase) ForcedNonTransactional() bool { return m.forcedTx != nil && !*m.forcedTx }
+
 func (n MigrationName) String() string {
 	return n.Library + ": " + n.Name
 }

apply.go

@@ -5,11 +5,11 @@ import (
 	"log"
 	"os"
 
-	"github.com/muir/libschema/internal"
-
 	"github.com/hashicorp/go-multierror"
+	"github.com/memsql/errors"
+
 	"github.com/muir/libschema/dgorder"
-	"github.com/pkg/errors"
+	"github.com/muir/libschema/internal"
 )
 
 // Migrate runs pending migrations that have been registered as long as
@@ -75,7 +75,7 @@ func (s *Schema) Migrate(ctx context.Context) (err error) {
 			return err
 		}
 	}
-	return
+	return err
 }
 
 func (d *Database) prepare(ctx context.Context) error {

config.go

@@ -51,5 +51,4 @@ type OverrideOptions struct {
 //	import "flag"
 //	import "github.com/muir/nfigure"
 //	nfigure.MustExportToFlagSet(flag.CommandLine, "flag", &libschema.DefautOverrides)
-//
 var DefaultOverrides OverrideOptions

dgorder/dependencies.go

@@ -3,7 +3,7 @@ package dgorder
 import (
 	"container/heap"
 
-	"github.com/pkg/errors"
+	"github.com/memsql/errors"
 )
 
 type Node struct {
@@ -74,6 +74,7 @@ func (h *IntHeap) Push(x interface{}) {
 	// not just its contents.
 	*h = append(*h, x.(int))
 }
+
 func (h *IntHeap) Pop() interface{} {
 	old := *h
 	n := len(old)

driver.go

@@ -2,8 +2,9 @@ package libschema
 
 import (
 	"database/sql"
-	"fmt"
 	"strings"
+
+	"github.com/memsql/errors"
 )
 
 // maps from DSN to driver name
@@ -22,7 +23,7 @@ func OpenAnyDB(dsn string) (*sql.DB, error) {
 				return sql.Open(driver, dsn)
 			}
 		}
-		return nil, fmt.Errorf("could not find database driver matching %s", wanted)
+		return nil, errors.Errorf("could not find database driver matching %s", wanted)
 	}
-	return nil, fmt.Errorf("could not find appropriate database driver for DSN")
+	return nil, errors.Errorf("could not find appropriate database driver for DSN")
 }

go.mod

@@ -8,18 +8,18 @@ require (
 	github.com/go-sql-driver/mysql v1.9.3
 	github.com/hashicorp/go-multierror v1.1.1
 	github.com/lib/pq v1.10.9
+	github.com/memsql/errors v0.2.0
 	github.com/muir/sqltoken v0.1.0
 	github.com/muir/testinglogur v0.0.1
-	github.com/pkg/errors v0.9.1
 	github.com/stretchr/testify v1.11.1
 )
 
 require (
 	filippo.io/edwards25519 v1.1.0 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/hashicorp/errwrap v1.0.0 // indirect
-	github.com/kr/pretty v0.1.0 // indirect
+	github.com/kr/text v0.2.0 // indirect
+	github.com/pkg/errors v0.9.1 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
-	gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 )