wal: validate recovery directory through a stable identifier

[annrpom]

We now persist a stable identifier to both the OPTIONS file and a file within the secondary directory (failover_identifier). If recovery finds that the secondary directory does not contain a matching identifier, we abort recovery indicating that the secondary seems incorrect / corrupt.

Fixes: #4416

[cockroach-teamcity]

This change is 

[jbowens]

@jbowens reviewed 4 of 9 files at r1, all commit messages.
Reviewable status: 4 of 9 files reviewed, 2 unresolved discussions (waiting on @xxmplus)

---

wal/failover_manager.go line 642 at r1 (raw file):

			o.SecondaryIdentifier = identifier
		} else {
			o.SecondaryIdentifier = existingIdentifier

instead of mutating the Options from inside the WAL manager initialization, maybe we should expose a ValidateOrInitWALDir func from the wal package that we'd use like:

walDir, err = wal.ValidateOrInitWALDir(walDir)

and we could use this during Open to validate and initialize them before initializing the WAL manager; wdyt?

---

options.go line 1381 at r1 (raw file):

	// write to the primary WAL stalls. The Lock field may be set during setup to
	// preacquire the lock on the secondary directory.
	Secondary wal.Dir

Sorry this is expanding the scope of the change, but I think maybe we should embed an ID field into the wal.Dir. If the user disables WAL failover and passes the directory as a WALRecoveryDir, we can again validate it.

[annrpom]

Reviewable status: 1 of 10 files reviewed, 2 unresolved discussions (waiting on @jbowens and @xxmplus)

---

options.go line 1381 at r1 (raw file):

Previously, jbowens (Jackson Owens) wrote…

Sorry this is expanding the scope of the change, but I think maybe we should embed an ID field into the wal.Dir. If the user disables WAL failover and passes the directory as a WALRecoveryDir, we can again validate it.

done i think

---

wal/failover_manager.go line 642 at r1 (raw file):

Previously, jbowens (Jackson Owens) wrote…

instead of mutating the Options from inside the WAL manager initialization, maybe we should expose a ValidateOrInitWALDir func from the wal package that we'd use like:

walDir, err = wal.ValidateOrInitWALDir(walDir)

and we could use this during Open to validate and initialize them before initializing the WAL manager; wdyt?

done