release-24.1: sql/schemachanger: avoid retries on invalid expressions during backfills

[blathers-crl]

Backport 1/1 commits from #142490 on behalf of @fqazi.

---

Currently, the schema changer uses a white list to determine which errors to be retired, which in some cases use a text comparison. Unfortunately, when we parse expressions their text can appear in the error and cause errors to be erroneously retried for backfills. To address this, this patch explicitly wraps errors from eval.Expr with SchemaChangerUserError to block retries.

Fixes: #141352

Release note (bug fix): Invalid default expressions could cause backfilling schema changes to retry forever

---

Release justification: bug fix.

[blathers-crl]

Thanks for opening a backport.

Please check the backport criteria before merging:

• Backports should only be created for serious
  issues or test-only changes.
• Backports should not break backwards-compatibility.
• Backports should change as little code as possible.
• Backports should not change on-disk formats or node communication protocols.
• Backports should not add new functionality (except as defined
  here).
• Backports must not add, edit, or otherwise modify cluster versions; or add version gates.
• All backports must be reviewed by the owning areas TL. For more information as to how that review should be conducted, please consult the backport
  policy.

If your backport adds new functionality, please ensure that the following additional criteria are satisfied:

• There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
• The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
• New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters. State changes must be further protected such that nodes running old binaries will not be negatively impacted by the new state (with a mixed version test added).
• The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.
• Your backport must be accompanied by a post to the appropriate Slack
  channel (#db-backports-point-releases or #db-backports-XX-X-release) for awareness and discussion.

Also, please add a brief release justification to the body of your PR to justify this
backport.

[cockroach-teamcity]

This change is 

[github-actions]

Reminder: it has been 3 weeks please merge or close your backport!

[yuzefovich]

@rafiss @fqazi friendly ping on getting these backports merged if we think we should

[fqazi]

@yuzefovich Let me dig into the failures here to get these stable

[blathers-crl]

✅ PR #147016 is compliant with backport policy

Confidence: high
Critical bug criteria met: [Bugs that can cause the DB to return incorrect results or result in suboptimal performance]
Backward compatible: true
Explanation: The pull request is compliant with the backport policy for several reasons. Firstly, the PR includes a 'Release justification: bug fix.' in the body which is an acceptable reason under the backport policy exceptions. Secondly, the changes address a critical bug as it fixes a bug that causes incorrect results or suboptimal performance, specifically by preventing backfilling schema changes from retrying indefinitely due to invalid default expressions. Additionally, the files changed do not include any that would break backward compatibility, nor do they remove any version gates. All changes are within production files, and their modification directly addresses the stated critical bug without introducing new features or behaviors that would require gating by a feature flag.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[github-actions]

Reminder: it has been 3 weeks please merge or close your backport!