refactor: introduce structured error system w/ host exception tunneling

[Haleshot]

Addresses #1377 by introducing a structured error system that supports host exception tunneling. The core CError enum has four variants: Context for wrapping errors with messages (replacing anyhow::Context), HostLang for tunneling host exceptions via Box<dyn HostError>, Client for validation/4xx errors, and Internal for 5xx errors; both of which I believe, capture the backtraces via/ std::backtrace::Backtrace.

On the Python side, PyErrWrapper implements HostError and cerror_to_pyerr() handles the reverse conversion, traversing the context chain and downcasting via Any to extract the original PyErr if present.

Following your suggestion about incremental/iteratively working within the same PR, I'm pushing this first for your review. If this looks good, the next step is migrating the a lot of anyhow::Result usages across the codebase to use CResult. The existing anyhow-based types should/will be preserved for backwards compatibility during that migration.

Edit 1: Tended the changes requested here and here; can proceed with the migration of anyhow::Result usages if it looks good!

Edit 2: Completed the migration of error macros from anyhow::bail!/anyhow::anyhow! to the new structured client_bail!/client_error! and internal_bail!/internal_error! macros across ~35 files.

The new macros create CError with Client/Internal variants that capture backtraces, then convert to anyhow::Error for compatibility with existing return types. This preserves all the structured error semantics (including host exception tunneling for Python) while avoiding a massive signature refactor. The anyhow::Result type remains as-is...

PS: I took, what I believe is a pragmatic approach here - rather than replacing every anyhow::Result<T> with CResult<T> (which would've required touching hundreds of serde impls and function signatures), I wrapped CError inside anyhow::Error. Is this acceptable, or would you prefer a full CResult<T> migration in a follow-up?

Full disclosure: Claude (via Cursor) helped me grep through the (relevant files in the) codebase to identify all the error macro usages. Turns out clicking through search results in an IDE beats wrestling w/ git grepping 🙃

[georgeh0]

@Haleshot Really fast! Looks very reasonable! Thanks!

[georgeh0]

The API looks good! Left a few comments for implementations. Feel free to start migrating existing code to the new error type.

[Haleshot]

Updated the PR description above with "Edits" to reflect the changes made incrementally; awaiting your review :)

[georgeh0]

seems this PyErrWrapper doesn't provide additional values. We may not need this layer.

[Haleshot]

I see it asa thin wrapper, but PyErr doesn't impl std::error::Error which HostLang(Box<dyn HostError>) needs (some bridging is needed in this case). Were you thinking of storing PyErr directly in a dedicated variant (CError::PyErr(PyErr) variant instead of using trait object?), or relaxing the trait bounds? Happy to go either direction.

[georgeh0]

We still want to keep the generic HostLang(Box<dyn HostError>) for future proof.

Looks like PyErr actually implemented std::error::Error?
https://docs.rs/pyo3/latest/pyo3/struct.PyErr.html#impl-Error-for-PyErr

[Haleshot]

Ahh, yes 🙈- PyErr does impl Error, thanks for the relevant docs webpage. Will remove PyErrWrapper and use PyErr directly with CError::host(err).

[georgeh0]

I think for client error, context is also valuable. Probably the only difference is backtrace is not necessary.

[georgeh0]

Actually both is_host_error() and is_client_error() are redundant (they're not used by any code any way, other than code to test themselves). Using without_contexts() can easily achieve the same goal. We don't need to create a tester for each error type.

Similar to find_host_error(). Callers can use without_contexts() + matcher directly too (and they need a mater any way for the Option).

[georgeh0]

About

PS: I took, what I believe is a pragmatic approach here - rather than replacing every anyhow::Result<T> with CResult<T> (which would've required touching hundreds of serde impls and function signatures), I wrapped CError inside anyhow::Error. Is this acceptable, or would you prefer a full CResult<T> migration in a follow-up?

There're two parts:

1. Error creations. e.g. serde related stuffs. Calling on the existing bail! and api_bail! macros also belong to this.
2. Error propagation. e.g. popping up by ?, with additional contexts, etc.

For 1, I think it's it's OK to handle different parts separately. The final goal is not to use anyhow, but it can happen in multiple steps.

For 2, the problem of using anyhow::Error as wrapper is: the structured information will be lost during propagation (it allows downcast to restore the original structured information, but after calling context(), downcast can no longer restore to the original structured error).

To make the migration easier, I have another idea. A lot of existing crates just use Error as name of their error type, and they also define a specialized Result type. For example, for sqlx (Error, Result). We can do the similar thing:

• Rename CError toError, and add a specialized Result type.
• Update this line in prelude to use our new types instead.
• One minor thing to keep in mind is that we use anyhow::Ok(...) sometimes (which is a syntax sugar, equivalent to Ok::<_, anyhow::Error>(...). They can be avoided by explicitly annotate return type values (example).

In this way, most modules won't need to be touched, if they're just propagating errors rather than creating errors.

What do you think?

[Haleshot]

On the approach suggested above: I'll go about doing the following:

• Rename CError → Error, CResult → Result in cocoindex_utils
• Update prelude.rs to use these instead of anyhow

Also, hope it's fine for putting this (& you) through multiple review rounds 😅 - definitely not intentional (hope you don't mind)! You're also more than welcome to push directly to this branch if you feel I'm not getting it (still making my way about Rust and the codebase.

[georgeh0]

On the approach suggested above: I'll go about doing the following:

• Rename CError → Error, CResult → Result in cocoindex_utils
• Update prelude.rs to use these instead of anyhow

Also, hope it's fine for putting this (& you) through multiple review rounds 😅 - definitely not intentional (hope you don't mind)! You're also more than welcome to push directly to this branch if you feel I'm not getting it (still making my way about Rust and the codebase.

Looks good! No worries, this is a large refactor and it's well expected to have multiple rounds of review :) Thanks a lot for pushing it through!