Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR redesigns the connector system and migrates (almost all) connectors into the arrow world
Operator/connector redesign
Currently, all operator infra and the operators themselves are implemented in arroyo-worker. Meanwhile connector definitions, configs, and testing logic (which often duplicates code in the connector implementation) is in arroyo-connectors.
To instantiate operators/connectors, we have a giant match blog in engine.rs.
This was (sort of) necessary back in the macro days, but with traits we can be more flexible.
There are a few things that are undesirable about this situation:
To address these issues this PR reorganizes the code as follows:
Longer term, we can look into using a crate like https://github.com/dtolnay/linkme to allow connectors to be registered at compile time, making it easy for advanced users to add custom connectors just be adding a new crate to the build.
Connector migrations
All connectors have now been migrated into arrow with the exception of the FileSystem/Delta sink and nexmark. These were excluded as they require significant work to migrate.