Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a new binary crate 'arroyo-bin' to package the entire system #514

Merged
merged 3 commits into from
Jan 31, 2024

Conversation

mwylde
Copy link
Member

@mwylde mwylde commented Jan 29, 2024

This PR adds a new crate, "arroyo-bin" that packages the entire Arroyo system (api/controller/worker). This will allow for deployment via a single, self-contained binary.

$ arroyo-bin
Usage: arroyo-bin <COMMAND>

Commands:
  api         Starts an Arroyo API server
  controller  Starts an Arroyo Controller
  cluster     Starts a complete Arroyo cluster
  worker      Starts an Arroyo worker
  help        Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version

This affects the dev process a bit:

  • Instead of running the api and the controller, run arroyo-bin (the cluster option, which will start both controller and api, or just one of the services)
  • The worker also now runs from this binary, so there's no need to separately build the worker any more. For the process scheduler, the controller will use its own binary for the worker. This means that a controller compiled as debug mode will use the debug worker, and likewise for release.

As part of this work I've introduced a more sophisticated shutdown system (in arroyo-server-common/src/shutdown.rs) which handles the two phases of clean shutdowns in an async application:

  1. Detecting that the system should shut down, due to either a signal or because a subcomponent shut down or crashed)
  2. Notifying the whole tree of tasks to stop via async cancellation, then waiting for all tasks to stop

I've also fixed and re-enabled the integration tests.

Copy link
Contributor

@jacksonrnewhouse jacksonrnewhouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to update docs?

Cargo.toml Outdated
@@ -21,7 +22,7 @@ members = [
"arroyo-types",
"arroyo-worker",
"copy-artifacts",
"integ",
"integ", "arroyo-bin",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Put this on its own line?

@mwylde
Copy link
Member Author

mwylde commented Jan 31, 2024

Yes, we'll need to update dev docs. I think it would make sense to do that when we're ready to merge into master.

@mwylde mwylde enabled auto-merge (squash) January 31, 2024 20:25
@mwylde mwylde merged commit b6928e9 into dev Jan 31, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants