Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run JobCoontroller as separate application #951

Merged
merged 26 commits into from
Aug 21, 2020

Conversation

pyalex
Copy link
Collaborator

@pyalex pyalex commented Aug 19, 2020

What this PR does / why we need it:

This is final PR in series of decoupling JobCoordinator (renamed to JobController) from Core.
Main features of PR:

  1. There's new module job-controller that should be run as standalone (spring) application (helm charts, docker compose, e2e tests updated)
  2. This module communicates with Core through GRPC API. Although job-controller use core in test scope.
  3. Ingestion related methods were moved out of CoreService API into JobController API (see CoreService.proto). Since it could be deployed with several instances of JobController - routing between them is now client's responsibility. Also, client code (in python SDK) calling those methods were moved out of Feast Client. Now it's in contrib (see feast.contrib.job_controller.client in python SDK).

Refactoring:

  1. Common code for integration tests moved to feast-common-test module

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

Dedicated spring application job-controller
feast.jobs properties moved from core to job-controller

[Deprecated] CoreService.ListIngestJobs, CoreService.RestartIngestJob, CoreService.StopIngestJob

[New API] JobControllerService.ListIngestJobs, JobControllerService.RestartIngestJob, JobControllerService.ListIngestJobs.StopIngestJob

[Python SDK] feast.client.Client.list_ingest_jobs -> feast.contrib.job_controller.client.Client.list_ingest_jobs
[Python SDK] feast.client.Client.restart_ingest_job -> feast.contrib.job_controller.client.Client.restart_ingest_job
[Python SDK] feast.client.Client.stop_ingest_job -> feast.contrib.job_controller.client.Client.stop_ingest_job

fix simple client
@pyalex
Copy link
Collaborator Author

pyalex commented Aug 19, 2020

/test test-end-to-end-batch-dataflow

@pyalex
Copy link
Collaborator Author

pyalex commented Aug 20, 2020

/test test-end-to-end-redis-cluster

@pyalex
Copy link
Collaborator Author

pyalex commented Aug 20, 2020

/test test-end-to-end

@pyalex
Copy link
Collaborator Author

pyalex commented Aug 20, 2020

/test test-end-to-end-redis-cluster

@pyalex
Copy link
Collaborator Author

pyalex commented Aug 20, 2020

/test test-end-to-end-redis-cluster

@pyalex
Copy link
Collaborator Author

pyalex commented Aug 20, 2020

/test test-end-to-end-auth
/test test-end-to-end

@pyalex pyalex requested a review from woop August 20, 2020 10:20
@pyalex
Copy link
Collaborator Author

pyalex commented Aug 20, 2020

/test test-end-to-end-auth

@pyalex
Copy link
Collaborator Author

pyalex commented Aug 21, 2020

/test test-end-to-end-batch

1 similar comment
@pyalex
Copy link
Collaborator Author

pyalex commented Aug 21, 2020

/test test-end-to-end-batch

@woop
Copy link
Member

woop commented Aug 21, 2020

/lgtm

@pyalex
Copy link
Collaborator Author

pyalex commented Aug 21, 2020

/test test-end-to-end-batch

3 similar comments
@pyalex
Copy link
Collaborator Author

pyalex commented Aug 21, 2020

/test test-end-to-end-batch

@pyalex
Copy link
Collaborator Author

pyalex commented Aug 21, 2020

/test test-end-to-end-batch

@pyalex
Copy link
Collaborator Author

pyalex commented Aug 21, 2020

/test test-end-to-end-batch

@feast-ci-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pyalex, woop

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@feast-ci-bot feast-ci-bot merged commit 16bb358 into feast-dev:master Aug 21, 2020
@mrzzy mrzzy added the compat/breaking Breaking user-facing change label Sep 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants