Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Universal Deploy #1819

Merged
merged 100 commits into from
Dec 10, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
100 commits
Select commit Hold shift + click to select a range
7d9652c
Still working on it
cicdw Nov 2, 2019
4055987
Remove client method that i will no longer use
cicdw Nov 2, 2019
94c2386
Merge branch 'local-labels' into deploy-2.0
cicdw Nov 3, 2019
3ec48c0
Add working run_agent command for running cloud flows locally
cicdw Nov 3, 2019
8adf268
Add a small docstring about the run_agent method
cicdw Nov 3, 2019
1afc970
Merge branch 'master' into deploy-2.0
cicdw Nov 4, 2019
8dcf31b
Merge branch 'master' into deploy-2.0
cicdw Nov 5, 2019
b82f14a
Begin reworking local agent
cicdw Nov 5, 2019
34e3c0d
Remove unused local code
cicdw Nov 5, 2019
7c8a670
Merge branch 'master' into deploy-2.0
cicdw Nov 7, 2019
fe690e8
Add missed docker files
cicdw Nov 7, 2019
1603a0d
Rename LocalAgent DockerAgent
cicdw Nov 7, 2019
55f8cd8
Rename Local Agent to Docker Agent in docs
joshmeek Nov 8, 2019
46fd6ae
Implement Docker Agent tests and remove all references of runner_toke…
joshmeek Nov 8, 2019
e8f2e9d
Add an untested version of the local agent
cicdw Nov 10, 2019
e21bddb
Add working truly local agent
cicdw Nov 10, 2019
53d4cd2
Always include 'local' label on local agents
cicdw Nov 10, 2019
f545d7c
Reset default logging level for agents
cicdw Nov 10, 2019
87b1b3e
Move local agent logic
cicdw Nov 11, 2019
665b5bd
Update references to old Local Agent in docs
joshmeek Nov 12, 2019
54e829c
Add doc for Local Agent
joshmeek Nov 12, 2019
497c4f8
Update Local Agent link in docstring
joshmeek Nov 12, 2019
52b2d33
Add note to Local Agent doc about multiple labels for local Flows
joshmeek Nov 13, 2019
b7d9dac
Set PYTHONPATH to preserve imports
cicdw Nov 14, 2019
55d9753
Slightly improved labeling
cicdw Nov 14, 2019
658450d
Add import paths keyword to local agent
cicdw Nov 14, 2019
9a54ddb
Add supervisor to requirements
cicdw Nov 14, 2019
b4affa7
Always conert agent labels to list
cicdw Nov 14, 2019
02d4db4
Set flow run state to Failed if deploy error occurs in local agent
cicdw Nov 14, 2019
b7b2686
Version fix
cicdw Nov 14, 2019
823167d
Add supervisor installation
cicdw Nov 15, 2019
cc0a403
Remove API option from CLI install for local agents
cicdw Nov 15, 2019
e67154e
Get local paths working on Mac
cicdw Nov 15, 2019
bbedda4
Fix formatting of cli options
cicdw Nov 17, 2019
c8a2b6e
Resolve merge conflicts with master
cicdw Nov 17, 2019
2e99cb8
Finish resolving merge conflicts
cicdw Nov 17, 2019
06d75d4
Remove unnecessary supervisor comment
cicdw Nov 17, 2019
3ee2de7
Fix agent CLI commands
cicdw Nov 17, 2019
94942da
Update help documentation
cicdw Nov 17, 2019
9096234
Add some more local agent polish
cicdw Nov 17, 2019
7c6ed25
Touch up type hints
cicdw Nov 17, 2019
8533b25
Begin fixing tests
cicdw Nov 17, 2019
7196084
Fix a few more tests
cicdw Nov 17, 2019
b4fcd9a
Stop providing auth token to subprocess environment
cicdw Nov 17, 2019
ac640b2
Change agent start default to local from docker
joshmeek Nov 18, 2019
106dd13
Remove default for agent install CLI command
joshmeek Nov 18, 2019
40fb697
Update CLI docstring
joshmeek Nov 18, 2019
873a178
Merge branch 'master' into deploy-2.1
cicdw Nov 22, 2019
3e8b550
Graceful retrieval of PWD
cicdw Nov 22, 2019
6f0e2a7
Reimplement agent runner token verification
joshmeek Nov 26, 2019
c6fb25f
Reimplement runner token pytest fixture
joshmeek Nov 26, 2019
b25c950
Refactor deploy -> register in core flow
joshmeek Dec 2, 2019
28e4dad
Update references of deploy to register
joshmeek Dec 2, 2019
042171d
Update client docstring
joshmeek Dec 2, 2019
eb6a73f
Add tests for new local agent
joshmeek Dec 2, 2019
1a9a113
Update agent cli tests for local agent
joshmeek Dec 2, 2019
fa6e661
Merge branch 'deploy-2.2' into register
joshmeek Dec 2, 2019
b337177
Update cloud doc structure
joshmeek Dec 2, 2019
9218884
correct agent start docs
Dec 2, 2019
a2b3e23
Add deprecation warning to flow.deploy
joshmeek Dec 3, 2019
2098ddf
Merge pull request #1789 from PrefectHQ/register
joshmeek Dec 3, 2019
a0d6ff1
Merge branch 'deploy-2.2' into deploy_docs
joshmeek Dec 3, 2019
b24238f
add local agent flow logging support
Dec 3, 2019
7ed22af
direct all logging to stdout (not stderr)
Dec 3, 2019
6cd9a38
Added documentation for user upstart with cloud
joshmeek Dec 4, 2019
9ef1837
add local agent heartbeat test
Dec 5, 2019
114f750
honor existing python path + fix pwd assumption
Dec 5, 2019
e79e2d5
install extras for testing in appveyor
Dec 5, 2019
1bef402
Building blocks of deployment documentation
joshmeek Dec 5, 2019
8acbf3d
Rearrange cloud docs and add templates for next steps
joshmeek Dec 8, 2019
95ff47c
Merge branch 'deploy_docs' into deploy-2.2
joshmeek Dec 8, 2019
dacb5a2
Added list of onboard docs to first Cloud page
joshmeek Dec 9, 2019
5446aee
Tweak documentation wording
zdhughes Dec 9, 2019
65e87ea
Add Cloud Tutorial docs
joshmeek Dec 10, 2019
131ae91
Resolve merge conflicts with master (I think)
cicdw Dec 10, 2019
591eef7
Tidy up so that docs can be served
cicdw Dec 10, 2019
f4a7bca
Add auto-labeling of flow environments based on storage type
cicdw Dec 10, 2019
9e26414
Add show_flow_logs to doc strings
cicdw Dec 10, 2019
1d784cd
Make auto-labels more specific
cicdw Dec 10, 2019
ec42cf4
Auto label the local agent with the new storage labels
cicdw Dec 10, 2019
8089f29
Fix tests caused by new autolabels on local agent
cicdw Dec 10, 2019
4e25362
Merge pull request #1817 from PrefectHQ/storage-auto-labels
cicdw Dec 10, 2019
ab1f3f1
Merge branch 'master' into deploy-2.2
cicdw Dec 10, 2019
866c5da
Skip graphviz test which fails on appveyor
cicdw Dec 10, 2019
8aec281
Merge branch 'master' into deploy-2.2
cicdw Dec 10, 2019
bbcdf1d
Move skipifs down to the execution submodule of tests
cicdw Dec 10, 2019
98a632c
Merge branch 'deploy-2.2' of https://github.com/PrefectHQ/prefect int…
cicdw Dec 10, 2019
35f0cc0
Import pytest
cicdw Dec 10, 2019
c308899
Remove unnecessary import of google cloud storage
cicdw Dec 10, 2019
4eaafee
Fix failing windows tests
cicdw Dec 10, 2019
e14bab7
Use pytestmark
cicdw Dec 10, 2019
83c9548
Remove path validation at deserialiation time for local storage
cicdw Dec 10, 2019
dda3513
Ensure agent labels are always a list
cicdw Dec 10, 2019
a477f1d
Add new storage classes to API documentation
cicdw Dec 10, 2019
89e2358
add show_flow_logs to run_agent fn
Dec 10, 2019
cbd71f0
Update CHANGELOG
joshmeek Dec 10, 2019
4233aef
Resolve merge conflicts
joshmeek Dec 10, 2019
9b96654
Update local agent doc to include new default labels
joshmeek Dec 10, 2019
5cf1238
Update storage options doc with notes about auto labels
joshmeek Dec 10, 2019
d0f3d39
Update storage class docstrings with notes about auto labels
joshmeek Dec 10, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Begin reworking local agent
  • Loading branch information
cicdw committed Nov 5, 2019
commit b82f14a61e63683989b76b822ae80a425f3d1cb1
1 change: 1 addition & 0 deletions src/prefect/agent/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
# others must be explicitly imported so they can raise helpful errors if appropriate

from prefect.agent.agent import Agent
import prefect.agent.docker
import prefect.agent.fargate
import prefect.agent.local
import prefect.agent.kubernetes
Expand Down
22 changes: 0 additions & 22 deletions src/prefect/agent/agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,6 @@ def __init__(self, name: str = None, labels: Iterable[str] = None) -> None:
token = config.cloud.agent.get("auth_token")

self.client = Client(api_token=token)
self._verify_token(token)

logger = logging.getLogger(self.name)
logger.setLevel(config.cloud.agent.get("level"))
Expand All @@ -63,27 +62,6 @@ def __init__(self, name: str = None, labels: Iterable[str] = None) -> None:

self.logger = logger

def _verify_token(self, token: str) -> None:
"""
Checks whether a token with a `RUNNER` scope was provided

Args:
- token (str): The provided agent token to verify

Raises:
- AuthorizationError: if token is empty or does not have a RUNNER role
"""
if not token:
raise AuthorizationError("No agent API token provided.")

# Check if RUNNER role
result = self.client.graphql(query="query { authInfo { apiTokenScope } }")
if (
not result.data # type: ignore
or result.data.authInfo.apiTokenScope != "RUNNER" # type: ignore
):
raise AuthorizationError("Provided token does not have a RUNNER scope.")

def start(self) -> None:
"""
The main entrypoint to the agent. This function loops and constantly polls for
Expand Down
65 changes: 2 additions & 63 deletions src/prefect/agent/local/agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,58 +12,19 @@

class LocalAgent(Agent):
"""
Agent which deploys flow runs locally as Docker containers. Information on using the
Agent which deploys flow runs locally. Information on using the
Local Agent can be found at https://docs.prefect.io/cloud/agent/local.html

Args:
- name (str, optional): An optional name to give this agent. Can also be set through
the environment variable `PREFECT__CLOUD__AGENT__NAME`. Defaults to "agent"
- labels (List[str], optional): a list of labels, which are arbitrary string identifiers used by Prefect
Agents when polling for work
- base_url (str, optional): URL for a Docker daemon server. Defaults to
`unix:///var/run/docker.sock` however other hosts such as
`tcp://0.0.0.0:2375` can be provided
- no_pull (bool, optional): Flag on whether or not to pull flow images.
Defaults to `False` if not provided here or in context.
"""

def __init__(
self,
name: str = None,
labels: Iterable[str] = None,
base_url: str = None,
no_pull: bool = None,
) -> None:
def __init__(self, name: str = None, labels: Iterable[str] = None,) -> None:
super().__init__(name=name, labels=labels)

if platform == "win32":
default_url = "npipe:////./pipe/docker_engine"
else:
default_url = "unix://var/run/docker.sock"
self.logger.debug(
"Platform {} and default docker daemon {}".format(platform, default_url)
)

# Determine Daemon URL
self.base_url = base_url or context.get("base_url", default_url)
self.logger.debug("Base docker daemon url {}".format(self.base_url))

# Determine pull specification
self.no_pull = no_pull or context.get("no_pull", False)
self.logger.debug("no_pull set to {}".format(self.no_pull))

self.docker_client = docker.APIClient(base_url=self.base_url, version="auto")

# Ping Docker daemon for connection issues
try:
self.logger.debug("Pinging docker daemon")
self.docker_client.ping()
except Exception as exc:
self.logger.exception(
"Issue connecting to the Docker daemon. Make sure it is running."
)
raise exc

def deploy_flows(self, flow_runs: list) -> None:
"""
Deploy flow runs on your local machine as Docker containers
Expand Down Expand Up @@ -111,28 +72,6 @@ def deploy_flows(self, flow_runs: list) -> None:
)
self.docker_client.start(container=container.get("Id"))

def populate_env_vars(self, flow_run: GraphQLResult) -> dict:
"""
Populate metadata and variables in the environment variables for a flow run

Args:
- flow_run (GraphQLResult): A flow run object

Returns:
- dict: a dictionary representing the populated environment variables
"""
return {
"PREFECT__CLOUD__API": config.cloud.api,
"PREFECT__CLOUD__AUTH_TOKEN": config.cloud.agent.auth_token,
"PREFECT__CLOUD__AGENT__LABELS": str(self.labels),
"PREFECT__CONTEXT__FLOW_RUN_ID": flow_run.id, # type: ignore
"PREFECT__CLOUD__USE_LOCAL_SECRETS": "false",
"PREFECT__LOGGING__LOG_TO_CLOUD": "true",
"PREFECT__LOGGING__LEVEL": "DEBUG",
"PREFECT__ENGINE__FLOW_RUNNER__DEFAULT_CLASS": "prefect.engine.cloud.CloudFlowRunner",
"PREFECT__ENGINE__TASK_RUNNER__DEFAULT_CLASS": "prefect.engine.cloud.CloudTaskRunner",
}


if __name__ == "__main__":
LocalAgent().start()
6 changes: 4 additions & 2 deletions src/prefect/core/flow.py
Original file line number Diff line number Diff line change
Expand Up @@ -1233,7 +1233,10 @@ def run_agent(self, executor=None, labels: list = None):
"local" along with a safe version of the current Flow's name
"""
labels = labels or ["local", slugify(self.name)]
agent = prefect.agent.agent.Agent(labels=labels)
with set_temporary_config(
{"cloud.agent.auth_token": prefect.config.cloud.auth_token}
):
agent = prefect.agent.agent.Agent(labels=labels)

while True:
flow_runs = agent.query_flow_runs(str(uuid.uuid4()))
Expand All @@ -1245,7 +1248,6 @@ def run_agent(self, executor=None, labels: list = None):
{
"cloud.use_local_secrets": False,
"logging.log_to_cloud": True,
"cloud.auth_token": agent.client._api_token,
}
):
runner_cls = prefect.engine.cloud.flow_runner.CloudFlowRunner
Expand Down