Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add config validation utility to Fargate Agent #2768

Merged
merged 7 commits into from
Jun 15, 2020
Merged

Conversation

joshmeek
Copy link

@joshmeek joshmeek commented Jun 11, 2020

Thanks for contributing to Prefect!

Please describe your work and make sure your PR:

  • adds new tests (if appropriate)
  • add a changelog entry in the changes/ directory (if appropriate)
  • updates docstrings for any new functions or function arguments, including docs/outline.toml for API reference docs (if appropriate)

Note that your PR will not be reviewed unless all three boxes are checked.

What does this PR change?

This PR adds a function to the fargate agent for validating the configuration that it was provided. For example something like this could happen:

agent = FargateAgent(
    aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
    aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
    region_name="us-east-1",
    cpu="1024",
    memory="8192",
    executionRoleArn="my_arn...",
    containerDefinitions=[
        {
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-region": "us-east-1",
                    "awslogs-group": "my_group",
                    "awslogs-stream-prefix": "prefect",
                },
            },
        }
    ],
    networkConfiguration={
        "awsvpcConfiguration": {
            "assignPublicIp": "ENABLED",
            "subnets": ["subnet-0773697d1a87dd105"],
            "securityGroups": [],
        }
    },
)

agent.validate_configuration()

And this validate_configuration function will test to see if the agent has the proper setup to register a task definition and run the task.

[2020-06-11 20:01:12,321] INFO - agent | Testing task definition registration...
[2020-06-11 20:01:12,499] INFO - agent | Task definition registration successful.
[2020-06-11 20:01:12,499] INFO - agent | Testing task run...
[2020-06-11 20:01:13,409] INFO - agent | Task run ...my_task_run... successful.

Why is this PR important?

It is not uncommon for questions around the fargate agent to be based around configuration and usually these are only discovered after getting it up and running. This can hopefully alleviate some issues before attempting to run a flow by validating ahead of time.

I'm also now thinking that maybe every agent should have its own validate_configuration to verify that it is configured properly.

@codecov
Copy link

codecov bot commented Jun 11, 2020

Codecov Report

Merging #2768 into master will increase coverage by 0.01%.
The diff coverage is 100.00%.

@joshmeek joshmeek merged commit 737b01d into master Jun 15, 2020
@joshmeek joshmeek deleted the fargate_enhancements branch June 15, 2020 12:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants