My Hybrid ECS agent is having trouble connecting to the code server tasks that it creates. How can I troubleshoot/configure my cluster so that it will work?

[gibsondan]

I'm seeing an error like this when my Hybrid ECS agent tries to spin up a code server task:

Most recent connection error: dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE

Stack Trace:
  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1710, in _wait_for_server_process
    client.ping("")
  File "/dagster/dagster/_grpc/client.py", line 200, in ping
    res = self._query("Ping", api_pb2.PingRequest, echo=echo)
  File "/dagster/dagster/_grpc/client.py", line 167, in _query
    self._raise_grpc_exception(
  File "/dagster/dagster/_grpc/client.py", line 150, in _raise_grpc_exception
    raise DagsterUserCodeUnreachableError(

The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.0.1.102:4000: Failed to connect to remote host: FD Shutdown"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:10.0.1.102:4000: Failed to connect to remote host: FD Shutdown {created_time:"2023-10-20T21:11:55.12632368+00:00", grpc_status:14}"

OR my hybrid ECS agent is not showing up on the Agents tab in Dagster Cloud. How can I fix it?

[gibsondan]

If you're using the Dagster Cloud Cloudformation template that creates a brand new VPC for you, it should be created with the correct networking permissions. This is what we recommend first if you're just starting out with Dagster Cloud.

If you're setting up a Dagster Cloud agent in an existing VPC, the agent needs to be able to egress to the Dagster Cloud servers, and needs to be able to communicate with any ECS services that it creates to run your code. Here's a list of requirements and how you can ensure that your cluster meets them:

• The VPC needs to use route53 for DNS
  • You can verify this by looking at the DHCP option set on the VPC
• The VPC needs to have assign_hostnames enabled
• The “default” security group in the VPC needs the following rules
  • An ingress rule that allows traffic from other addresses within the default security group. this allows the agent and code server to communicate with each other
  • Open egress from addresses in the Security Group to the internet. This allows the agent to communicate with Dagster Cloud
• (if using private subnets) The network ACL should allow the same rules as the security group, egress to the public internet and ingress from other hosts in the private subnet

How to check things:

• For the VPC DNS you can go to the VPC console, find your VPC, and click on the DHCP option set
• For the security group go to the security groups section in the VPC console, filter for your VPC and find the one named “default”
• For the network ACLs you’ll first need to find the subnet which you can also find from the VPC console and click on the tab for network ACLs