Problems using AWS IoT Device Client with Greengrass V2

[feborja]

Just like another issue in 2021, I am also trying to enable secure tunnelling in a Device using Greengrass v2, right now I only added the component in the deployment and declared the OS as "raspberry", yet I get errors with the settings with the certificates (Permissions not set to desired value) as well as an error that seems like a placeholder is not edited for the client:

2024-07-12T17:44:05.165Z [WARN] {Config.cpp}: Path replace_with_root_ca_file_location to RootCA is invalid. Ignoring... Will attempt to use default trust store.. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}

The documentation for the component does not specify anything aditional step therefore I am not sure if I am doing something wrong, any help is appreciated.

OS: Raspbian GNU/Linux 11 (bullseye)
Tunnelling component version: 1.0.19
Nucleus Version: 2.12.6

The only configuration merged into the recipe is: "OS_DIST_INFO": "raspberry"

[HarshGandhi-AWS]

Hey @feborja , can you share your setup steps? Just by reading the issue description, I think you might not be setting up the device correctly. Can you share all details and steps to reproduce the issue?

[feborja]

@HarshGandhi-AWS Of course, so as context this is a device that was created beforehand and contains another components, so it has already been deployed before, I had worked with the client with Greengrass V1 before following the steps in the readme from this repository, but I found that Greengass v2 added this as a component. So the only thing I did was revise the existing deployment, add the component, merge the configuration "OS_DIST_INFO": "raspberry" and deploy.
Now, the policies the device has now are:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"iot:Connect",
"iot:Publish",
"iot:Subscribe",
"iot:Receive",
"greengrass:"
],
"Resource": ""
}
]
}
and

{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Action": "iot:AssumeRoleWithCertificate",
"Resource": "arn:aws:iot:us-east-1:979146024221:rolealias/GreengrassV2TokenExchangeRoleAlias"
}
}

I am not sure if this is the information you needed, please let me know

[HarshGandhi-AWS]

Just want to make sure, (a) you are using the Greengrass v2 on your device and (b) is your device is not able to connect to secure tunnel in device mode? Is that the case? From the logs you shared it shows the secure tunnel component is running and did not crash. Are you seeing any issues connecting to the end device via secure tunnel?

2024-07-12T17:44:05.165Z [WARN]  {Config.cpp}:  ... serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}

[feborja]

a) Yes, I am using Greengrass V2
b) I am not able to connect to secure tunnelling, when I create a tunnel the following logs appear:

2024-07-12T21:05:09.325Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [INFO ] 2024-07-12 17:05:09.324 [AwsEventLoop 1] SubscribeResponseHandler - Received new tunnel notification message.. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2024-07-12T21:05:09.395Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2024-07-12 17:05:09.392 [pool-3-thread-2] SubscribeResponseHandler - Secure Tunneling Process: 2024-07-12T21:05:09.391Z [WARN]  {FileUtils.cpp}: Permissions to given file/dir path '/tmp/' is not set to recommended value... {Permissions: {desired: 745, actual: 777}}. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2024-07-12T21:05:09.397Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2024-07-12 17:05:09.396 [pool-3-thread-2] SubscribeResponseHandler - Secure Tunneling Process: 2024-07-12T21:05:09.391Z [WARN]  {FileUtils.cpp}: Permissions to given file/dir path '/tmp/device-client-settings.json97283231374049021111720810127642' is not set to recommended value... {Permissions: {desired: 640, actual: 644}}. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2024-07-12T21:05:09.399Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2024-07-12 17:05:09.398 [pool-3-thread-2] SubscribeResponseHandler - Secure Tunneling Process: 2024-07-12T21:05:09.391Z [WARN]  {Config.cpp}: Path replace_with_root_ca_file_location to RootCA is invalid. Ignoring... Will attempt to use default trust store.. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2024-07-12T21:05:26.999Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2024-07-12 17:05:26.998 [pool-3-thread-2] SubscribeResponseHandler - Secure Tunneling Process: 2024-07-12T21:05:26.998Z [ERROR] {TcpForward.cpp}: TcpForward::OnConnectionResult error_code=1047. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}

After that nothing else happens (when I try to connect it never does, it is loading forever), and if I generate new access tokens, the same thing happens again
Sorry for the painful format, my console shows it like that.

[feborja]

@HarshGandhi-AWS Tried with local proxy with docker, same type of error

2024-07-15T19:50:49.733Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2024-07-15 15:50:49.729 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: 2024-07-15T19:50:49.675Z [WARN]  {FileUtils.cpp}: Permissions to given file/dir path '/tmp/' is not set to recommended value... {Permissions: {desired: 745, actual: 777}}. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2024-07-15T19:50:49.735Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2024-07-15 15:50:49.731 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: 2024-07-15T19:50:49.680Z [WARN]  {FileUtils.cpp}: Permissions to given file/dir path '/tmp/device-client-settings.json79754660964669720811721073030236' is not set to recommended value... {Permissions: {desired: 640, actual: 644}}. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2024-07-15T19:50:49.736Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2024-07-15 15:50:49.732 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: 2024-07-15T19:50:49.681Z [WARN]  {Config.cpp}: Path replace_with_root_ca_file_location to RootCA is invalid. Ignoring... Will attempt to use default trust store.. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2024-07-15T19:53:49.385Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2024-07-15 15:53:49.383 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: 2024-07-15T19:53:49.383Z [ERROR] {TcpForward.cpp}: TcpForward::OnConnectionResult error_code=1047. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}

[RogerZhongAWS]

If you are using the latest version of localproxy to connect to device client can you try passing in destination-client-type V1 as an arg to your localproxy run command? This is a new requirement from v3.1.2 onwards

[feborja]

I get the same error, the localproxy shows:

[2024-07-24 05:36:05.925914] (0x00007f8e57eda840) [warning] Found access token supplied via CLI arg. Consider using environment variable AWSIOT_TUNNEL_ACCESS_TOKEN instead
[2024-07-24 05:36:05.927014] (0x00007f8e57eda840) [info] Starting proxy in source mode
[2024-07-24 05:36:05.939698] (0x00007f8e57eda840) [info] Attempting to establish web socket connection with endpoint wss://data.tunneling.iot.us-east-1.amazonaws.com:443
[2024-07-24 05:36:06.525095] (0x00007f8e57eda840) [info] Web socket session ID: 12232afffebbd6c9-000014d0-00008876-2b86524f69c959b5-0f1cf446
[2024-07-24 05:36:06.525205] (0x00007f8e57eda840) [info] Successfully established websocket connection with proxy server: wss://data.tunneling.iot.us-east-1.amazonaws.com:443
[2024-07-24 05:36:06.526631] (0x00007f8e57eda840) [info] Updated port mapping for v1 format:
[2024-07-24 05:36:06.526710] (0x00007f8e57eda840) [info] SSH = 23410
[2024-07-24 05:36:06.526726] (0x00007f8e57eda840) [info] calling setup from loop
[2024-07-24 05:36:06.527100] (0x00007f8e57eda840) [info] Listening for new connection on port 23410
[2024-07-24 05:37:45.689077] (0x00007f8e57eda840) [info] creating tcp connection id 1
[2024-07-24 05:37:45.689288] (0x00007f8e57eda840) [info] Accepted tcp connection on port 23410 from [::1]:45398

and my device shows the same logs as before:

2024-07-24T05:34:25.600Z [INFO] (pool-3-thread-50) aws.greengrass.SecureTunneling: shell-runner-start. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=STARTING, command=["java -jar /greengrass/v2/packages/artifacts/aws.greengrass.SecureTunneling/1.0..."]}
2024-07-24T05:34:28.406Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [INFO ] 2024-07-24 05:34:28.398 [main] SecureTunneling - Starting secure tunneling component!. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2024-07-24T05:34:29.003Z [WARN] (Copier) aws.greengrass.SecureTunneling: stderr. Jul 24, 2024 5:34:28 AM software.amazon.awssdk.eventstreamrpc.EventStreamRPCConnection$1 onConnectionSetup. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2024-07-24T05:34:29.004Z [WARN] (Copier) aws.greengrass.SecureTunneling: stderr. INFO: Socket connection /greengrass/v2/ipc.socket:8033 to server result [AWS_ERROR_SUCCESS]. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2024-07-24T05:34:29.165Z [WARN] (Copier) aws.greengrass.SecureTunneling: stderr. Jul 24, 2024 5:34:29 AM software.amazon.awssdk.eventstreamrpc.EventStreamRPCConnection$1 onProtocolMessage. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2024-07-24T05:34:29.165Z [WARN] (Copier) aws.greengrass.SecureTunneling: stderr. INFO: Connection established with event stream RPC server. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2024-07-24T05:34:29.173Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [INFO ] 2024-07-24 05:34:29.172 [main] SecureTunnelingExecutor - Starting secure tunneling.. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2024-07-24T05:34:29.563Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [INFO ] 2024-07-24 05:34:29.561 [AwsEventLoop 1] SecureTunnelingTask - Successfully subscribed to topic: $aws/things/10000000e0570554/tunnels/notify. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2024-07-24T05:35:38.310Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [INFO ] 2024-07-24 05:35:38.309 [AwsEventLoop 1] SubscribeResponseHandler - Received new tunnel notification message.. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2024-07-24T05:35:38.379Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2024-07-24 05:35:38.378 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: 2024-07-24T05:35:38.332Z [WARN]  {FileUtils.cpp}: Permissions to given file/dir path '/tmp/' is not set to recommended value... {Permissions: {desired: 745, actual: 777}}. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2024-07-24T05:35:38.380Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2024-07-24 05:35:38.379 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: 2024-07-24T05:35:38.332Z [WARN]  {FileUtils.cpp}: Permissions to given file/dir path '/tmp/device-client-settings.json5519849066001650531721799268640' is not set to recommended value... {Permissions: {desired: 640, actual: 664}}. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2024-07-24T05:35:38.381Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2024-07-24 05:35:38.380 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: 2024-07-24T05:35:38.333Z [WARN]  {Config.cpp}: Path replace_with_root_ca_file_location to RootCA is invalid. Ignoring... Will attempt to use default trust store.. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2024-07-24T05:37:46.510Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2024-07-24 05:37:46.510 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: 2024-07-24T05:37:46.509Z [ERROR] {TcpForward.cpp}: TcpForward::OnConnectionResult error_code=1047. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}

I also checked if I the device has port 443 open, which it does:

root@<user>:/greengrass/v2/logs# sudo lsof -i :443
COMMAND    PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
aws-iot-d 1623 ggc_user    7u  IPv4  31670      0t0  TCP <user>:59562->ec2-3-213-214-212.compute-1.amazonaws.com:https (ESTABLISHED)