(greengrass): Greengrass cannot report the job status back to the cloud

[DeboBurro]

Describe the bug
After provisioning a greengrass device, I saw the /greengrass/v2/logs/greengrass.log

To Reproduce

1. Install greengrass via

sudo -E java -Droot="/greengrass/v2" -Dlog.store=FILE \
  -jar ~/GreengrassInstaller/lib/Greengrass.jar \
  --aws-region us-west-2 \
  --thing-name $HOSTNAME \
  --tes-role-name GreengrassV2TokenExchangeRole \
  --tes-role-alias-name GreengrassCoreTokenExchangeRoleAlias \
  --component-default-user $USER:$USER \
  --provision true \
  --setup-system-service true

2. We make a deployment to the device. and it looks like when it tried to report the completed deployment job status, it shows the error in /greengrass/v2/logs

2024-11-14T14:21:41.439Z [WARN] (pool-3-thread-7) com.aws.greengrass.deployment.ShadowDeploymentListener: Caught exception while subscribing to shadow topics, will retry shortly. {}
java.util.concurrent.TimeoutException
        at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
        at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
        at com.aws.greengrass.mqttclient.MqttClient.subscribe(MqttClient.java:532)
        at com.aws.greengrass.mqttclient.WrapperMqttClientConnection.subscribe(WrapperMqttClientConnection.java:77)
        at software.amazon.awssdk.iot.iotshadow.IotShadowClient.SubscribeToUpdateNamedShadowRejected(IotShadowClient.java:1233)
        at com.aws.greengrass.deployment.ShadowDeploymentListener.subscribeToShadowTopics(ShadowDeploymentListener.java:247)
        at com.aws.greengrass.deployment.ShadowDeploymentListener.lambda$setupShadowCommunications$2(ShadowDeploymentListener.java:218)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

2024-11-14T14:21:47.772Z [WARN] (pool-3-thread-6) com.aws.greengrass.deployment.IotJobsHelper: No connection available during subscribing to Iot Jobs descriptions topic. Will retry in sometime. {ThingName=burro-8-475}
java.util.concurrent.TimeoutException
        at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
        at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
        at com.aws.greengrass.mqttclient.MqttClient.subscribe(MqttClient.java:532)
        at com.aws.greengrass.mqttclient.WrapperMqttClientConnection.subscribe(WrapperMqttClientConnection.java:77)
        at com.aws.greengrass.deployment.IotJobsClientWrapper.SubscribeToDescribeJobExecutionAccepted(IotJobsClientWrapper.java:198)
        at software.amazon.awssdk.iot.iotjobs.IotJobsClient.SubscribeToDescribeJobExecutionAccepted(IotJobsClient.java:284)
        at com.aws.greengrass.deployment.IotJobsHelper.subscribeToGetNextJobDescription(IotJobsHelper.java:533)
        at com.aws.greengrass.deployment.IotJobsHelper.subscribeToJobsTopics(IotJobsHelper.java:489)
        at com.aws.greengrass.deployment.IotJobsHelper.lambda$setupCommWithIotJobs$5(IotJobsHelper.java:347)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

2024-11-14T14:22:37.035Z [WARN] (Thread-6) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Connection interrupted. {reason=null, clientId=burro-8-475, reasonCode=KEEP_ALIVE_TIMEOUT, error=Mqtt5 client connection interrupted by server DISCONNECT.}
2024-11-14T14:22:37.036Z [ERROR] (Thread-6) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Error subscribing to topic. {clientId=burro-8-475, topic=$aws/things/burro-8-475/shadow/name/AWSManagedGreengrassV2Deployment/update/rejected}
java.util.concurrent.CompletionException: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
        at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
Caused by: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)

2024-11-14T14:22:37.036Z [ERROR] (Thread-6) com.aws.greengrass.mqttclient.MqttClient: Error subscribing. {topic=$aws/things/burro-8-475/shadow/name/AWSManagedGreengrassV2Deployment/update/rejected}
java.util.concurrent.CompletionException: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
        at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
Caused by: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)

2024-11-14T14:22:37.037Z [ERROR] (Thread-6) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Error subscribing to topic. {clientId=burro-8-475, topic=$aws/things/burro-8-475/jobs/$next/namespace-aws-gg-deployment/get/accepted}
java.util.concurrent.CompletionException: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
        at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
Caused by: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)

Expected behavior
It should report the completed deployment job status

Actual behavior
It cannot report any job status. In the cloud side, the status of the job keeps being active

Environment

• OS: Ubuntu 20.04
• JDK version: openjdk version "11.0.19" 2023-04-18
• Nucleus version: 2.13

Additional context
This issue starts happening in the past 1~2 weeks. Wondering if there is a bug in new release or something.