Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

puppet: stack output doesn't update value in SSM parameter on puppet account. #719

Closed
mpratsch opened this issue Aug 5, 2024 · 12 comments
Closed

puppet: stack output doesn't update value in SSM parameter on puppet account. #719

mpratsch opened this issue Aug 5, 2024 · 12 comments

Comments

@mpratsch
Copy link

mpratsch commented Aug 5, 2024

Hi Eamonn,

Describe the bug

puppet version: v0.247.0

1 year ago we have created this stack where we only deploy a lambda layer and save the output to a SSM parameter.
Now we had to update the layer because of a security issue.

A single action run rolled out the new version v2.

The value in the stack output has changed to layer version *:2 but not in the SSM parameter in the puppet account.
There we still have the lambda version *:1 referenced what would cause issues with all our stacks where this stack is dependent on.

(The sst tag is the puppet account)

Puppet config:

  ccoe-aws-sct-shared-lambda-layer-3-11:
    name: ccoe-aws-sct-shared-lambda-layer-3-11
    active: true
    deploy_to:
      tags:
        - tag: "type:sst"
          regions: default_region
    version: v2
    outputs:
      ssm:
        - param_name: /deployment/lambda/ccoe_lambda_layer_library_311
          stack_output: CcoeLambdaLayer311Arn

factory_config:


  - Name: "ccoe-aws-sct-shared-lambda-layer-3-11"
    Description: |
      Creates a shared Lambda layer which supports python3.11.
    Distributor: "CCoE"
    Tags:
      - Key: "Creator"
        Value: "CCoE"
      - Key: "DeployedVia"
        Value: "Service Catalog Tools"
    Versions:
      - Name: "v1"
        Description: "Creates a shared Lambda layer which supports python3.11."
        Active: True
        Stages:
          Build:
            BuildSpec: buildspec.yaml
            BuildSpecImage: aws/codebuild/standard:7.0
        Source:
          Provider: CodeStarSourceConnection
          Configuration:
            BranchName: "v1"
            ConnectionArn: "arn:aws:codeconnections:eu-central-1:0778******:connection/3051ede8-7056-4744-bc1d-********"
            FullRepositoryId: "***/ccoe-aws-sct-shared-lambda-layer-3-11"
            OutputArtifactFormat: "CODE_ZIP"
      - Name: "v2"
        Description: "Creates a shared Lambda layer which supports python3.11."
        Active: True
        Stages:
          Build:
            BuildSpec: buildspec.yaml
            BuildSpecImage: aws/codebuild/standard:7.0
        Source:
          Provider: CodeStarSourceConnection
          Configuration:
            BranchName: "v2"
            ConnectionArn: "arn:aws:codeconnections:eu-central-1:0778******:connection/3051ede8-7056-4744-bc1d-********"
            FullRepositoryId: "***/ccoe-aws-sct-shared-lambda-layer-3-11"
            OutputArtifactFormat: "CODE_ZIP"

Part of the puppet log:

2024-08-03 19:13:47 : INFO MainProcess scheduler starting batch                                                                                                                                                   
2024-08-03 19:13:47 : INFO MainProcess scheduler sending: stacks_ccoe-***-sct-shared-lambda-layer-3-11_4225********_eu_central_14225********
2024-08-03 19:13:47 : INFO MainProcess worker#3 executing task: stacks_ccoe-***-sct-shared-lambda-layer-3-11_4225********_eu_central_1                                                                           
2024-08-03 19:13:47 : INFO MainProcess worker#3 ProvisionStackTask:stacks_ccoe-***-sct-shared-lambda-layer-3-11_4225********_eu_central_1 started
2024-08-03 19:13:49 : INFO MainProcess worker#3 stacks_ccoe-***-sct-shared-lambda-layer-3-11_4225********_eu_central_1: params unchanged                                                                          
2024-08-03 19:13:49 : INFO MainProcess worker#3 stacks_ccoe-***-sct-shared-lambda-layer-3-11_4225********_eu_central_1: template changed
2024-08-03 19:13:51 : INFO MainProcess worker#3 getting get_puppet_stack_role_arn
2024-08-03 19:13:51 : INFO MainProcess worker#3 getting puppet_stack_role_name
2024-08-03 19:13:51 : INFO MainProcess worker#3 Creating or updating: ccoe-***-sct-shared-lambda-layer-3-11
2024-08-03 19:13:51 : INFO MainProcess worker#3 Updating (without changeset): ccoe-***-sct-shared-lambda-layer-3-11
2024-08-03 19:13:51 : INFO MainProcess worker#3 Waiting for update_stack to complete: ccoe-***-sct-shared-lambda-layer-3-11
2024-08-03 19:14:21 : INFO MainProcess worker#3 Finished stack: ccoe-***-sct-shared-lambda-layer-3-11
2024-08-03 19:14:21 : INFO MainProcess worker#3 executed task [success]: stacks_ccoe-***-sct-shared-lambda-layer-3-11_4225********_eu_central_1 got lock to unlock resources
2024-08-03 19:14:21 : INFO MainProcess scheduler receiving: [1]: stacks_ccoe-***-sct-shared-lambda-layer-3-11_4225********_eu_central_1, COMPLETED
2024-08-03 19:14:21 : INFO MainProcess scheduler tasks now scheduled                                                                                                                                              
2024-08-03 19:14:21 : INFO MainProcess scheduler finished batch
2024-08-03 19:14:21 : INFO MainProcess scheduler starting batch                                                                                                                                                   
2024-08-03 19:14:21 : INFO MainProcess scheduler sending: ssm_outputs-4225********-eu-central-1-/deployment/lambda/ccoe_lambda_layer_library_311                                                                  
2024-08-03 19:14:21 : INFO MainProcess worker#4 executing task: ssm_outputs-4225********-eu-central-1-/deployment/lambda/ccoe_lambda_layer_library_311                                                            
2024-08-03 19:14:21 : INFO MainProcess worker#4 SSMOutputsTasks:ssm_outputs-4225********-eu-central-1-/deployment/lambda/ccoe_lambda_layer_library_311 started                                                    
2024-08-03 19:14:21 : INFO MainProcess worker#4 executed task [success]: ssm_outputs-4225********-eu-central-1-/deployment/lambda/ccoe_lambda_layer_library_311 got lock to unlock resources                      
2024-08-03 19:14:21 : INFO MainProcess scheduler receiving: [1]: ssm_outputs-4225********-eu-central-1-/deployment/lambda/ccoe_lambda_layer_library_311, COMPLETED
2024-08-03 19:14:21 : INFO MainProcess scheduler tasks now scheduled                                                                                                                                              
2024-08-03 19:14:21 : INFO MainProcess scheduler finished batch

Expected behavior
I would expect to have also the ssm parameter updated too

I think it's easy to test. I assume it happens with all updates?!

If you need more information Eamonn, just ping me via email and I'll provider more logs/information.

All the best,
Martina
@eamonnfaherty
Copy link
Contributor

This sounds like it could be a caching issue. I will take a look at this today or tomorrow.

@eamonnfaherty
Copy link
Contributor

Could you please send me the logs.

@mpratsch
Copy link
Author

mpratsch commented Aug 5, 2024

Sure, you only need the whole puppet-deploy codebuild log or more?
I'll send that via email.

@eamonnfaherty
Copy link
Contributor

eamonnfaherty commented Aug 5, 2024 via email

@mpratsch
Copy link
Author

mpratsch commented Aug 5, 2024 •
edited
Loading

I sent you everything via email to your company's email address.
I forgot in the email the account id starting with 4225 and ending with 844 is the SST (puppet account).

@mpratsch
Copy link
Author

mpratsch commented Aug 7, 2024

Hey Eamonn,
Did you have the chance to take a look already?
We need to update those resources in prod soon.

Thanks,
Martina

@mpratsch
Copy link
Author

mpratsch commented Aug 8, 2024

@eamonnfaherty any news on that?
We have updated the SSM parameter manually with the new before we start the puppet run.
It worked and it kept our value what indicates not update happens at all.

We tried to update all stacks what are using the layer and they have a dependency on this layer resource.
We ran a single action run with reverse dependencies = True.

Should this update also those stacks? Because there we still see the old lambda layer version and this didn't get update either.

Thanks,
Martina

@mpratsch
Copy link
Author

mpratsch commented Aug 9, 2024

@eamonnfaherty ?

@eamonnfaherty
Copy link
Contributor

Sorry I have been working on other issues. I have blocked out 2 hours for this on Monday. I have figured out the issue and have designed a solution. It is a caching issue and I need to add cache purging in some combinations of actions.

@mpratsch
Copy link
Author

mpratsch commented Aug 11, 2024 via email

eamonnfaherty added a commit that referenced this issue Aug 12, 2024
@eamonnfaherty
Provisioning changes to stacks and products did not cause updates to …
01f773d 
…SSM outputs issue #719
@eamonnfaherty eamonnfaherty mentioned this issue Aug 12, 2024
Merged
@eamonnfaherty
Copy link
Contributor

resolved in https://github.com/awslabs/aws-service-catalog-puppet/releases/tag/0.249.0

Stack and Product updates were not triggering SSM output updates. There was an undocumented parameter you could use to force an update (that was used in testing). Stack and product updated now trigger SSM output updates. When upgrading to this version you may need to force an update of the stack or product so the SSM output is updated. You can do this by terminating the stack or product of forcing it into update rollback complete by trying to update it with a broken CFN template.

@mpratsch
Copy link
Author

Thanks a lot Eamonn!
I'm going to try that out today. I needed to wait until today because we had an active release waiting for on Wednesday, and I didn't want to change anything before.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@eamonnfaherty @mpratsch