Change to HCL from JSON

[aleqsss]

Hi,

Will you change the already existing Packer configurations to be JSON instead of HCL?
I can see that you're creating new ones, such as the ubuntu2204.pkr.hcl as HCL instead of JSON.

The reason why I'm asking, is because we're processing the JSON files and change some values of the Packer configuration.
If you change to HCL instead of JSON, this will fail our existing configurations and pipelines.

What does the roadmap look like in regards of this?

Thank you!

[mikhailkoliada]

@aleqsss Hello!

We have no plans of converting existing configuration, but any new configurations will be provided in the hcl2 format, as we did with Ubuntu 22.04

[aleqsss]

Thank you for your reply @mikhailkoliada, much appreciated.

[brgsstm]

We are also changing parameters in the JSON currently. I've only scratched the surface so far in terms of what we will need to achieve this with hcl2 but it doesn't look like it will be anywhere near as simple...

We have strict Azure Policy (as per MS recommendation) so deploying VHD based VMs with public IPs is a no-go for us.

[aleqsss]

We can easily convert JSON to objects with PowerShell and do some stuff with that, but I'm a bit unsure on how we should go on about it with HCL2. Tips are appreciated. 🙂

[rybal06]

We can easily convert JSON to objects with PowerShell and do some stuff with that, but I'm a bit unsure on how we should go on about it with HCL2. Tips are appreciated. 🙂

I am also parsing the JSON objects in Powershell; I use this to append additional provisioners. I have not yet researched how to serialize HCL into powershell objects or something similar in another language yet; but I may end up running packer twice -

1. Build MS stock Images
2. Create a second image with our custom pre-installed software by using the built "MS stock images" as the image source.

[aleqsss]

Understood, perhaps we need to do something similar. Or something nasty with search and replace (with or without regular expressions).

[brgsstm]

We can easily convert JSON to objects with PowerShell and do some stuff with that, but I'm a bit unsure on how we should go on about it with HCL2. Tips are appreciated. 🙂

I am also parsing the JSON objects in Powershell; I use this to append additional provisioners. I have not yet researched how to serialize HCL into powershell objects or something similar in another language yet; but I may end up running packer twice -

1. Build MS stock Images
2. Create a second image with our custom pre-installed software by using the built "MS stock images" as the image source.

So I got as far as serializing into a Python Dict using the python-hcl2 module, however the challenge is then outputting that back into HCL2, so far it looks like that is only possible using Go (i've not spent much time researching this though yet...)

[aleqsss]

Interesting @brgsstm, I did a quick search for the possibility to (in a "stable" way) convert HCL2 <---> JSON. Even if there is some good implementation of that (which I doubt because of various dilemmas) there would be a lot of conversion going on and potential trouble.

One could however just convert it to JSON, do the modifications on it, then just run Packer with that modified JSON template file as an input, without converting it back to HCL2 again. Still, I'd like to keep away from doing that conversion.

[aleqsss]

Below I'll write out some different reflections/angles/approaches/tools and stuff to try. Perhaps it'll benefit someone, perhaps not.

python-hcl2 module (or any other parser for that matter):

As brought up by @brgsstm the python-hcl2 module. However, as I see it (might be totally wrong) it could be hard to use any "parser" (not only this searlizer to a Python dict) because will they actually preserve all information? Such as comments and ordering of arguments within blocks? Will the parser actually retain that, if so - how? Perhaps for some scenarios (such as this) that's superfluous anyway?

Convert HCL2 <---> JSON (optionally back and forth):

Possibility to (in a "stable" way) convert HCL2 <---> JSON. Even if there is some reliable implementation of that (which I doubt because of various dilemmas) there would be a lot of conversion going on and potential trouble.

One could however just convert it to JSON, do the modifications on it, then just run Packer with that modified JSON template file as an input, without converting it back to HCL2 again. Still, I'd like to keep away from doing that conversion.

Override files (terraform only?)

Override Files feature of Terraform, haven't read too much about it so don't know if it works/there's an equivalent for Packer. But for Terraform this seems to be a solution for some scenarios where you'd like to override some stuff.

hclwrite

Perhaps the hclwrite GO package could be of use?

Package hclwrite deals with the problem of generating HCL configuration and of making specific surgical changes to existing HCL configurations.

It operates at a different level of abstraction than the main HCL parser and AST, since details such as the placement of comments and newlines are preserved when unchanged.

[aleqsss]

I'm also doing this with the hclwrite package which I mentioned in this post.

[zioalex]

@iZnoGod722 did you do any progress?

[zioalex]

@aleqsss can you clarify the challenge that you foresee in using pythin-hcl2 package?

[iZnoGod722]

@iZnoGod722 did you do any progress?

@zioalex sorry I'd put the project on hold as we were only needed to build from the ubuntu20. 04 packer template which I thought would remain as json until recently when I returned from holiday I see now all templates are now in hcl format.

I've spent the past few days reviving the project and I have successfully used hclwrite package to make modifications to the templates, specifically adding my own custom provisioners.

Will post some code snippets here as soon as I get a chance.

[iZnoGod722]

@zioalex Here's a sample code.
This assumes the template file is located in the same directory as the go script.

It will read the template and perform the following:

1. Update the default value of image_version variable
2. Adds an aditional shell provisioner block

It will then save it to a new file.

If you need further assistance let me know.

package main

import (
	"flag"
	"fmt"
	"log"
	"os"

	"github.com/hashicorp/hcl/v2"
	"github.com/hashicorp/hcl/v2/hclsyntax"
	"github.com/hashicorp/hcl/v2/hclwrite"
	"github.com/zclconf/go-cty/cty"
)

func main() {
	
	f, err := os.ReadFile("ubuntu20.04.pkr.hcl")
	if err != nil {
		log.Fatal(err)
	}

	s, diags := hclwrite.ParseConfig(f, "ubuntu20.04.pkr.hcl", hcl.InitialPos)
	if diags.HasErrors() {
		log.Fatal(diags)
	}
        
        // hardcoding the default value for variable - image_version
	imageVersionBlock := s.Body().FirstMatchingBlock("variable", []string{"image_version"})
	if imageVersionBlock == nil {
		log.Fatal("Error: image version variable not found")
	}
        
	imageVersionBlock.Body().SetAttributeValue("default", cty.StringVal("1.0.0"))

	buildBlock := s.Body().FirstMatchingBlock("build", nil)
	if buildBlock != nil {
		sourcesAttribute := buildBlock.Body().GetAttribute("sources")

		existingBlocks := buildBlock.Body().Blocks()

		buildBlock.Body().Clear()
		buildBlock.Body().AppendNewline()
		buildBlock.Body().AppendUnstructuredTokens(sourcesAttribute.BuildTokens(nil))
                
                // Adding my own shell provisioner block
                addCertsProvisioner := buildBlock.Body().AppendNewBlock("provisioner", []string{"shell"})
		addCertsProvisioner.Body().SetAttributeValue("execute_command", cty.StringVal("sudo sh -c '{{ .Vars }} {{ .Path }}'"))
		addCertsProvisioner.Body().SetAttributeRaw("script", tokensGenerator("/scripts/ubuntu/custom/install-certificates.sh"))
               
               // Adding the original provisioners back
               for _, x := range existingBlocks {
			buildBlock.Body().AppendBlock(x)
		}
                
                err = os.WriteFile("ubuntu-20.04-new.pkr.hcl", s.Bytes(), 0644)
	        if err != nil {
		        log.Fatal(err)
	        }
}

func tokensGenerator(value string) hclwrite.Tokens {
	return hclwrite.Tokens{
		{
			Type:  hclsyntax.TokenOQuote,
			Bytes: []byte(`"`),
		},
		{
			Type:  hclsyntax.TokenTemplateInterp,
			Bytes: []byte("${"),
		},
		{
			Type:  hclsyntax.TokenQuotedLit,
			Bytes: []byte("path.root"),
		},
		{
			Type:  hclsyntax.TokenTemplateSeqEnd,
			Bytes: []byte("}"),
		},
		{
			Type:  hclsyntax.TokenQuotedLit,
			Bytes: []byte(value),
		},
		{
			Type:  hclsyntax.TokenCQuote,
			Bytes: []byte(`"`),
		},
	}
}

[Marco7412]

May bien

[HoLengZai]

There is a module in Powershell which has been released recently
https://www.powershellgallery.com/packages/Hcl2PS/0.2.0
But wondering how to "convert"/generate it back to Hcl format (through JSON?).
If I cannot put it back to HCL, I will probably stick with regular expression for my use case as I did the "injection" in JSON format using specific keyword which should not be change over the time.

[DevOpsAzurance]

I actually update the hcl in my build script using Octopus Deploy's HCLParser using powershell:

if(-not (Test-Path -Path "$agentPackerBuildPath\NugetPackages")){
    New-Item -ItemType Directory "$agentPackerBuildPath\NugetPackages"
}

$nugetPackages = @{
    "Octopus.CoreParsers.Hcl"="1.1.3"
    "Sprache"="2.3.1"
    "Octopus.CoreUtilities"="1.1.2"
}

ForEach($nugetPackage in $nugetPackages.Keys){
    Invoke-WebRequest "$nugetAPIBaseUrl/$nugetPackage/$($nugetPackages[$nugetPackage])" -Method Get -OutFile "$agentPackerBuildPath\NugetPackages\$nugetPackage.zip"
    Expand-Archive -Path "$agentPackerBuildPath\NugetPackages\$nugetPackage.zip" -DestinationPath "$agentPackerBuildPath\NugetPackages\$nugetPackage\" -Force
}

Add-Type -AssemblyName System.Linq
Add-Type -AssemblyName System.Collections
Add-Type -AssemblyName System

...

ForEach($dllFile in $(Get-ChildItem -Path "$agentPackerBuildPath\NugetPackages" -Recurse -File -Filter *.dll | ?{$_.PSParentPath -match "$dotNetVersionBase````$"})){
        try{
            Import-Module $dllFile.FullName
        }catch{
            Write-Warning "Failed to load $dllFile.Name"
            Write-Warning $_
        }
    }

    ### Start Copy Scripts ###

    ForEach($customDevOps_CommonScriptWin in $customDevOps_CommonScriptsWin){
    
        (Get-Content "$DevOps_CommonScriptsPath\$customDevOps_CommonScriptWin") | %{$_ `
            -replace 'STUFF' `
            -replace 'STUFF"
        } | Set-Content "$DevOps_CommonScriptsPath\$customDevOps_CommonScriptWin"

        Copy-Item -Path "$DevOps_CommonScriptsPath\$customDevOps_CommonScriptWin" -Destination $scriptsPathWin -Force
    }

    ### Getting Current Image Version Information
    $currentImageVersionWin = Get-AzGalleryImageVersion -ResourceGroupName $computeGalleryRG -GalleryName $imgGalleryName -GalleryImageDefinitionName $imgDefNameWin
    if($null -eq $currentImageVersionWin){
        $latestImageVersionWin = [System.Version]::new(1,0,0)
    }else{
        $latestImageVersionWin = ([System.Version[]] ($currentImageVersionWin.Name) | Sort-Object -Descending)[0]
    }

    [int[]] $templateYearsWin = (Get-ChildItem -Path "$agentPackerBuildPath\images\windows\templates").Name | %{([Regex]::Split($_, '[-.]'))[1]}

    $latestWin = ($templateYearsWin | Sort-Object -Descending)[0]

    ### Update PKR.HCL ###

    $hclParseWin = [Octopus.CoreParsers.Hcl.HclParser]::HclTemplate.Invoke([Sprache.Input]::new((Get-Content -Path "$agentPackerBuildPath\images\windows\templates\windows-$latestWin.pkr.hcl" -Raw)))

They have the source public for it:

https://github.com/OctopusDeploy/HCLParser

Which helped me in figuring out how to use the classes accordingly. One caveat I have is that the .ToString() formats in a way that my packer does not like, due to formatting, so I add the following:

### Outputing Hcl to File ###
$hclParseWin.Value.ToString() `
    -replace "([a-z])[(]$([char]10)[`"]",'$1(' `
    -replace "([a-z])[`"]$([char]10) *[)]",'$1)' `
    -replace '([\(\[{])\s*([}\]\)])','$1$2' `
    -replace "$([char]10)`(\s*`)azure_tag [{]", "$([char]10)$([char]10)`$1`dynamic `"azure_tag`" {" `
    | Set-Content -Path "$agentPackerBuildPath\images\windows\templates\windows-$latestWin.pkr.hcl" -Force -Encoding UTF8

This resolves issues with the dynamic variable in the image section for the tags and some new-line issues I saw that the encoding of UTF8 was not resolving.