HCL2 <-> JSON Transformer?

[Aedalus]

Hi all!

I was wondering if there was anything in the works for a library that could convert HCL2 to JSON and back again, similar to https://github.com/kvz/json2hcl

Thanks!

[apparentlymart]

Hi @Aedalus,

It isn't possible in general to convert arbitrary HCL to JSON and back because the meaning of particular JSON as HCL depends on the application's configuration schema. This was true even in HCL1, and so json2hcl will sometimes generate a result that cannot be parsed.

However, we could potentially have some helpers in this codebase to help an application provide its own tool for such a conversion, by specifying its configuration schema similar to how it would during decoding and thus giving enough information to know whether to treat a JSON object as a block or a map value, etc.

We won't be able to build such a thing in the very near future since the current focus is on fixing bugs in the main parser/decoder flow, but we'll leave this issue open to represent the request and take a closer look at it once this codebase is more stable.

[tmccombs]

So, it's pretty clear to me why JSON cannot be unambigously converted to HCL, but it seems like going the other way (HCL to JSON) should be quite possible. I've started work on a program to do that, but wonder if ti would make sense for the hcl2 library to support serializing an hcl.Body or hclsyntax.Body as json (https://godoc.org/github.com/hashicorp/hcl2/hcl/json doesn't work, because it uses a different, undocumented, format).

[apparentlymart]

Hi @tmccombs,

The implementation of hcl.Body for JSON is in terms of a direct representation of a JSON object, but that representation is focused on decoding and not on re-serialization. If you intend to convert from HCL native syntax to JSON then I would suggest framing it as a conversion from hclsyntax.Body to map[string]interface{} and then use encoding/json from there. there's nothing special about HCL's own specific internal representation of JSON that you can't also do with encoding/json; HCL just has its own parser and decoder so that it can produce better error messages that take HCL context into account, whereas Go's encoding/json can only report problems at the JSON syntax level.

With that said, there are still some details you will not be able to handle correctly without access to application schema: some of the special static analysis modes call for a different JSON serialization. For example, in Terraform's variable blocks the type argument is decoded using the "static traversal" and "static call" decoding modes, which call for a different representation in JSON than for a general expression:

variable "example" {
  type = string
}

{
  "variable": {
    "example": {
      "type": "string"
    }
  }
}

HCL is not designed to allow schema-agnostic translation between syntaxes. Instead, its goal is to allow all syntaxes to be able to represent all constructs for a given schema. I've written a longer-form description of how HCL approaches JSON compatibility in my personal blog article JSON and HCL, if you're interested in more details.

With that said, if you are planning to work with an application whose schema does not include any of these static analysis mechanisms then you're right that a mechanical translation from native syntax to JSON is possible in principle, though a specific detail you'll need to contend with is that there is no generalized mechanism in HCL to serialize an arbitrary hclsyntax.Expression back to source code, so I think you'd need to do something like this:

• Let rng := expr.Range()
• Let fileBytes := ReadFileBytes(rng.Filename)
• Let exprSrc := rng.SliceBytes(fileBytes)
• If exprSrc does not start with "${":
  • exprSrc = "${" + exprSrc + "}"
• Use exprSrc as the value of a JSON string in the result.

There are some other optional exceptions you could make here to optimize the output to be more like what a human might write, such as recognizing true, false, null and numbers and writing them out directly as the corresponding JSON values without any extra quoting.

Given that there are a bunch of gotchas for this that make it not fully general for any caller, I expect we would not try to add functionality in this codebase directly to do this, but it is likely possible to write a "good enough" native syntax to JSON converter if you know that in your situation the above caveats don't matter. A fully-general solution will would always require some sort of hints about the specific language being converted though, and the Terraform language does use features that require schema hints to produce a valid result.

[tmccombs]

Well if anyone is interested, I've created https://github.com/tmccombs/hcl2json which converts from native hcl to json, with the caveats that @apparentlymart mentioned.

[samuelvl]

Is it possible to convert variables file to JSON and convert to HCL2 again?

var1 = "foo"
var2 = "bar"
var3 = {
  key = "value"
}

Will produce the following JSON result using json2hcl:

{
  "var1": "foo",
  "var2": "bar",
  "var3": [
    {
      "key": "value"
    }
  ]
}

Is there any way to retrieve original content back?

[PetrusHahol]

Give it a try https://github.com/PetrusHahol/pyhcl2

[vikas027]

@PetrusHahol - I can't seem to install pyhcl2 through pip3

$ pip3 install pyhcl2        
ERROR: Could not find a version that satisfies the requirement pyhcl2 (from versions: none)
ERROR: No matching distribution found for pyhcl2

hcl2json Go binary recommended by @tmccombs works like a charm on both HCL v1 and v2

[rulatir]

Rulatir's Prophecy

Heed my words: not designing for rigorous bidirectionality from day 0 will be HCL's and terraform's undoing. Within two years starting from now a competing product will emerge, designed from the ground up to support 100% mutually translatable representations for humans and for machines. As a result, its rate of adoption will be 5000% that of terraform's rate of adoption, and it will relegate terraform to obsolescence within another 2 years.

[mitchellh]

not designing for rigorous bidirectionality from day 0 will be HCL's and terraform's undoing

Please note @rulatir that as of Terraform 0.12 (w/ HCL2) there is "rigorous bidirectionality" in the sense that there is an exact 1:1 mapping between JSON and HCL. The issue at hand here is that we don't provide an automated way to do that conversion to/from, which is indeed an open issue.

[tmccombs]

as of Terraform 0.12 (w/ HCL2) there is "rigorous bidirectionality" in the sense that there is an exact 1:1 mapping between JSON and HCL

There is only a 1:1 mapping between JSON and HCL if you know the schema, so you can't really have a general tool to convert between JSON and HCL. Maybe a tool could allow you to specify a schema somehow. But right now the schema is expressed entirely in go code that is tightly coupled to the implementation, and in terraform's case it is even split between terraform itself and individual plugins.

[jkevlin]

I agree, the inability to Marshal/Unmarshal TF hcl in go, is a huge negative. I would not recommend TF for this reason.

[IvanJosipovic]

I agree, the inability to Marshal/Unmarshal TF hcl in go, is a huge negative. I would not recommend TF for this reason.

Agreed

[apparentlymart]

Hi all,

This discussion seems to be veering off-topic. It was originally about converting from JSON syntax to native syntax, but recent comments seem to instead be talking about support for programmatically generating Terraform configuration in Go.

As I mentioned before, converting between syntaxes is a concern that would be handled at the application layer, because only the application knows how it intends to interpret the constructs in each syntax. I expect that a shared library could provide some functionality to help applications provide such a mapping in a more convenient way, but that alone would not be sufficient to make inter-syntax conversions available for individual HCL-based languages because each application would need to define its mapping in terms of whatever that is.

If you want to share use-cases for converting between HCL syntaxes in a particular application then I would suggest opening an issue in that application's own repository to advocate for them, and then if one of those application teams decides that such a capability is a high priority for them then they can work with me or one of the other HCL maintainers to design a means to get that done. So far, no team developing an HCL-based application has indicated their intent to support such a thing, and so I have no practical use-cases with which to evaluate potential solutions.

This issue doesn't represent something actionable on its own, so I'm going to close it. If a team working on an HCL-based application wants to pursue something like this, please open a new issue where we can discuss your specific use-cases and produce a suitable technical design with them in mind. Thanks!

[adrian-gierakowski]

Given that it’s now possible to convert HCL to CDKTF and back, and that CDKTF can also produce json output, what’s the reason not to support JSON => HCL conversion? Is some information lost when CDKTF is used to generate json (vs hcl)?

The reason I’d like to be able to do JSON => hcl is that I’m using a tool which generates terraform projects in JSON format and I’d like to produce hcl from its output.

[apparentlymart]

Interconversion between CDK for Terraform programs and Terraform language programs is an example of an application-layer conversion like I was describing in my previous comment.

The system making that conversion is working at the Terraform level of abstraction, and not at the HCL level of abstraction, despite the authors of that documentation incorrectly using the term "HCL" when they ought to have said "the Terraform language".

For example, the code that's supporting part of that functionality includes a function renderResource, where "resource" is a Terraform-specific concept that HCL alone knows nothing about. CDK for Terraform is not generating JSON and then translating it to native syntax as a separate step.

Converting Terraform language native syntax to Terraform language JSON syntax and back is possible in principle, and would require a similar awareness of Terraform-specific concepts to allow the system to correctly understand how to represent a particular JSON input as the equivalent native syntax document.

As I mentioned before, if you want to discuss use-cases for doing that in the Terraform language then the appropriate place to discuss that is in Terraform's own repository. If the Terraform team chooses to pursue that then I will help them to evaluate different strategies for doing so.

[adrian-gierakowski]

@apparentlymart thank you for the thorough explanation and apologies for not having read through all comments in this issue. I’ll raise the issue in the terraform repo (if there isn’t one already).