opa eval cant parse json file when it used in GitHub Actions. But locally it works.

[vladimirpodr]

Short description

I evaluate OPA policy to find the Azure resources without the required tags. Locally it works and outputs non-compliant resources. But when I try to evaluate the policy in the GitHub Actions pipeline - the output is empty. I rechecked everything: local and 'in pipeline' configs, versions are the same, all files exist, permissions, etc. In debug mode, I see that "in pipeline" OPA replaces input data with a simple "command" string.

Config:
Local and "in pipeline" OPA version - 0.48.0
Local machine - MacOS Ventura
Pipeline runner image based on - Ubuntu 22.04

OPA policy:

package policy

mandatory_tags[msg] {
  check_tags(changes[c])
  #msg := sprintf("fail: %v is missing required tags.", [changes[c].address])
	msg := sprintf("%v", [changes[c].address])
}

check_tags(resource) {
 # read the tags based on the resource type
 tags = read_tags(resource)
 # check for the tag enforcement
 match(resource.change.after)
}

read_tags(resource) = tags {
 tags = resource.change.after.tags
}

match(i) {
	not i.tags
}

match(i) {
	not i.tags.AgileTeam
}

match(i) {
	not i.tags.Contact
}

match(i) {
	not i.tags.Environment
}

match(i) {
	not i.tags.Repository
}

# get the changed resources
changes := { c |
	some path, value
	walk(input, [path, value])
	reverse_index(path, 1) == "resource_changes"
	c = value[_]
}

reverse_index(path, idx) = value {
	value := path[count(path) - idx]
}

I run the next commands:

terraform init
terraform plan --out tfplan.binary
terraform show -json tfplan.binary > tfplan.json

opa eval data.policy.all_policies -d ../../.opa/ -i tfplan.json -f pretty

Local output:

[
  "fail: azurerm_nat_gateway.development is missing required tags.",
  "fail: azurerm_public_ip_prefix.development is missing required tags."
]

In pipeline output:

[]

Local debug:

In pipeline debug:

And, again, all needed files are in place, OPA version is the same. Locally it works but doesn't work in GitHub Actions Pipeline

[ashutosh-narkar]

@vladimirpodr it could be that in the pipeline OPA does not see the input as you expect it to? What happens if you run locally with an empty input file for example. How did you set this up? Here's a reference to setup OPA cli in GHA if it helps.

[vladimirpodr]

@ashutosh-narkar Hi

1. Yes, I set up opa with open-policy-agent/setup-opa@v2 action.
2. I recheked several times. Inpup file exists and has correct data. As you can see on screenshot ^^^
3. Please, return the BUG label

[vladimirpodr]

@ashutosh-narkar I tested with different environments the order of the next commands:

cd /path-to-some-folder-with-tf-code
terraform init
terraform validate
terraform plan -lock=false --out tfplan.binary
terraform show -json tfplan.binary > tfplan.json
opa eval data.policy.mandatory_tags -d /path-to-opa-folder-with-policies/ -i tfplan.json -f pretty

1. Local laptop MacOS: all ok
2. container with ubuntu 22.04: all ok
3. GitHub Runner on ubuntu-latest (ubuntu 22.04): not ok. empty result []
4. GitHub Runner on macos-latest: not ok. empty result []

[anderseknert]

In your GitHub runner, what does this output?

cat tfplan.json | opa eval -I -f pretty input

FWIW, we run opa eval in quite a few GHA pipelines, including OPA itself.

[vladimirpodr]

Hi @anderseknert !
The result is:

But should be (local run with the same config and input data):

[anderseknert]

Hey @vladimirpodr 👋 That doesn't look like the command I asked about though? I literally just want to see what OPA thinks the input is:

cat tfplan.json | opa eval -I -f pretty input

[vladimirpodr]

Hey @anderseknert ! Oh, sorry))
The result of the command cat tfplan.json | opa eval -I -f pretty input :

[
  "command"
]

[anderseknert]

Thanks Vladimir. That is indeed truly mysterious 🕵🏻‍♂️ I'm afraid I don't know what could be going on here. I don't suppose the code is public? If not, perhaps we could try and reproduce this in as minimal example as possible? I.e. just copy the tfplan file to some public repo and setup a simple GHA flow there? Or if you could try with another tfplan file, possibly with only a tiny little change?

[igoritos22]

I'm facing the same situation.

[anderseknert]

If anyone is able to provide a link to a repo where this is reproducible, that would be helpful.

[anderseknert]

Did anyone manage to make any findings on this?

[mkurimeti]

I'm having a similar problem; any advice from anyone?

[anderseknert]

Nothing yet :/ If anyone is able to reproduce this in a public repo, do let us know!

[mkurimeti]

On GitHub - https://github.com/actions/runner-images/tree/main/images, I have seen there were few updates on the runner images and now its working fine.

[anderseknert]

That's good news, @mkurimeti . Did you have to do anything in order to update the runner?

[mkurimeti]

Nope, I haven't made any changes on my end; instead, the community in general has made all the fixes.

[anderseknert]

Thanks @mkurimeti 👍

@vladimirpodr @igoritos22 have you seen this fixed by recent updates too?

[anderseknert]

Closing due to inactivity. Let me know if anyone is experiencing this still with the latest versions of the runner, and I'll have it re-opened.

[nafarlee]

@anderseknert
I do not mean to invalidate @mkurimeti's experience, but I think I understand why this is happening (and it definitely is still happening for us). If I am correct, it wouldn't be unique to the self-hosted runners either, but I haven't tested that yet.

To be brief, this isn't really an opa bug. It is a "gotcha" that comes from extrapolating the steps at https://www.openpolicyagent.org/docs/latest/terraform for use within GitHub Actions.

To determine this, I inspected our self-hosted runner's filesystem directly while a Terraform-related opa workflow was running. I noticed something peculiar about the tfplan.json that wasn't present in local testing and wasn't showing in the GitHub Actions log itself:

$ cat tfplan.json
[command]/runner/_work/_temp/e16010ee-5ac0-417f-8ffe-7aab4de2e9a0/terraform-bin show -json tfplan
...
::debug::stderr: 
::debug::exitcode: 0

The ... above represents the expected JSON contents, but this "header" and "footer" metadata is also present. Like many others, I was creating the JSON representation of the Terraform plan with a command like:

terraform show -json tfplan > tfplan.json

I believe this extra metadata is being added by the terraform_wrapper that is installed by default when using hashicorp/setup-terraform.

Later, when this file is provided as input to opa, I think it is parsing the first part ([command]) as a YAML array with a single, unquoted string element. This would explain the [ "command" ] output noted earlier in this issue. As an example:

$ echo -e '[command]qwerty\n{}' | opa eval -I -f pretty input
[
  "command"
]

The solution here is to ensure the terraform_wrapper is disabled when terraform show -json ... is executed. Even with the terraform_wrapper enabled, you may be able to usehead and tail to strip the "invisible" GitHub Actions metadata as you write the file.

I am not sure if it is worth including this somewhere in the Open Policy Agent docs. In any case, I hope this helps someone in the future!

[anderseknert]

Stellar research! That sounds like a very plausible explanation. While it would be a little unusal to point this out in docs that have nothing to do with GitHub Actions, I think a note might be warranted given how that's likely a common way to evaluate these plans. Would you like to submit a PR for the docs?

[nafarlee]

Thank you for your kind words! As a small addendum, I did create a reproduction example here. In summary, this is indeed caused by the terraform_wrapper and is unrelated to the use of self-hosted or GitHub-provided runners.

I agree that this would feel somewhat out of place in OPA's Terraform documentation when it is isolated to GitHub Actions. Might I make a suggestion? This note might be better placed in the open-policy-agent/setup-opa documentation. If you agree, I don't mind submitting a PR to create a "Troubleshooting" section that includes this as its first entry.

[anderseknert]

Please do! 👍