For_each on data source like http

[mahsoud]

Description

for_each attribute support on configuration resource

Use Case(s)

At times we may iterate on multiple data sources.

Potential configuration

data "http" "this" {
  for_each = local.fileset
  URL         = "https://git.myserver/api/v4/projects/1000/repository/files/${replace(each.value, "/", "%2F")}?ref=master"
  request_headers = {
    PRIVATE-TOKEN = var.TOKEN
  }
}

Potential References

[lbajolet-hashicorp]

Hi @mahsoud,

Regarding this feature, this would imply we support dynamic on top-level blocks, which is something that is unsupported for everything.
Dynamic blocks are not even supported for data sources for now (though this may be supported someday).

It looks to me that what you're looking for is a list of URIs to fetch, which is something that may be implemented separately from this language change, as part of a data source specialised to do this.
The http data-source is only meant to get one resource through HTTP, maybe a new datasource that accepts a list of URIs and returns them as a list or a map would be appropriate for your use-case. That being said, there's the problem of how to handle errors in this case, I'm not sure what would be expected if one or multiple URIs fail to be fetched, at which point is this considered an error?

Out of curiosity, do you have a practical example of the problem you're trying to solve? Maybe there's something we can help to express with the current capabilities of HCL in Packer.

I'll keep this open for discussion purposes, but I'll mark this as a wontfix for now, since it's likely not something we'll take-on soon (and possibly long-term even).

Thanks!

[lmayorga1980]

I am doing an experiment that might be related to this. I have a custom plugin that fetches some external API and returns a list of objects but currently the Datasource can only use map of strings instead of a list of objects. This causes my Custom Plugin Data Source to only support a return of 1 object(map of strings) and i have to make multiple calls of this plugin inside a loop. Wouldn't this quality for something that Terraform already does in their loop expressions

[lbajolet-hashicorp]

Hi @lmayorga1980, not sure I understand your use-case, could you provide a quick example template to illustrate what you're trying to do?

[cblkwell]

I can say I have a similar use case; we have AMIs pushed to various regions and we'd like to go back and tag them after the fact with "ready for production". So to get the AMI IDs into a list, what would be great is if I could do something like this, which is what I would do in Terraform:

data "amazon-ami" "ami" {
  for_each = var.ami_regions
  filters = {
    virtualization-type = "hvm"
    name                = "${var.name}*"
    root-device-type    = "ebs"
  }
  owners      = ["559031956274"]
  most_recent = true
  region      = each.value
}

[djaboxx]

any updates on this? I'm also running into this

# Define data sources to fetch values from the Amazon Parameter Store
data "amazon-parameterstore" "get_parameter_list" {
  name = "/image-pipeline/${var.project_name}/parameters"
}

data "amazon-parameterstore" "parameters" {
  for_each = toset(split(",", data.amazon-parameterstore.get_parameter_list.value))
  name = "/image-pipeline/${var.project_name}/${each.key}"
}