tpgtools
is the generator responsible for creating DCL-based resources in the
Terraform provider for Google Cloud (TPG). The DCL provides
OpenAPI schema objects to
describe the available types, and tpgtools
uses those to construct Terraform
resource schemas.
tpgtools
expects to run targeting a "root service directory", a dir-of-dirs
where the child dirs contain OpenApi specs for resources such as the api/
path
above. Additionally, overrides are expected in a similar structure (as seen in
the overrides/
path. For example:
go run . --path "api" --overrides "overrides"
This will output the file contents of each resource to stdout, for fast iteration. You can filter by service and resource to make it more useful:
go run . --path "api" --overrides "overrides" --service redis --resource instance
To persist the output(s) to files you can specify an output path. This is the most familiar experience for MMv1 developers. For example:
go run . --path "api" --overrides "overrides" --output ~/tpg-fork
If generation fails, an error should be logged showing what went wrong. The raw output will be returned, and line numbers (if available in the error) will correspond to the line numbers in the output.
You can specify a version such as beta
using the --version
:
go run . --path "api" --overrides "overrides" --output ~/tpg-fork --version "beta"
To generate accessory code such as serializarion
, you can specify the
--mode
:
go run . --path "api" --overrides "overrides" --output ~/some/dir --mode "serialization"
This guide is written to document the process for adding a resource to the Terraform provider for Google Cloud (TPG) after it has been added to the DCL.
Every resource added via tpgtools needs an override file for every version it is available at. This file should be empty, but must exist. A resource available at GA (TPG) must also exist at beta (TPGB) and needs a corresponding override file at beta. These override files are often identical between versions. This file should exist at tpgtools/overrides/$PRODUCT_NAME/$VERSION/$RESOURCE.yaml. For example, this override exists for the product assuredworkloads resource workload at beta version.
Override files contain information on how the Terraform representation of the resource should be different from the DCL's representation. This could be naming or behavior differences, but for a new resource implemented through tpgtools there should be no differences from the DCL's representation.
For a deeper understanding on test anatomy please read the accompanying Tests and Sample Anatomy
Create a meta.yaml file in the overrides directory for the resource. For example, this meta.yaml file exists for the assured workloads resource. This file will merge with any tests. You can customize behavior of the tests and examples generated dcl samples data (injections, hiding, ect..). See the section of the meta.yaml file for a more detailed dive.
Start by copying the relevant samples from the DCL for your new resource. These will be added to the tpgtools/api folder under the relevant product. These samples can be found under the samples/ folder within the DCL for the resource being added. For example, assured workloads can be found here.
Re-serialize serialization.go
to enable generating configs from samples by
running:
make serialize
In some cases you may need to add a non DCL test when either the current tests are insufficient or you want to showcase/test some specific behavior not present in the dcl tests.
If you need to write tests manually you can add terraform templates to the
relevant ./overrides/<product>/samples/<resource>
folder.
A terraform template test has the following anatomy.
<test-name>.yaml
- this is the test definition<test-name>.tf.tmpl
- this is the accompanying terraform configuration. A companion to the definition if you will.
The <test-name>.yaml
test specific configurations. For example it lists the
variables to replace in the template companion. You can also add additional
templates as updates. This will act as sequential applies and are useful for
testing update specific behavior. Make sure any templates added as an update has
the _update.tf.tmpl
extension.
The following is an example test definition.
updates:
- resource: ./basic_update.tf.tmpl
variables:
- name: "name"
type: "resource_name"
- name: "region"
type: "region"
The <test-name>.tf.tmpl
file is simply a terraform configuration. Any
replacements should be surrounding by double brackets {{ }}
. The variable name
from the test definition will be used to key into and replace these.
Provided you have added samples for the resource, documentation will be automatically generated based on the resource description in the DCL and examples will be generated from the samples for this resource. If you did not provide samples for the resource, then documentation will need to be written by hand.
Sometimes you may need to test unusual resource behavior in a way that does not fit well with generated tests. In this circumstance you can write a handwritten test file and add it here.
These tests can be used for more granular testing of specific behavior and add custom checks. Tests in these files will not have examples generated for them, so handwritten tests should not be considered a replacement for samples.
tpgtools
builds resources using Go Templates, with the templates stored under
the templates/
directory. They're fed the Resource
type, which contains
resource-level metadata and a list of Property
types which represent top-level
fields in a Terraform resource. Additionally, Property
s can contain other
Property
objects to create a nested structure of fields.
main.go
parses the OpenAPI schemas you've selected into Go structs and then
transforms them into Resource
/Property
before running them through Go
Templates.
Overrides are specified per-resource, with a directory structure parallel to the
OpenAPI specs. Inside each resource file is an unordered array of overrides made
up of an override type (like CUSTOM_DESCRIPTION
or VIRTUAL_FIELD
) as well as
a field they affect (if a field is omitted, they affect the resource) and an
optional details
object that will be parsed for additional metadata.
For example, override entries will look like the following:
- type: CUSTOM_DESCRIPTION
field: lifecycle.rule.condition.age
details:
description: Custom description here.
Adding samples is essential for generating tests and documentation. In fact Documentation won't generate without it!
Tests come from two sources.
- The top level api (
./api
) folder. If you look in here you'll see a bunch of yaml files and json files. These are DCL tests! Forked from the dcl library. - The override folder
(
./overrides/<product>/samples/<resource>
). This containsmeta.yaml
a file used for managing resource-wide test configurations and custom non-dcl tests.
In either case, DCL or non-DCL, every test definition is a yaml file which takes in variables.
variables:
- name: "name"
type: "resource_name"
- name: "org_id"
type: "org_id"
type
is inferred from sample.go
's translation map to figure out what needs
to be placed in the field. name
is used for substitution in the file itself
and in the case of resource_name
, actually used to create the value itself.
In the ./overrides/<product>/samples/<resource>
a meta.yaml
file exists
which controls configuration which applies to multiple tests or hiding/showing
specific tests.
If you need to hide sample from a doc or disable a sample from the tests you can do so here.
doc_hide:
- basic.tf.tmpl
- full.tf.tmpl
test_hide:
- basic_workload.yaml
If you want to add additional rules the following options are currently supported within meta.yaml
ignore_read:
- "billing_account"
- "kms_settings"
- "resource_settings"
- "provisioned_resources_parent"
check:
- deleteAssuredWorkloadProvisionedResources(t)
extra_dependencies:
- "time"
- "log"
- "strconv"
- "github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
code_inject:
- delete_assured_workload_provisioned_resources.go
doc_hide:
- basic.tf.tmpl # basic_update.tf.tmpl auto hides
- full.tf.tmpl
test_hide:
- basic_workload.yaml