Skip to content

(MODULES-9428) Clarify composite namevar edgecases and add examples #140

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Jun 28, 2019
Merged

(MODULES-9428) Clarify composite namevar edgecases and add examples #140

Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
3cdf450
(maint) typos and phrasing
DavidS May 24, 2019
563c50c
(maint) Reformat `get` definition for easier readability
DavidS Jun 18, 2019
eb55e45
(maint) The `source` attribute is a parameter
DavidS Jun 18, 2019
25b809a
(maint) Add a example for `set` `changes` argument
DavidS Jun 18, 2019
a2f8049
(maint) Refactor the description of the SimpleProvider methods to hav…
DavidS Jun 18, 2019
c1a108a
(maint) Fix up and extend example for `canonicalize`
DavidS Jun 18, 2019
8df5350
(MODULES-9428) Clarify composite namevar edgecases and add examples
DavidS May 24, 2019
2b97229
(maint) remove noop parameter from remote_resource example
DavidS Jun 19, 2019
592d7c2
(MODULES-9428) Improvements from review
DavidS Jun 20, 2019
f99d267
(MODULES-9428) Make title_patterns mandatory
DavidS Jun 25, 2019
dfc520a
(maint) Minor update to `changes` example
da-ar Jun 28, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
151 changes: 111 additions & 40 deletions language/resource-api/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ Puppet::ResourceApi.register_type(
},
source: {
type: 'String',
behaviour: :parameter,
desc: 'Where to retrieve the key from, can be a HTTP(s) URL, or a local file. Files get automatically required.',
},
# ...
Expand Down Expand Up @@ -69,16 +70,15 @@ The `Puppet::ResourceApi.register_type(options)` function takes the following ke
* `autorequire`, `autobefore`, `autosubscribe`, and `autonotify`: a hash mapping resource types to titles. The titles must either be constants, or, if the value starts with a dollar sign, a reference to the value of an attribute. If the specified resources exist in the catalog, Puppet will create the relationsships requested here.
* `features`: a list of API feature names, specifying which optional parts of this spec the provider supports. Currently defined features: `canonicalize`, `simple_get_filter`, and `supports_noop`. See below for details.

For autoloading work, this code needs to go into `lib/puppet/type/<name>.rb` in your module.
For autoloading to work, this code needs to go into `lib/puppet/type/<name>.rb` in your module.

### Composite Namevars ("title_patterns")

Each resource being managed must be identified by a unique title. Usually this is fairly straightforward and a single attribute can be used to act as an identifier. Sometimes though, you need a composite of two attributes to uniquely identify the resource you want to manage.
Each resource being managed must be identified by a unique title. Usually this is fairly straightforward and a single attribute can be used to act as an identifier. Sometimes though, you need a composite of two or more attributes to uniquely identify the resource you want to manage.

If multiple attributes are defined with the `namevar` behaviour, the type SHOULD specify `title_patterns` that will tell Resource API how to get at the attributes from the title. If `title_patterns` is not specified a default pattern is applied, and matches against the first declared `namevar`.
If multiple attributes are defined with the `namevar` behaviour, the type must specify `title_patterns` that will tell Resource API how to get at the attributes from the title.


> Note: The order of title_patterns is important. You should declare the most specific pattern first and end with the most generic.
The `title_patterns` are evaluated in the order they are specified. Evaluation stops after the first match is achieved.

Each title pattern contains the:
* `pattern`, which is a ruby regex containing named captures. The names of the captures MUST be that of the namevar attributes.
Expand All @@ -96,7 +96,7 @@ Puppet::ResourceApi.register_type(
title_patterns: [
{
pattern: %r{^(?<package>.*[^-])-(?<manager>.*)$},
desc: 'Where the package and the manager are provided with a hyphen separator',
desc: 'Where the package and the manager are separated by a hyphen',
},
{
pattern: %r{^(?<package>.*)$},
Expand Down Expand Up @@ -127,19 +127,20 @@ Matches the first title pattern:
```puppet
# /etc/puppetlabs/code/environments/production/manifests/site.pp
software { php-yum:
ensure=>'present'
ensure => 'present',
}

software { php-gem:
ensure=>'absent'
ensure => 'absent',
}
```

Matches the second title pattern:
```puppet
# /etc/puppetlabs/code/environments/production/manifests/site.pp
software { php:
manager='yum'
ensure=>'present'
ensure => 'present',
manager => 'yum',
}
```

Expand Down Expand Up @@ -176,58 +177,109 @@ class Puppet::Provider::AptKey::AptKey
end
```

The `get` method reports the current state of the managed resources. It returns an enumerable of all existing resources. Each resource is a hash with attribute names as keys, and their respective values as values. It is an error to return values not matching the type specified in the resource type. If a requested resource is not listed in the result, it is considered to not exist on the system. If the `get` method raises an exception, the provider is marked as unavailable during the current run, and all resources of this type will fail in the current transaction. The exception message will be reported to the user.
The `get` method reports the current state of the managed resources. It returns an enumerable of all existing resources. Each resource is a hash with attribute names as keys, and their respective values as values.

* It is an error to return values not matching the type specified in the resource type.
* If a requested resource is not listed in the result, it is considered to not exist on the system.
* If the `get` method raises an exception, the provider is marked as unavailable during the current run, and all resources of this type will fail in the current transaction. The exception message will be reported to the user.
* If the type has more than one namevar, each resource hash must have a `:title` key with a single formatted string representing all namevar values. The title must match one of the title patterns. The namevar values computed from that pattern must match their counterparts in the hash.
> Note: This value allows the runtime environment to present resources in a way familiar to the user.

> Example: Referring back to the `software` example above, resources returned from the `get` method require a `title:` value matching the `package:` and `manager:` values:
> ```
> {
> title: 'php-yum',
> package: 'php',
> manager: 'yum',
> ensure: 'present',
> }
> ```

The `set` method updates resources to a new state. The `changes` parameter gets passed a hash of change requests, keyed by the resource's name. Each value is another hash with the optional `:is` and `:should` keys. At least one of the two has to be specified. The values will be of the same shape as those returned by `get`. After the `set`, all resources should be in the state defined by the `:should` values.

> Example for a `changes` request:
> ```
> changes = {
> 'foo' => {
> is: {
> name: 'foo',
> ensure: 'present',
> value: 'original value',
> },
> should: {
> name: 'foo',
> ensure: 'present',
> value: 'new value',
> },
> },
> }
> ```

A missing `:should` entry indicates that a resource should be removed from the system. Even a type implementing the `ensure => [present, absent]` attribute pattern still has to react correctly on a missing `:should` entry. `:is` may contain the last available system state from a prior `get` call. If the `:is` value is `nil`, the resources were not found by `get`. If there is no `:is` key, the runtime did not have a cached state available.

If a type has more than one namevar, the resource's name key is replaced by a hash of the namevars and their values:

> Example for a `changes` request with multiple namevars:
> ```
> changes = {
> { package: 'php', manager: 'yum' } => {
> is: {
> package: 'php',
> manager: 'yum',
> ensure: 'present',
> value: 'original value',
> },
> should: {
> package: 'php',
> manager: 'yum',
> ensure: 'present',
> value: 'new value',
> },
> },
> }
> ```

The `set` method should always return `nil`. Any progress signaling should be done through the logging utilities described below. If the `set` method throws an exception, all resources that should change in this call and haven't already been marked with a definite state, will be marked as failed. The runtime will only call the `set` method if there are changes to be made, especially in the case of resources marked with `noop => true` (either locally or through a global flag). The runtime will not pass them to `set`. See `supports_noop` below for changing this behaviour if required.

Both methods take a `context` parameter which provides utilties from the runtime environment, and is decribed in more detail there.
Both methods take a `context` parameter which provides utilities from the runtime environment, and is described in more detail in its own section below.

## Implementing simple providers

In many cases, the resource type follows the conventional patterns of puppet, and does not gain from the complexities around batch-processing changes. For those cases, the `SimpleProvider` class supplies a proven foundation that reduces the amount of code necessary to get going.

`SimpleProvider` requires that your type follows some common conventions:

* `name` is the name of your namevar attribute
* if the type has a single namevar attribute, it must be called `name`
* `ensure` attribute is present and has the `Enum[absent, present]` type

To start using `SimpleProvider`, inherit from the class like this:

```ruby
require 'puppet/resource_api/simple_provider'

# Implementation for the wordarray type using the Resource API.
# Implementation for the apt_key type using the Resource API.
class Puppet::Provider::AptKey::AptKey < Puppet::ResourceApi::SimpleProvider
# ...
```

Once all of that is in place, instead of the `set` method, the provider needs to implement the `create`, `update` or `delete` methods:

* `create(context, name, should)`: This is called when a new resource should be created.
* `context`: provides utilties from the runtime environment, and is decribed in more detail there.
* `name`: the name or hash of the new resource.
* `should`: a hash of the attributes for the new instance.

* `update(context, name, should)`: This is called when a resource should be updated.
* `context`: provides utilties from the runtime environment, and is decribed in more detail there.
* `name`: the name or hash of the resource to change.
* `should`: a hash of the desired state of the attributes.

* `delete(context, name)`: This is called when a resource should be deleted.
* `context`: provides utilties from the runtime environment, and is decribed in more detail there.
* `name`: the name or hash of the resource that should be deleted.

The parameters of these methods always carry the same values:

* `context`: provides utilities from the runtime environment, and is described in more detail in its own section below.
* `name`: the name (if there is only one namevar) or hash of namevars (if there are multiple) of the resource.
* `should`: a hash of the desired state of the attributes. This is not passed to the delete method.

The `SimpleProvider` takes care of basic logging, and error handling.

When a `type` has only a single namevar defined, `SimpleProvider` will pass the value of that attribute as `name` to the `create`, `update` and `delete` methods. If multiple namevars are defined, `SimpleProvider` will instead pass a hash. The hash contains the composite name of `title`, and all the namevars and their values, for example:
When a `type` has only a single namevar defined, `SimpleProvider` will pass the value of that attribute as `name` to the `create`, `update` and `delete` methods. If multiple namevars are defined, `SimpleProvider` will instead pass a hash. The hash contains all the namevars and their values, for example:

```
{
title: 'foo/bar',
name: 'bar',
group: 'foo',
}
Expand All @@ -252,11 +304,11 @@ Puppet::ResourceApi.register_type(
class Puppet::Provider::AptKey::AptKey
def canonicalize(context, resources)
resources.each do |r|
r[:name] = if r[:name].start_with?('0x')
r[:name][2..-1].upcase
else
r[:name].upcase
end
r[:id] = if r[:id].start_with?('0x')
r[:id][2..-1].upcase
else
r[:id].upcase
end
end
end
```
Expand All @@ -265,7 +317,21 @@ The runtime environment needs to compare user input from the manifest (the desir

The `canonicalize` method transforms its `resources` argument into the standard format required by the rest of the provider. The `resources` argument to `canonicalize` is an enumerable of resource hashes matching the structure returned by `get`. It returns all passed values in the same structure with the required transformations applied. It is free to reuse or recreate the data structures passed in as arguments. The runtime environment must use `canonicalize` before comparing user input values with values returned from `get`. The runtime environment always passes canonicalized values into `set`. If the runtime environment requires the original values for later processing, it protects itself from modifications to the objects passed into `canonicalize`, for example through creating a deep copy of the objects.

The `context` parameter is the same passed to `get` and `set`, which provides utilties from the runtime environment, and is decribed in more detail there.
The `context` parameter provides utilities from the runtime environment, and is described in more detail there. It is the same as is passed to `get` and `set`.

> Example: The `resources` parameter is an array of resources hashes:
> ```ruby
> resources = [
> {
> id: '0x12345678',
> source: '/net/gold/example.com/www/key1.gpg',
> },
> {
> id: '1234567812345678123456781234567812345678',
> source: '/net/gold/example.com/www/key2.gpg',
> },
> ]
> ```

> Note: When the provider implements canonicalization, it aims to always log the canonicalized values. As a result of `get` and `set` producing and consuming canonically formatted values, this is not expected to present extra cost.

Expand Down Expand Up @@ -304,17 +370,22 @@ Puppet::ResourceApi.register_type(
# lib/puppet/provider/apt_key/apt_key.rb
class Puppet::Provider::AptKey::AptKey
def get(context, names = nil)
[
{
name: 'name',
# ...
},
]
if names == nil
return all_instances
else
names.collect { |n| find_instance(n) }
end
end
```

Some resources are very expensive to enumerate. The provider can implement `simple_get_filter` to signal extended capabilities of the `get` method to address this. The provider's `get` method will be called with an array of resource names, or `nil`. The `get` method must at least return the resources mentioned in the `names` array, but may return more than those. If the `names` parameter is `nil`, all existing resources should be returned. The `names` parameter defaults to `nil` to allow simple runtimes to ignore this feature.

For types with multiple namevars, the `names` array will consist of hashes of the namevars and their values instead of simple values:

```ruby
[ { package: 'php', manager: 'yum' }, {package: 'mysql', manager: 'yum'} ]
```

The runtime environment calls `get` with a minimal set of names, and keeps track of additional instances returned to avoid double querying. To gain the most benefits from batching implementations, the runtime minimizes the number of calls into `get`.

### Provider feature: `supports_noop`
Expand Down Expand Up @@ -353,12 +424,12 @@ Puppet::ResourceApi.register_type(

# lib/puppet/provider/nx9k_vlan/nexus.rb
class Puppet::Provider::Nx9k_vlan::Nexus
def set(context, changes, noop: false)
def set(context, changes)
changes.each do |name, change|
is = change.has_key? :is ? change[:is] : get_single(name)
should = change[:should]
# ...
context.transport.do_something unless noop
context.transport.do_something
end
end
```
Expand Down