r10k, invoked by r10k::webhook, does not update the module code

[posledov]

Affected software versions/distributions

• Distribution: Centos 7
• r10k version: 3.3.0
• r10k-puppet module version: 7.0.0
• Git provider: self-hosted Gitea instance

Manifest for deploying r10k

class { 'r10k':
  remote => 'git@git.server.lan:puppet/control-repo.git',
}

file { '/usr/local/bin/prefix_command.rb':
  ensure => file,
  mode   => '0755',
  owner  => 'root',
  group  => '0',
  source => 'puppet:///modules/r10k/prefix_command.rb',
}

class { 'r10k::webhook::config':
  use_mcollective => false,
  prefix          => true,
  prefix_command  => '/usr/local/bin/prefix_command.rb',
  enable_ssl      => false,
  protected       => false,
  notify          => Service['webhook'],
  require         => File['/usr/local/bin/prefix_command.rb'],
}

# this exposes http[s]://0.0.0.0:8088/<payload|module> for git webhooks
class { 'r10k::webhook':
  require         => Class['r10k::webhook::config'],
  use_mcollective => false,
  user            => 'root',
  group           => 'root',
}

How to reproduce

Environment already deployed

root@puppet ~ # cat /etc/puppetlabs/code/environments/feature_sysctl/Puppetfile

forge 'https://forge.puppet.com'

mod 'sysctl',
  :git    => 'git@git.server.lan:puppet/sysctl.git',
  :branch => 'dev'

Module deployed too

root@puppet ~ # git --git-dir=/etc/puppetlabs/code/environments/feature_sysctl/modules/sysctl/.git/ status HEAD

# HEAD detached at 88b2039
nothing to commit, working directory clean

Push some changes to module repository

user@laptop ~/puppet/sysctl/ $ git log --pretty=format:'%h' -n 1

88b2039

user@laptop ~/puppet/sysctl/ $ date > test ; git commit -am "some changes" ; git push

[dev 432c2f4] some changes
 1 file changed, 1 insertion(+), 1 deletion(-)
Counting objects: 3, done.
...

Module code not changed/updated

root@puppet ~ # git --git-dir=/etc/puppetlabs/code/environments/feature_sysctl/modules/sysctl/.git/ status HEAD

# HEAD detached at 88b2039
nothing to commit, working directory clean

What behaviour did you expect instead

Module code must be updated

root@puppet ~ # git --git-dir=/etc/puppetlabs/code/environments/feature_sysctl/modules/sysctl/.git/ status HEAD

# HEAD detached at 432c2f4
nothing to commit, working directory clean

Output log

root@puppet ~ # tail -fn0 /var/log/webhook/access.log

[2019-06-20 xx:xx:xx] DEBUG accept: 192.168.xx.xx:39760
[2019-06-20 xx:xx:xx] DEBUG Rack::Handler::WEBrick is invoked.
[2019-06-20 xx:xx:xx] INFO  Deploying module puppet/sysctl in the background.
[2019-06-20 xx:xx:xx] INFO  message: triggered: umask 0022; r10k deploy module puppet/sysctl module_name: puppet/sysctl
[2019-06-20 xx:xx:xx] DEBUG close: 192.168.xx.xx:39760
^C

Any additional information you'd like to impart

Request/response details (Gitea webhook)

Request

Headers

Request URL: http://puppet.server.lan:8088/module
Request method: POST
Content-Type: application/json
X-GitHub-Delivery: 9486c496-1766-4711-b8b3-e27bd353437b
X-GitHub-Event: push
X-Gitea-Delivery: 9486c496-1766-4711-b8b3-e27bd353437b
X-Gitea-Event: push
X-Gogs-Delivery: 9486c496-1766-4711-b8b3-e27bd353437b
X-Gogs-Event: push
  

Content

{
  "secret": "",
  "ref": "refs/heads/dev",
  "before": "88b2039b301ae54ea03b55e56c7efcebc8bfae30",
  "after": "432c2f475577dca05a5a8a5830240735cd97b51b",
  "compare_url": "https://git.server.lan/puppet/sysctl/compare/88b2039b301ae54ea03b55e56c7efcebc8bfae30...432c2f475577dca05a5a8a5830240735cd97b51b",
  "commits": [
    {
      "id": "432c2f475577dca05a5a8a5830240735cd97b51b",
      "message": "some changes\n",
      "url": "https://git.server.lan/puppet/sysctl/commit/432c2f475577dca05a5a8a5830240735cd97b51b",
      "author": {
        "name": "User Name",
        "email": "user@mail.lan",
        "username": "user"
      },
      "committer": {
        "name": "User Name",
        "email": "user@mail.lan",
        "username": "user"
      },
      "verification": null,
      "timestamp": "2019-06-20Txx:xx:xx+03:00",
      "added": null,
      "removed": null,
      "modified": null
    }
  ],
  "head_commit": null,
  "repository": {
    "id": 46,
    "owner": {
      "id": 3,
      "login": "puppet",
      "full_name": "",
      "email": "",
      "avatar_url": "https://git.server.lan/avatars/c5ba0c19f190c3008eee743a9ae472d6",
      "language": "",
      "is_admin": false,
      "username": "puppet"
    },
    "name": "sysctl",
    "full_name": "puppet/sysctl",
    "description": "",
    "empty": false,
    "private": true,
    "fork": false,
    "parent": null,
    "mirror": false,
    "size": 98,
    "html_url": "https://git.server.lan/puppet/sysctl",
    "ssh_url": "git@git.server.lan:puppet/sysctl.git",
    "clone_url": "https://git.server.lan/puppet/sysctl.git",
    "website": "",
    "stars_count": 0,
    "forks_count": 0,
    "watchers_count": 2,
    "open_issues_count": 0,
    "default_branch": "master",
    "archived": false,
    "created_at": "2019-06-19Txx:xx:xx+03:00",
    "updated_at": "2019-06-20Txx:xx:xx+03:00",
    "permissions": {
      "admin": false,
      "push": false,
      "pull": false
    }
  },
  "pusher": {
    "id": 2,
    "login": "user",
    "full_name": "",
    "email": "user@mail.lan",
    "avatar_url": "https://git.server.lan/avatars/3c2d8ed771b100ee780be50f52c11de8",
    "language": "ru-RU",
    "is_admin": true,
    "username": "user"
  },
  "sender": {
    "id": 2,
    "login": "user",
    "full_name": "",
    "email": "user@mail.lan",
    "avatar_url": "https://git.server.lan/avatars/3c2d8ed771b100ee780be50f52c11de8",
    "language": "ru-RU",
    "is_admin": true,
    "username": "user"
  }
}
  

Response [202]

Headers

Connection: Keep-Alive
Content-Length: 42
Content-Type: text/html;charset=utf-8
Date: Thu, 20 Jun 2019 xx:xx:xx GMT
Server: WebHook
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
  

Body

{"status":"accepted","message":"accepted"}
  

Manual run r10k

The command is the same as in the webhook's log:
root@puppet ~ # r10k deploy module puppet/sysctl --verbose=debug2

output

[2019-06-20 17:30:48 - DEBUG2] Reading configuration from "/etc/puppetlabs/r10k/r10k.yaml"
[2019-06-20 17:30:48 - DEBUG1] Testing to see if feature pe_license is available.
[2019-06-20 17:30:48 - DEBUG2] Attempting to load library 'pe_license' for feature pe_license
[2019-06-20 17:30:48 - DEBUG2] Error while loading library pe_license for feature pe_license: cannot load such file -- pe_license
[2019-06-20 17:30:48 - DEBUG1] Feature pe_license is not available.
[2019-06-20 17:30:48 - DEBUG2] pe_license feature is not available, PE only Puppet modules will not be downloadable.
[2019-06-20 17:30:48 - DEBUG1] Testing to see if feature shellgit is available.
[2019-06-20 17:30:48 - DEBUG2] Evaluating proc # to test for feature shellgit
[2019-06-20 17:30:48 - DEBUG2] Proc # for feature shellgit returned "/usr/bin/git"
[2019-06-20 17:30:48 - DEBUG1] Feature shellgit is available.
[2019-06-20 17:30:48 - DEBUG1] Testing to see if feature shellgit is available.
[2019-06-20 17:30:48 - DEBUG2] Evaluating proc # to test for feature shellgit
[2019-06-20 17:30:48 - DEBUG2] Proc # for feature shellgit returned "/usr/bin/git"
[2019-06-20 17:30:48 - DEBUG1] Feature shellgit is available.
[2019-06-20 17:30:48 - DEBUG1] Setting Git provider to R10K::Git::ShellGit
[2019-06-20 17:30:48 - DEBUG1] Testing to see if feature shellgit is available.
[2019-06-20 17:30:48 - DEBUG2] Evaluating proc # to test for feature shellgit
[2019-06-20 17:30:48 - DEBUG2] Proc # for feature shellgit returned "/usr/bin/git"
[2019-06-20 17:30:48 - DEBUG1] Feature shellgit is available.
[2019-06-20 17:30:48 - DEBUG1] Setting Git provider to default provider shellgit
[2019-06-20 17:30:48 - DEBUG2] Starting process: ["git", "--git-dir", "/opt/puppetlabs/puppet/cache/r10k/git@git.server.lan-puppet-control-repo.git", "for-each-ref", "refs/heads", "--format", "%(refname)"]
[2019-06-20 17:30:48 - DEBUG2] Finished process:
Command: git --git-dir /opt/puppetlabs/puppet/cache/r10k/git@git.server.lan-puppet-control-repo.git for-each-ref refs/heads --format %(refname)
Stdout:
refs/heads/feature/sysctl
refs/heads/production
Exit code: 0
[2019-06-20 17:30:48 - WARN] Environment "feature/sysctl" contained non-word characters, correcting name to feature_sysctl
[2019-06-20 17:30:48 - INFO] Using Puppetfile '/etc/puppetlabs/code/environments/feature_sysctl/Puppetfile'
[2019-06-20 17:30:48 - INFO] Using Puppetfile '/etc/puppetlabs/code/environments/production/Puppetfile'
[2019-06-20 17:30:48 - DEBUG1] Updating modules # in environment /etc/puppetlabs/code/environments/feature_sysctl
[2019-06-20 17:30:48 - DEBUG1] Only updating modules #, skipping module sysctl
[2019-06-20 17:30:48 - DEBUG1] Updating modules # in environment /etc/puppetlabs/code/environments/production
  

By specifying the module name "sysctl" instead of "puppet/sysctl", the result is correct: the module code will be updated.

[0x6d617474]

If you name the git repository in form of something-module, then it works as you expect.

For instance, given a module named "data", if we name the repository foobar-data, then the webhook output is:

[2019-06-20 16:48:48] DEBUG accept: 152.1.0.44:60000
[2019-06-20 16:48:48] DEBUG Rack::Handler::WEBrick is invoked.
[2019-06-20 16:48:48] INFO  Deploying module data in the background.
[2019-06-20 16:48:48] DEBUG close: 152.1.0.44:60000
[2019-06-20 16:48:52] INFO  message: triggered: umask 0022; r10k deploy module data module_name: data

and the module is updated as expected.

This has to do with how the module name is parsed off of the repository name.
https://github.com/voxpupuli/puppet-r10k/blob/master/templates/webhook.bin.erb#L118

The module_name parsing expects something in the form of foo-bar. If you don't have a dash in your repository name, then the entire string is used as the module name.

If the name value was used, this actually isn't an issue, since the name will match the module name; however, if full_name is set, then that value is used instead of name, and the full_name value is ownername/reponame, which creates an invalid module name.

Alternatively, there could be a check to see if the full_name contained a dash, and if not then use the last section of the value split by slash.

[posledov]

@0x6d617474, you are absolutely right.
Thanks a lot!

[rnelson0]

So I understand this clearly, is the issue the name set in the metadata.json of the module? Otherwise, the repro case looks like it states just sysctl where I would expect it.

[0x6d617474]

Given a module named sysctl, hosted at github.com/example/sysctl.git, the values passed in the webhook are as follows:

full_name: example/sysctl
name: sysctl

When those values are parsed by the webhook, it checks to see if full_name exists first, and if so it uses it after running it through a regex replacement. The replacement strips everything before the first dash in the name. Since there is no dash in the name, the result is the value example/sysctl. This value is passed down to r10k, which correctly responds that there is no module named example/sysctl, and bails.

If instead the git repo is named github.com/example/foobar-sysctl.git, then when the regex replacement happens on the full_name value, it strips everything before the dash leaving sysctl, which is the correct name of the module.

The regex for full_name needs to be corrected to account for the case where the repository name does not contain a dash, or the name value should be used instead. The full_name value was added to support BitBucket, but I don't know enough about how BitBucket webhook payloads are structured to know if these changes will break anything there.

[posledov]

The result of the current version – https://regex101.com/r/tl7Kzc/2

Slightly modified version – https://regex101.com/r/2ulz1X/2

What do you think?

[0x6d617474]

I like it, but we should note the change in behavior for cases where there are multiple dashes in the URL, as it has the possibility to break existing deployments due to the change in behavior.

[posledov]

Current regex truncates everything till the last dash in the full_name, but only if there is a dash[es] — https://regex101.com/r/tl7Kzc/2 :

owner/module-multiple-dashes => dashes
but
owner/module_wo_dash => owner/module_wo_dash

Slightly modified version – https://regex101.com/r/2ulz1X/2

This version truncates everything till the first dash, otherwise — till the (last) slash (/):
owner/module-multiple-dashes => multiple-dashes
owner/module_wo_dash => module_wo_dash

Once more modified version — https://regex101.com/r/erVwT6/2
It truncates everything till the last dash, else — till the (last) slash (/):

owner/module-multiple-dashes => dashes
owner/module_wo_dash => module_wo_dash

P.S.:

cases where there are multiple dashes in the URL

Could you give an example of such cases? That would help a lot when writing the right expression. Thanks.

[0x6d617474]

I meant full_name instead of "URL".