Skip to content

Commit b078b95

Browse files
committed
Auto-Install of dependencies based on metadata.json info
Adding support to install modules from remote Puppet Forge, automatically, by reading contents of `metadata.json`. There are some configuration options added, like option to define `forge` address. Automatic installation is disabled by default to keep backward compatibility - in next major version it should be enabled by default. Adding installed module tree display after dependencies installation. Added more tests, including acceptance, so that rake spec using this module will be asserted.
1 parent 0797846 commit b078b95

File tree

15 files changed

+342
-93
lines changed

15 files changed

+342
-93
lines changed

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,4 @@
77
/pkg/
88
/spec/reports/
99
/tmp/
10+
Gemfile.local

.rubocop_todo.yml

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# This configuration was generated by
22
# `rubocop --auto-gen-config`
3-
# on 2018-06-17 09:41:41 +0200 using RuboCop version 0.49.1.
3+
# on 2019-06-28 22:09:48 +0200 using RuboCop version 0.49.1.
44
# The point is for the user to remove these configuration records
55
# one by one as the offenses are removed from the code base.
66
# Note that changes in the inspected code, or installation of new
@@ -23,7 +23,7 @@ Lint/EndAlignment:
2323
- 'lib/puppetlabs_spec_helper/rake_tasks.rb'
2424
- 'spec/watchr.rb'
2525

26-
# Offense count: 4
26+
# Offense count: 7
2727
Lint/HandleExceptions:
2828
Exclude:
2929
- 'lib/puppetlabs_spec_helper/puppet_spec_helper.rb'
@@ -43,17 +43,17 @@ RSpec/FilePath:
4343
- 'spec/unit/puppetlabs_spec_helper/tasks/beaker_spec.rb'
4444
- 'spec/unit/puppetlabs_spec_helper/tasks/fixtures_spec.rb'
4545

46-
# Offense count: 3
46+
# Offense count: 7
4747
# Configuration parameters: AssignmentOnly.
4848
RSpec/InstanceVariable:
4949
Exclude:
50-
- 'spec/acceptance/smoke_spec.rb'
50+
- 'spec/acceptance/spec_spec.rb'
5151

5252
# Offense count: 6
5353
RSpec/MultipleExpectations:
5454
Max: 3
5555

56-
# Offense count: 13
56+
# Offense count: 16
5757
RSpec/NamedSubject:
5858
Exclude:
5959
- 'spec/unit/puppetlabs_spec_helper/puppetlabs_spec/puppet_internals_spec.rb'
@@ -65,9 +65,10 @@ RSpec/VerifiedDoubles:
6565
Exclude:
6666
- 'spec/unit/puppetlabs_spec_helper/puppetlabs_spec/puppet_internals_spec.rb'
6767

68-
# Offense count: 2
68+
# Offense count: 3
6969
Security/Eval:
7070
Exclude:
71+
- 'Gemfile'
7172
- 'lib/puppetlabs_spec_helper/tasks/fixtures.rb'
7273

7374
# Offense count: 4
@@ -79,7 +80,7 @@ Style/ClassAndModuleChildren:
7980
- 'lib/puppetlabs_spec_helper/puppetlabs_spec/matchers.rb'
8081
- 'lib/puppetlabs_spec_helper/puppetlabs_spec/puppet_internals.rb'
8182

82-
# Offense count: 11
83+
# Offense count: 14
8384
Style/Documentation:
8485
Exclude:
8586
- 'spec/**/*'
@@ -88,8 +89,8 @@ Style/Documentation:
8889
- 'lib/puppetlabs_spec_helper/puppetlabs_spec/matchers.rb'
8990
- 'lib/puppetlabs_spec_helper/puppetlabs_spec/puppet_internals.rb'
9091
- 'lib/puppetlabs_spec_helper/tasks/beaker.rb'
91-
- 'lib/puppetlabs_spec_helper/tasks/fixtures.rb'
9292
- 'lib/puppetlabs_spec_helper/tasks/check_symlinks.rb'
93+
- 'lib/puppetlabs_spec_helper/tasks/fixtures.rb'
9394

9495
# Offense count: 1
9596
Style/DoubleNegation:
@@ -103,7 +104,7 @@ Style/GlobalVars:
103104
- 'lib/puppetlabs_spec_helper/puppet_spec_helper.rb'
104105
- 'lib/puppetlabs_spec_helper/puppetlabs_spec/files.rb'
105106

106-
# Offense count: 1
107+
# Offense count: 2
107108
# Configuration parameters: MinBodyLength.
108109
Style/GuardClause:
109110
Exclude:

.travis.yml

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,8 @@ matrix:
2929
env: PUPPET_GEM_VERSION='~> 3.0'
3030
- rvm: '2.0'
3131
env: PUPPET_GEM_VERSION='~> 3.0'
32-
- rvm: '1.9'
33-
env: PUPPET_GEM_VERSION='~> 3.0'
32+
- rvm: '2.0'
33+
before_install: rvm use $RVM --install
34+
env: PUPPET_GEM_VERSION='~> 3.0' RVM='1.9'
3435
notifications:
3536
email: false

Gemfile

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,9 @@ group :development do
1414
gem 'rubocop', '< 0.50'
1515
gem 'rubocop-rspec', '~> 1'
1616
end
17+
if Gem::Version.new(RUBY_VERSION.dup) >= Gem::Version.new('2.3.0')
18+
gem 'pry-byebug'
19+
end
1720
end
1821

1922
# json_pure 2.0.2 added a requirement on ruby >= 2. We pin to json_pure 2.0.1

README.md

Lines changed: 29 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -191,6 +191,14 @@ file named `.fixtures.yml` in the root of the project. You can specify a alterna
191191
You can use the `MODULE_WORKING_DIR` environment variable to specify a diffent location when installing module fixtures via the forge. By default the
192192
working directory is `<module directory>/spec/fixtures/work-dir`.
193193

194+
*Puppet Labs Spec Helper* supports installting modules from:
195+
196+
* SCM repositories via `repositories` key,
197+
* Forge repositories via `forge_modules` key,
198+
* Forge repositories from dependencies defined in `metadata.json` via `metadata` key.
199+
200+
For more details on how to use those setting see [examples](#fixtures-examples).
201+
194202
When specifying the repo source of the fixture you have a few options as to which revision of the codebase you wish to use, and optionally, the puppet versions where the fixture is needed.
195203

196204
* `repo` - the url to the repo
@@ -302,6 +310,15 @@ fixtures:
302310
ref: "2.6.0"
303311
```
304312

313+
Install modules based on dependencies from `metadata.json`:
314+
315+
```yaml
316+
fixtures:
317+
metadata:
318+
autoinstall_dependencies: true
319+
forge: https://puppetforge.acmecorp.lan # optional
320+
```
321+
305322
Pass additional flags to module installation:
306323

307324
```yaml
@@ -310,29 +327,29 @@ fixtures:
310327
stdlib:
311328
repo: "puppetlabs/stdlib"
312329
ref: "2.6.0"
313-
flags: "--module_repository https://my_repo.com"
314-
repositories:
315-
firewall:
316-
repo: "git://github.com/puppetlabs/puppetlabs-firewall"
317-
ref: "2.6.0"
318-
flags: "--verbose"
330+
flags: "--module_repository https://puppetforge.acmecorp.lan"
331+
repositories:
332+
firewall:
333+
repo: "git://github.com/puppetlabs/puppetlabs-firewall"
334+
ref: "2.6.0"
335+
flags: "--verbose"
319336
```
320337

321338
Use `defaults` to define global parameters:
322339

323340
```yaml
324341
defaults:
325342
forge_modules:
326-
flags: "--module_repository https://my_repo.com"
343+
flags: "--module_repository https://puppetforge.acmecorp.lan"
327344
fixtures:
328345
forge_modules:
329346
stdlib:
330347
repo: "puppetlabs/stdlib"
331348
ref: "2.6.0"
332-
repositories:
333-
firewall:
334-
repo: "git://github.com/puppetlabs/puppetlabs-firewall"
335-
ref: "2.6.0"
349+
repositories:
350+
firewall:
351+
repo: "git://github.com/puppetlabs/puppetlabs-firewall"
352+
ref: "2.6.0"
336353
```
337354

338355
Testing Parser Functions
@@ -389,7 +406,7 @@ environment variable``TEST_TIERS=high,medium``
389406
390407
By default ``TEST_TIERS`` only accepts low, medium and high as valid tiers. If you would like to use your own keywords to set the environment variable ``TEST_TIERS_ALLOWED``.
391408
392-
For example: to use the keywords dev, rnd, staging and production you can set
409+
For example: to use the keywords dev, rnd, staging and production you can set
393410
``TEST_TIERS_ALLOWED=dev,rnd,staging,production``. Then you would be able to run tests marked ``tier_dev => true``, ``tier_production => true`` with ``TEST_TIERS=dev,production``
394411
395412
Note, if the ``TEST_TIERS`` environment variable is set to empty string or nil, all tiers will be executed.

Rakefile

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,25 @@ def gem_present(name)
77
!Bundler.rubygems.find_name(name).empty?
88
end
99

10-
RSpec::Core::RakeTask.new(:spec) do |spec|
11-
spec.pattern = FileList['spec/**/*_spec.rb'].exclude('spec/fixtures/**/*_spec.rb')
10+
desc 'Runs unit tests'
11+
RSpec::Core::RakeTask.new(:'spec:unit') do |spec|
12+
spec.pattern = FileList['spec/**/*_spec.rb']
13+
.exclude('spec/fixtures/**/*_spec.rb')
14+
.exclude('spec/acceptance/**/*_spec.rb')
1215
end
1316

17+
desc 'Runs acceptance tests'
18+
RSpec::Core::RakeTask.new(:'spec:acceptance') do |spec|
19+
spec.pattern = FileList['spec/acceptance/**/*_spec.rb']
20+
end
21+
22+
Rake::Task[:spec].clear
23+
desc 'Runs all tests'
24+
task spec: [
25+
:'spec:unit',
26+
:'spec:acceptance',
27+
]
28+
1429
require 'yard'
1530
YARD::Rake::YardocTask.new
1631

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# Main module
2+
module PuppetlabsSpec; end
3+
# A Metadata JSON releated functions
4+
module PuppetlabsSpec::Metadata
5+
# This method returns an array of dependencies from the metadata.json file
6+
# in the format of an array of hashes, containing 'remote' (module_name) and
7+
# optionally 'ref' (version) elements. If no dependencies are specified,
8+
# empty array is returned
9+
def module_dependencies_from_metadata(metadata_opts)
10+
metadata = module_metadata
11+
return [] unless metadata.key?('dependencies')
12+
13+
opts = metadata_opts['opts']
14+
forge = if !opts.nil? && !opts['forge'].nil?
15+
opts['forge']
16+
else
17+
'https://forge.puppet.com/'
18+
end
19+
dependencies = []
20+
metadata['dependencies'].each do |dep|
21+
tmp = { 'remote' => dep['name'].sub('/', '-') }
22+
23+
if dep.key?('version_requirement')
24+
tmp['ref'] = module_version_from_requirement(
25+
tmp['remote'], dep['version_requirement'], forge
26+
)
27+
end
28+
dependencies.push(tmp)
29+
end
30+
31+
dependencies
32+
end
33+
34+
# This method uses the module_source_directory path to read the metadata.json
35+
# file into a json array
36+
def module_metadata
37+
metadata_path = "#{module_source_dir}/metadata.json"
38+
unless File.exist?(metadata_path)
39+
raise "Error loading metadata.json file from #{module_source_dir}"
40+
end
41+
JSON.parse(File.read(metadata_path))
42+
end
43+
44+
private
45+
46+
# This method takes a module name and the version requirement string from the
47+
# metadata.json file, containing either lower bounds of version or both lower
48+
# and upper bounds. The function then uses the forge rest endpoint to find
49+
# the most recent release of the given module matching the version requirement
50+
def module_version_from_requirement(mod_name, vr_str, forge_api)
51+
require 'net/http'
52+
forge_api = File.join(forge_api, '')
53+
uri = URI.parse("#{forge_api}v3/modules/#{mod_name}")
54+
req = Net::HTTP::Get.new(uri.request_uri)
55+
http = Net::HTTP.new(uri.host, uri.port)
56+
http.use_ssl = (uri.scheme == 'https')
57+
response = http.request(req)
58+
forge_data = JSON.parse(response.body)
59+
60+
vrs = version_requirements_from_string(vr_str)
61+
62+
# Here we iterate the releases of the given module and pick the most recent
63+
# that matches to version requirement
64+
forge_data['releases'].each do |rel|
65+
return rel['version'] if vrs.all? { |vr| vr.match?('', rel['version']) }
66+
end
67+
68+
raise "No release version found matching '#{vr_str}'"
69+
end
70+
71+
# This method takes a version requirement string as specified in the link
72+
# below, with either simply a lower bound, or both lower and upper bounds and
73+
# returns an array of Gem::Dependency objects
74+
# https://docs.puppet.com/puppet/latest/modules_metadata.html
75+
def version_requirements_from_string(vr_str)
76+
ops = vr_str.scan(%r{[(<|>|=)]{1,2}}i)
77+
vers = vr_str.scan(%r{[(0-9|\.)]+}i)
78+
79+
raise 'Invalid version requirements' if ops.count != 0 &&
80+
ops.count != vers.count
81+
82+
vrs = []
83+
ops.each_with_index do |op, index|
84+
vrs.push(Gem::Dependency.new('', "#{op} #{vers[index]}"))
85+
end
86+
87+
vrs
88+
end
89+
90+
# This is a helper for the self-symlink entry of fixtures.yml
91+
def module_source_dir
92+
Dir.pwd
93+
end
94+
end

0 commit comments

Comments
 (0)