Adhoc scan subset of assets within a site, with a different scan template and scan engine #195

gschneider-r7 · 2015-11-12T23:07:52Z

I still need feedback on the method name. I added an and to it to be more English-y, but it's super long and I'm not a huge fan of it right now.

Changes include:

New scan method that imitates the "Scan Now" dialog in the Nexpose UI
Capture error responses from /data/ endpoints when they are plain text

Additional testing needed:

Ensure error handling doesn't negatively impact other methods' errors

Blurb:
This behaves like the "Scan Now" dialog in the UI where you can select a subset of a site's defined assets, a scan template, and a scan engine to scan them with. These settings are not persisted to a site configuration and allows multiple simultaneous scans on a single site.

Example usage:

nsc = Nexpose::Connection.new(...)
# Arbitrary site id for this example.
site_id = 1
# Default scan template.
template = 'full-audit-without-web-spider'
# Local scan engine is typically id 3 on newer installs, but can be 2 on older installs.
# Use Nexpose::Connection#list_engines and filter for the engine you want.
engine_id = 3
# Array of 1 or more strings, although passing in Nexpose::IPRange 
# and Nexpose::HostName objects also works correctly.
assets_to_scan = ['example.fqdn', '192.168.1.1', '10.1.5.0 - 10.1.5.60']

scan_id = nsc.scan_assets_with_template_and_engine(site_id, assets_to_scan, template, engine_id)

puts scan_id

Resolves #188

ben-git-hub99 · 2015-11-16T07:13:26Z

Naming could be scan_assets_full_options. I think it describes it and fits prior naming conventions.

ben-git-hub99 · 2015-11-16T07:30:49Z

I am getting the following error when using peach to allow for parallel processing:

C:/Ruby21/lib/ruby/2.1.0/rexml/parsers/baseparser.rb:340:in `pull_event': Missing end tag for 'script' (got "html") (REXML::ParseException)

Line: 282
Position: 106761
Last 80 unconsumed characters:
    from C:/Ruby21/lib/ruby/2.1.0/rexml/parsers/baseparser.rb:184:in `pull'
    from C:/Ruby21/lib/ruby/2.1.0/rexml/parsers/treeparser.rb:22:in `parse'
    from C:/Ruby21/lib/ruby/2.1.0/rexml/document.rb:287:in `build'
    from C:/Ruby21/lib/ruby/2.1.0/rexml/document.rb:44:in `initialize'
    from C:/Ruby21/lib/ruby/gems/2.1.0/gems/nexpose-2.2.0.pre/lib/nexpose/scan_template.rb:56:in `new'
    from C:/Ruby21/lib/ruby/gems/2.1.0/gems/nexpose-2.2.0.pre/lib/nexpose/scan_template.rb:56:in `initialize'
    from C:/Ruby21/lib/ruby/gems/2.1.0/gems/nexpose-2.2.0.pre/lib/nexpose/scan_template.rb:519:in `new'
    from C:/Ruby21/lib/ruby/gems/2.1.0/gems/nexpose-2.2.0.pre/lib/nexpose/scan_template.rb:519:in `load'
    from C:/Users/../scan.rb:78:in `remediate'
    from C:/Users/../jira-search.rb:82:in `block in <main>'
    from C:/Ruby21/lib/ruby/gems/2.1.0/gems/schleyfox-peach-0.3/lib/peach.rb:18:in `block (3 levels) in peach'
    from C:/Ruby21/lib/ruby/gems/2.1.0/gems/schleyfox-peach-0.3/lib/peach.rb:17:in `each'
    from C:/Ruby21/lib/ruby/gems/2.1.0/gems/schleyfox-peach-0.3/lib/peach.rb:17:in `block (2 levels) in peach'

Specifically, $data['issues'].pmap(10){|issue| scan_remediations(issue)} where scan_remediations is a custom function taking in array data from JIRA.

It's important to note the Rexml error does not occur until about 20 JIRA tickets in (pull data, kick off scan, evaluate results, update jira ticket, repeat).

I have tried to limit the processing to only 10 concurrent threads .pmap(10), but I think I am overwhelming the nsc still:

NexposeAPI: POST request to /data/site/1/scan failed. response body: The requested scan cannot run at this time. This site is in use and cannot be locked for this request.

^ This block will repeat multiple times and I am not sure how this condition is being met. The scans start and finish ok. However, maybe the scan_assets_with_template_and_engine cannot handle simultaneous requests and I need to sleep it?

Let me know how I can help here or if you need more information. Not looking for help with the 3rd party plugin, just curious on the Rexml error as it relates to Nexpose and also the NexposeAPI error.

gschneider-r7 · 2015-11-16T18:48:16Z

Yes there is currently a locking behavior with sites, I believe original intent was to prevent accidental simultaneous scans. You could sleep for a bit and/or retry after sleep since the lock shouldn't be very long. Although increased load on the console may potentially result in locks holding longer while waiting for the scan to start.

sgreen-r7 · 2015-11-16T20:54:16Z

lib/nexpose/scan.rb

+    # @param [Fixnum] scan_engine The scan engine ID.
+    # @return [Fixnum] Scan ID.
+    #
+    def scan_assets_with_template_and_engine(site_id, assets, scan_template, scan_engine)


per your name comment, i think that looks mostly okay but if makes more sense maybe we can include like either site or ad_hoc?

so like scan_adhoc_with_template_and_engine or scan_adhoc_site_with_template_and_engine.

but i don't know how much that helps in the way of clarity.

gschneider-r7 · 2015-11-16T21:23:15Z

As for the rexml error it seems that something caused a strange response when working with scan templates based on your output. Would have to inspect the actual output that rexml tried to parse or look at console logs to see what error might have been thrown.

ben-git-hub99 · 2015-11-16T21:31:38Z

How can I get you the output?

gschneider-r7 · 2015-11-16T21:50:37Z

You can PM me on community, email me (attachment or link to gist paste or something), or if absolutely needed you can go through support to get it to me.

ben-git-hub99 · 2015-12-01T07:37:06Z

Hi @gschneider-r7,

I apologize for the delay in response. I haven't had the ability to work on this for a little while.

It appears that the issue for RexML was related to Peach. I resolved it by using pmap. The #peach method was doing an iterative loop over and over again. The error occurred when attempting to transition an issue key already transitioned.

At this point, I can launch over 30 concurrent scans at the same time and they all finish. I don't have any additional bugs to report.

ben-git-hub99 · 2015-12-07T18:07:39Z

Hi @gschneider-r7,

It looks like the bundle install adds rex -v 2.0.9. This is not currently compatible based on what I am seeing. I have verified that uninstalling rex 2.0.9 resolves the issue.

If you have any other ideas, this is how I am requiring the version of Nexpose:

gem 'nexpose', '=2.1.3'
# comment here
require 'rest_client'
require 'nexpose'

This is the error I see when running my code in script.rb:

C:/Ruby21/lib/ruby/gems/2.1.0/gems/rex-2.0.9/lib/rex/random_identifier_generator.rb:36:in `<class:RandomIdentifierGenerator>': uninitialized constant Rex::Text (NameError)
    from C:/Ruby21/lib/ruby/gems/2.1.0/gems/rex-2.0.9/lib/rex/random_identifier_generator.rb:24:in `<top (required)>'
    from C:/Ruby21/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from C:/Ruby21/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from C:/Ruby21/lib/ruby/gems/2.1.0/gems/rex-2.0.9/lib/rex/powershell/payload.rb:2:in `<top (required)>'
    from C:/Ruby21/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from C:/Ruby21/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from C:/Ruby21/lib/ruby/gems/2.1.0/gems/rex-2.0.9/lib/rex/powershell.rb:2:in `<top (required)>'
    from C:/Ruby21/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from C:/Ruby21/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from C:/Ruby21/lib/ruby/gems/2.1.0/gems/rex-2.0.9/lib/rex/text.rb:6:in `<top (required)>'
    from C:/Ruby21/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from C:/Ruby21/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from C:/Ruby21/lib/ruby/gems/2.1.0/gems/rex-2.0.9/lib/rex/mime/header.rb:6:in `<class:Header>'
    from C:/Ruby21/lib/ruby/gems/2.1.0/gems/rex-2.0.9/lib/rex/mime/header.rb:4:in `<module:MIME>'
    from C:/Ruby21/lib/ruby/gems/2.1.0/gems/rex-2.0.9/lib/rex/mime/header.rb:3:in `<module:Rex>'
    from C:/Ruby21/lib/ruby/gems/2.1.0/gems/rex-2.0.9/lib/rex/mime/header.rb:2:in `<top (required)>'
    from C:/Ruby21/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from C:/Ruby21/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from C:/Ruby21/lib/ruby/gems/2.1.0/gems/rex-2.0.9/lib/rex/mime.rb:5:in `<module:MIME>'
    from C:/Ruby21/lib/ruby/gems/2.1.0/gems/rex-2.0.9/lib/rex/mime.rb:3:in `<module:Rex>'
    from C:/Ruby21/lib/ruby/gems/2.1.0/gems/rex-2.0.9/lib/rex/mime.rb:2:in `<top (required)>'
    from C:/Ruby21/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:126:in `require'
    from C:/Ruby21/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:126:in `require'
    from C:/Ruby21/lib/ruby/gems/2.1.0/gems/nexpose-2.1.3/lib/nexpose.rb:56:in `<top (required)>'
    from C:/Ruby21/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:73:in `require'
    from C:/Ruby21/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:73:in `require'
    from C:/Users/../Documents/Aptana Studio 3 Workspace/Git/branches/my_branch/script.rb:4:in `<main>'

gschneider-r7 · 2015-12-07T18:35:50Z

If you're still using my branch or pre-built gem it is not up to date with the Rex changes on the official releases. The official 2.1.3 and 2.2.0 versions set Rex to 2.0.8 to prevent this issue.

This behaves like the "Scan Now" dialog in the UI where you can select a subset of a site's defined assets, a scan template, and a scan engine to scan them with. These settings are not persisted to a site configuration and allows multiple simultaneous scans on a single site.

This makes it easier to identify mistakes when using the experimental adhoc scan method from this branch. Not tested against other UI endpoints used elsewhere in the gem.

This allows passing in an array with only 1 address in it and not having to worry about trailing commas.

ben-git-hub99 · 2015-12-07T18:46:52Z

ok. Thanks.

gschneider-r7 · 2015-12-07T18:48:41Z

I just rebased all of my branches so they should all require Rex 2.0.8 now.

Adhoc scan subset of assets within a site, with a different scan template and scan engine

gschneider-r7 added Feature Need Feeback labels Nov 12, 2015

sgreen-r7 reviewed Nov 16, 2015
View reviewed changes

gschneider-r7 added 4 commits December 7, 2015 10:39

Capture error messages from UI endpoints

3dc1680

This makes it easier to identify mistakes when using the experimental adhoc scan method from this branch. Not tested against other UI endpoints used elsewhere in the gem.

Only join when multiple addresses are provided

28cbd69

This allows passing in an array with only 1 address in it and not having to worry about trailing commas.

Rename method and update doc

4b24437

gschneider-r7 force-pushed the experimental_adhoc_scan branch from e808f14 to 4b24437 Compare December 7, 2015 18:40

sgreen-r7 added a commit that referenced this pull request Dec 10, 2015

Merge pull request #195 from gschneider-r7/experimental_adhoc_scan

83c76fb

Adhoc scan subset of assets within a site, with a different scan template and scan engine

sgreen-r7 merged commit 83c76fb into rapid7:master Dec 10, 2015

gschneider-r7 modified the milestone: 3.0 Dec 10, 2015

gschneider-r7 removed the Need Feeback label Dec 10, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adhoc scan subset of assets within a site, with a different scan template and scan engine #195

Adhoc scan subset of assets within a site, with a different scan template and scan engine #195

gschneider-r7 commented Nov 12, 2015

ben-git-hub99 commented Nov 16, 2015

ben-git-hub99 commented Nov 16, 2015

gschneider-r7 commented Nov 16, 2015

sgreen-r7 Nov 16, 2015

gschneider-r7 commented Nov 16, 2015

ben-git-hub99 commented Nov 16, 2015

gschneider-r7 commented Nov 16, 2015

ben-git-hub99 commented Dec 1, 2015

ben-git-hub99 commented Dec 7, 2015

gschneider-r7 commented Dec 7, 2015

ben-git-hub99 commented Dec 7, 2015

gschneider-r7 commented Dec 7, 2015

Adhoc scan subset of assets within a site, with a different scan template and scan engine #195

Adhoc scan subset of assets within a site, with a different scan template and scan engine #195

Conversation

gschneider-r7 commented Nov 12, 2015

ben-git-hub99 commented Nov 16, 2015

ben-git-hub99 commented Nov 16, 2015

gschneider-r7 commented Nov 16, 2015

sgreen-r7 Nov 16, 2015

Choose a reason for hiding this comment

gschneider-r7 commented Nov 16, 2015

ben-git-hub99 commented Nov 16, 2015

gschneider-r7 commented Nov 16, 2015

ben-git-hub99 commented Dec 1, 2015

ben-git-hub99 commented Dec 7, 2015

gschneider-r7 commented Dec 7, 2015

ben-git-hub99 commented Dec 7, 2015

gschneider-r7 commented Dec 7, 2015