Vulnerability Exception 'approve' function applies vulnerability exception but throws a ruby exception

[derpadoo]

When using the exception approved function (http://www.rubydoc.info/gems/nexpose/5.3.1/Nexpose/VulnException#approve-instance_method), an exception is approved, but takes about 60 seconds, and then returns a message:

[-] Reason: Action failed: The state Approved is not allowed against the action APPROVE

Expected Behavior

The function used to take only a few seconds and did not through the ruby exception, but a change was noticed around 4/5-7 2017.

Current Behavior

Returns the message below, even though the exception is approved:

[-] Reason: Action failed: The state Approved is not allowed against the action APPROVE

Possible Solution

It appears one of the updates 6.4.30, 6.4.31, or 6.4.31 (https://help.rapid7.com/nexpose/en-us/release-notes/index.html) changed something.

Steps to Reproduce (for bugs)

Ruby code that reproduces the issue:

exc = Nexpose::VulnException.new(vuln_id, Nexpose::VulnException::Scope::SPECIFIC_INSTANCE_OF_SPECIFIC_ASSET, Nexpose::VulnException::Reason::OTHER)
exc.port = port
exc.asset_id = asset_id
exc.submitter = 'user1'
exc.reviewer = 'user1'

exception_id = exc.save(@nsc, "Ticket Number: #{ticket_number}") # Submitter_comment
success = exc.update_expiration_date(@nsc, expiration_date)
success = exc.approve(@nsc, "Auto-Approved by script, Ticket Number #{ticket_number}") # Reviewer_comment

rescue Nexpose::APIError => exp
  $logger.error "[-] Error when creating/saving the exception: #{exp} -- Code: #{exp.code} -- Req: #{exp.req} -- Reason: #{exp.reason}"
  puts "[-] Reason: #{exp.reason}"
end

Context

Trying to submit and approved vulnerability exceptions through the Nexpose API.

Your Environment

• Nexpose gem version:
  5.3.1

• Ruby version:
  ruby -v
  ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-linux]

• Operating System and version:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"
NAME="Ubuntu"
VERSION="14.04.5 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.5 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

• Nexpose product version:
  6.4.34

[sgreen-r7]

@derpadoo first off, thank you for taking the time and effort to submit this issue.

I think you're on to something here, but my first question to ask would be, are you seeing this happen when you're submitting exceptions using reasons other than Nexpose::VulnException::Reason::OTHER?

[derpadoo]

Changed reason to "ACCEPTABLE_RISK" and got the same result. It applies the exception, but takes about a minute, and throws the ruby exception

[-] Reason: Action failed: The state Approved is not allowed against the action APPROVE

exc = Nexpose::VulnException.new(vuln_id,
                                       Nexpose::VulnException::Scope::SPECIFIC_INSTANCE_OF_SPECIFIC_ASSET,
                                       Nexpose::VulnException::Reason::ACCEPTABLE_RISK)

[sgreen-r7]

@derpadoo perfect, thank you.

Will do some research then update this issue later today.

[sgreen-r7]

Alright, I think we're going to have to do some more intense debugging. Using your same steps to repro, I was unable to get either the delay with the approval time, and also did not receive the error message.

Just so we can remove my assumptions from my testing, can you provide what data you're using for those few variables you listed?

• vuln_id
• asset_id
• port
• expiration_date

Also, could you please run this command, and give me the approximate count?
@nsc.list_vuln_exceptions.count

[derpadoo]

@sgreen-r7 Appreciate you taking a look at this...

1.They are all type Strings:

vuln_id - type String, "cifs-smb-signing-not-required"
asset_id - type String, "2513"
port - type String, "445"
expiration_date - type String, "2017-10-06"

Here are the pastables put into the irb interpreter with trace, error message, and the ~60 seconds it takes to apply the exception:

asset_id = '77942'
port = '22058'
vuln_id = 'apache-httpd-cve-2012-0883'
ticket_number = 'VULN0019765'
expiration_date = '2017-10-17'

exc = Nexpose::VulnException.new(vuln_id, Nexpose::VulnException::Scope::SPECIFIC_INSTANCE_OF_SPECIFIC_ASSET, Nexpose::VulnException::Reason::ACCEPTABLE_RISK)

exc.port = port
exc.asset_id = asset_id
exc.submitter = 'api-script'
exc.reviewer = 'api-script'

exception_id = exc.save(@nsc, "Ticket Number: #{ticket_number}")
success = exc.update_expiration_date(@nsc, expiration_date)

puts "Current Time : " + Time.now.inspect
success = exc.approve(@nsc, "Auto-Approved by api-script, Ticket # #{ticket_number}")
puts "Current Time : " + Time.now.inspect

and the result

2.3.0 :034 > asset_id = '77942'
 => "77942"
2.3.0 :035 > port = '22058'
 => "22058"
2.3.0 :036 > vuln_id = 'apache-httpd-cve-2012-0883'
 => "apache-httpd-cve-2012-0883"
2.3.0 :037 > ticket_number = 'VULN0019765'
 => "VULN0019765"
2.3.0 :038 > expiration_date = '2017-10-17'
 => "2017-10-17"
2.3.0 :039 >
2.3.0 :040 >   exc = Nexpose::VulnException.new(vuln_id, Nexpose::VulnException::Scope::SPECIFIC_INSTANCE_OF_SPECIFIC_ASSET, Nexpose::VulnException::Reason::ACCEPTABLE_RISK)
 => #<Nexpose::VulnException:0x0000000224e490 @vuln_id="apache-httpd-cve-2012-0883", @scope="Specific Instance of Specific Asset", @reason="Acceptable Risk", @status=nil>
2.3.0 :041 >
2.3.0 :042 > exc.port = port
 => "22058"
2.3.0 :043 > exc.asset_id = asset_id
 => "77942"
2.3.0 :044 > exc.submitter = 'api-script'
 => "api-script"
2.3.0 :045 > exc.reviewer = 'api-script'
 => "api-script"
2.3.0 :047 > exception_id = exc.save(@nsc, "Ticket Number: #{ticket_number}")
 => 121517
2.3.0 :048 > success = exc.update_expiration_date(@nsc, expiration_date)
 => true
2.3.0 :049 >
2.3.0 :050 >   puts "Current Time : " + Time.now.inspect
Current Time : 2017-04-20 20:43:30 -0500
 => nil
2.3.0 :051 > success = exc.approve(@nsc, "Auto-Approved by api-script, Ticket # #{ticket_number}")
Nexpose::APIError: NexposeAPI: Action failed: The state Approved is not allowed against the action APPROVE
        from /home/derp/.rvm/gems/ruby-2.3.0/gems/nexpose-5.3.1/lib/nexpose/api_request.rb:146:in `execute'
        from /home/derp/.rvm/gems/ruby-2.3.0/gems/nexpose-5.3.1/lib/nexpose/connection.rb:118:in `execute'
        from /home/derp/.rvm/gems/ruby-2.3.0/gems/nexpose-5.3.1/lib/nexpose/vuln_exception.rb:219:in `approve'
        from (irb):51
        from /home/derp/.rvm/rubies/ruby-2.3.0/bin/irb:11:in `<main>'
2.3.0 :052 > puts "Current Time : " + Time.now.inspect
Current Time : 2017-04-20 20:44:32 -0500
 => nil
2.3.0 :053 >

2. Output of @nsc.list_vuln_exceptions.count

2.3.0 :053 > @nsc.list_vuln_exceptions.count
 => 38587

[derpadoo]

Sometimes these messages show up too:

[-] Reason: Action failed: FATAL: sorry, too many clients already


[-] Reason: Action failed: FATAL: remaining connection slots are reserved for non-replication superuser connections

[derpadoo]

So I think I figured out what's going on. When

success = exc.approve(@nsc, "Auto-Approved by api-script, Ticket # #{ticket_number}")`

is called, the exception is approved immediately according to the web gui, however, the script doesn't move on to the next exception. It waits 60 seconds to approve it, but it's already been approved, hence the error message The state Approved is not allowed against the action APPROVE. I believe that the 60 seconds is set somewhere else in the gem. If I change the approve function in lib/nexpose/vuln_exception.rb by adding a timeout: 1 to connection.execute, it is able to apply the exception much quicker, but now throws a new error

#Nexpose::APIRequest:0x0000001c055408 -- Reason: Action failed: Nexpose did not respond within 1 seconds.

This updated function works for our case, but I'm not sure what the correct solution for the gem/function is.

def approve(connection, comment = nil)
      xml = connection.make_xml('VulnerabilityExceptionApproveRequest',
                                { 'exception-id' => @id })
      if comment
        cxml = REXML::Element.new('comment')
        cxml.add_text(comment)
        xml.add_element(cxml)
        @reviewer_comment = comment
      end

      connection.execute(xml, '1.2', timeout: 1).success
    end

[sgreen-r7]

@derpadoo Yeah I'm inclined to agree with your conclusion, which is where I kinda ended up yesterday with some of my testing. I think right now your console might "have too many vuln exceptions", and the gem response/reaction time is underperforming. So the .save works fine, but when the .approve happens something like:

• submit .approve
• vuln exception approved
• <something takes 60seconds>
• nexpose-gem does a retry of submitting .approve
• error of the exception approved

I'm currently in the process of testing that theory specifically. I think we should be able to remedy this with extra options inside the gem. However, that probably won't be a today (Fri 4/21) thing, but more than likely a next week thing. I can help you some options to make your script run more smoothing with error rescuing for that duplicate approval error - if you would like?

That should help for the 'now' and then once we update the gem, we can revisit this error of the double approval, and the unresponsiveness of the gem for this action.

Again, thank you for the amount of logs/output you've provided thus far -- it's been extremely helpful. Let me know what you would like to do in the meantime.

[derpadoo]

@sgreen-r7 So our script is good enough for now and can work under the current configuration. For your awareness, we are also working this through the Rapid7 internal ticketing system. We are experiencing an issue now where we reach the postgresql max_connections limit (300 in our case) which generates the error message I previously posted.

[-] Reason: Action failed: FATAL: remaining connection slots are reserved for non-replication superuser connections

Running a process list shows 200-300 idle postgres processes which prevent further connections and degrade the web GUI. These processes linger for hours and I'm afraid to just kill them.

nxpgsql   6155  2636  0 Apr21 ?        00:00:00 postgres: admin_default nexpose 127.0.0.1(52108) idle
nxpgsql   6251  2636  0 Apr21 ?        00:00:00 postgres: admin_default nexpose 127.0.0.1(52143) idle
nxpgsql   6326  2636  0 Apr21 ?        00:00:00 postgres: admin_default nexpose 127.0.0.1(52170) idle
nxpgsql   6365  2636  0 Apr21 ?        00:00:00 postgres: admin_default nexpose 127.0.0.1(52184) idle
nxpgsql   6396  2636  0 Apr21 ?        00:00:00 postgres: admin_default nexpose 127.0.0.1(52190) idle
nxpgsql   6413  2636  0 Apr21 ?        00:00:00 postgres: admin_default nexpose 127.0.0.1(52194) idle
nxpgsql   6434  2636  0 Apr21 ?        00:00:00 postgres: admin_default nexpose 127.0.0.1(52201) idle
nxpgsql   6471  2636  0 Apr21 ?        00:00:00 postgres: admin_default nexpose 127.0.0.1(52213) idle
nxpgsql   6498  2636  0 Apr21 ?        00:00:00 postgres: admin_default nexpose 127.0.0.1(52224) idle
nxpgsql   6532  2636  0 Apr21 ?        00:00:00 postgres: admin_default nexpose 127.0.0.1(52233) idle
nxpgsql   6581  2636  0 Apr21 ?        00:00:00 postgres: admin_default nexpose 127.0.0.1(52246) idle

While running our script when no scans were going, I ran the command

watch 'netstat -nat | grep 5432 | wc -l'

and watched the count increase by 2 consistently until the database connections give out. It appears two of the functions below are making a database connection, but not closing it cleanly when it is finished so the connections keep stacking up.

exc.save
exc.update_expiration_date
exc.approve

I tried wrapping each vulnerability exception request by connecting to the API, applying the exception, and then disconnecting from the API, but that didn't help.

I can submit this as a separate issue if you want.

[sgreen-r7]

@derpadoo -- That db connection issue is something else, and that would be better serviced via our internal ticketing system.

I'll still go ahead and leave this issue open for now, as a reference to point back to, when we rollout our new features to allow for customizing timeouts and whatnot.

Feel free to make additional comments on this issue, if needed. We can troubleshoot any specific "gem issue" within github -- while any other general nexpose questions can be addressed within our internal ticketing system.
Thanks again for all your help and being a champ for providing all the logs/script data!

[EverythingShines]

I am also experiencing the same error as above:
C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/nexpose-6.1.1/lib/nexpose/api_request.rb:147:in execute': NexposeAPI: Action failed: The state Approved is not allowed against the action APPROVE (Nexpose::APIError) from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/nexpose-6.1.1/lib/nexpose/connection.rb:118:in execute'
from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/nexpose-6.1.1/lib/nexpose/vuln_exception.rb:282:in `approve'

I originally had this issue back in early June, but then it went away for awhile and is now back. I tried the timeout mentioned above by @derpadoo but that did not work for my case. Any help getting this working would be appreciated.

[sgreen-r7]

Try upgrade to >7.0.0 and editing the timeout.
https://github.com/rapid7/nexpose-client/releases/tag/v7.0.0

Feel free to comment on this issue, or reopen a new one if this is still happening after upgrading and setting a custom timeout.

[derpadoo]

@EverythingShines How many exceptions are in your database?

Just another data point...seeing similar behavior when using the Python client (https://github.com/rapid7/nexpose-client-python). That library has a timeout that can be set, so exceptions are approved quickly, but the idle postgresql connections don't go away and eventually render the web GUI inoperable. I don't think this is a client (Ruby or Python) issue, but more of a server/console issue. Approving exceptions through the web GUI takes ~15 to return after hitting approve as well. Same type of behavior with the exception being applied immediately, but then something is cranking away in the background until the web GUI exception page returns.

[EverythingShines]

We currently have a little over 5,000 total exceptions in our DB. Upgrading to 7.0.0 worked for me with the default value. I think I just wasn't setting the timeout high enough. When I was playing with it earlier, I initially set the timeout to 10 seconds, which was fine for the first two exceptions I wanted to make, which only had 30 or so instances across the environment. But as soon as I hit the third one, which had about 225 instances, it errored out. Once I removed that tweak and left the timeout at the default, the script completed successfully. Thank you!