Out of Memory Error on Large Adhoc Report #99
Comments
Code was needlessly parsing XML and HTML reports through REXML and Nokogiri. It will be faster and less memory intensive to just pass the string along, since #to_s was called on the resulting parse. If we need to validate the XML for reports, that should be done in testing.
Attempts to address #99 The generate method can now be passed with a "raw" flag to indicate that the response should not be parsed as XML for exception handing. Example usage: ``` config = AdhocReportConfig.new('audit-report', 'raw-xml-v2', site.id) report = config.generate(nsc, 3600, true) ```
|
@bk5115545 ... see if this addresses the issue with the larger files. An example usage for grabbing the raw data is: config = AdhocReportConfig.new('audit-report', 'raw-xml-v2', site.id)
report = config.generate(nsc, 3600, true)
File.write("export-#{site.id}.xml", report)If that works out, I'll keep it in there. Otherwise, I'll strip it out, as it adds complexity. |
|
It worked on a couple of reports however then this happened. Line 66 in sample.rb is the |
|
When that happens, you should be able to inspect the raw xml to see if there is an error in there. That error message should only happen because the response didn't contain the string 'success="1"' ... there may be a legitimate (non-Ruby) error in there. You can view the raw response by inspecting it on the connection: local_nexpose.response_xmlIf you haven't made any further calls to the nexpose console, that will contain the ad hoc report response. |
|
While I'm waiting on the application to crash again so I can inspect the XML, I was looking through the scan log provided by the web gui. I found the below quite frequently. 2014-12-30T21:36:29 [INFO] [Thread: VulnerabilityCheckContext.performTests] [Site: <SITENAME>] [<target_ip>:443] apache-httpd-cve-2007-6388 (apache-httpd-cve-2007-6388-mod_status-open-redir-exploit13) - ERROR - (Enable verbose logging for more information):
java.lang.NullPointerException
at com.rapid7.nexpose.plugin.vulnck.ValueTest.matches(Unknown Source)
at com.rapid7.nexpose.plugin.http.BaseHTTPCheckHandler$HTTPResponseTest.isHeaderMatch(Unknown Source)
at com.rapid7.nexpose.plugin.http.BaseHTTPCheckHandler$HTTPResponseTest.isHeadersMatch(Unknown Source)
at com.rapid7.nexpose.plugin.http.BaseHTTPCheckHandler$HTTPResponseTest.isMatch(Unknown Source)
at com.rapid7.nexpose.plugin.http.BaseHTTPCheckHandler.getResponseMatch(Unknown Source)
at com.rapid7.nexpose.plugin.http.HTTPCheckHandler$HTTPReqRespTest.isMatch(Unknown Source)
at com.rapid7.nexpose.plugin.http.HTTPCheckHandler$HTTPCheckTest.performCheck(Unknown Source)
at com.rapid7.nexpose.plugin.http.HTTPCheckHandler.handle(Unknown Source)
at com.rapid7.nexpose.plugin.BaseCheckContext.invokeTest(Unknown Source)
at com.rapid7.nexpose.nse.VulnerabilityCheckContext.performTests(Unknown Source)
at com.rapid7.nexpose.nse.VulnerabilityCheckContext.performTests(Unknown Source)
at sun.reflect.GeneratedMethodAccessor975.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.rapid7.thread.ThreadedCall.invokeCall(Unknown Source)
at com.rapid7.thread.ThreadedCall.execute(Unknown Source)
at com.rapid7.thread.ThreadedCallRunner.executeCall(Unknown Source)
at com.rapid7.thread.ThreadedCallRunner.run(Unknown Source)I know that it's not related to our changes with the gem but does this mean that the Nexpose console can't generate a report for sites with this error? I also know that this error could have been caused by my team writing a faulty vuln check but we have never written our own vuln checks. |
|
It's hard to tell exactly what that stacktrace is. It could be harmless, but we'd have to have more context to really get to the heart of it (i.e., a support case with logs, etc.). It might show up in a report card with the "Unknown" status, if the check fully failed and couldn't continue. Something in the scan log shouldn't affect what you are seeing in an export report, but there could be something about the data collected. |
|
Alright I have found the actual issue. The above stack trace isn't good but the issue was that there were too many connections to the Nexpose console internal postgres db and the db denied the connection request. It's in the logs on the Nexpose console and the timestamps nearly match. At that time, we had about 40 scans going on 40 different scan engines. I'll just catch that exception and retry after a long delay. The fix works great but some other issue got in the way of seeing it. |
|
OK. And, yeah, Nexpose (and the DB) will throttle activity if there's too much going on to protect itself. If you see something new, just reopen this issue or a new one. Thanks. |
|
As an aside, I am fairly certain that stack trace is hinting at a real, unrelated defect. I've filed it internally to get it tracked down (CONTENT-7079, internally) |
|
I just started getting this issue as well. I'm generating hundreds of ad hoc reports, is there a fix or do we just have some sort of timeout in between requests? |
|
@TheCaucasianAsian I tried adding a delay between my requests but it wasn't very effective. All my reports took varying times and a static delay didn't use the resources efficiently. I settled for the If you use |
|
Damn, right now I'm at the point where I just keep track of all the report names I try to generate, then loop back thru the directory to see which ones failed. I'm surprised this is still an issue. Thank you for the quick response I'll have to try out your suggestions! |
I have several huge reports to download and I have been having the console generate them after every scan. This approach works but I have to have enough disk space for the console to store them. My problem is that I don't have this much disk space to work with.
My solution is to switch from the console generating the report and then me downloading it to using the new
Connection::past_scansmethod andNexpose::AdhocReportConfig. I run my code in a cronjob that generates and downloads scans since it's last run. Seems simple enough.Minimal code snippit is below.
Commit 7903b8d was a great step since I can now download 200MB reports however I still crash on reports larger than 350MB. I have looked at making the reports smaller however I need every report to contain exactly 1 whole network site. Some of these sites have about 1000 RHSA updates that need to be applied and each of these updates contains way too much vulnerability information.
Stacktrace is below.
According to the stacktrace, the issue is actually in REXML however REXML developers already closed the issue with a status of "WILL NOT FIX" saying that it's a problem with Base64 encoding and XML.
As far as I can tell, the only reason the Nexpose gem is parsing the XML is to check the value of the success attribute on the root node. There is probably not a reason to have the gem parse the entire report if the report is larger than 32MB. I have a monkey patch that drops the header and footer and uses the Base64 module to get me the raw xml. This patch is really slow though since it causes the generic errors in APIRequest::execute to trigger retrying 5 times. I would rather help fix the issue upstream than throw 3 or 4 monkey patches into production.
The text was updated successfully, but these errors were encountered: