Connection Timeout When Creating DynamicAssetGroup with large number of criteria

[0xW1sKy]

When creating a dynamic asset group via the API with a large set of criteria, the connection appears to timeout waiting for a response from Nexpose causing the script window to close. However, the asset group does proceed to be created in Nexpose.

Expected Behavior

The expected behavior I would prefer would be one of two items.

1. something is done to keep the ruby window alive while waiting for the asset group to finish being created. When finished the process would end allowing the rest of a script to be carried out.

2. Something is returned to ruby notifying the user that the asset group is in the process of being created, allowing the user to know that all criteria were successfully passed to Nexpose.

Current Behavior

Criteria is passed to nexpose via ruby gem, while waiting on the asset group to finish it's creation, i believe the web connection times out and creates a terminal error in the script.

Possible Solution

keep-alives while waiting for asset group creation to finish.
or
confirmation asset group creation process was started, and return an asset group id? (although this doesn't appear to be available to nexpose until after the process is completely finished). in order for this to work, i assume much larger changes would end up having to be made on the backend. which would probably make this a non-viable solution.

Steps to Reproduce (for bugs)

I can't get to gist.github due to corporate proxy, but you should be able to access this:
https://github.com/0xW1sKy/Nexpose-Ruby-Gem/blob/master/CreateDAGfromFiles.rb

This creates the dynamic asset group using 2 txt files. 1 would contain a list of IP addresses (1 per line). The other would contain a list of hostnames (1 per line)

I encountered the error in an instance were we had 600 hostnames and 600 ip addresses.

Context

Main issue here is after the group.save(nsc) process is started, it times out with an error causing scripts to close. Nothing else is able to be ran after in the same script.
Also this creates confusion as it appears that asset group creation failed. First time I ran this script, i ended up creating a handful of different asset groups while attempting to diagnose the issue (running multiple times). Luckily after 2-3 attempts I happened to see the asset group creation process by using the 'show db act' command in the nexpose console.

Your Environment

• Nexpose gem version: 5.3.1
• Ruby version: Ruby 2.3.3p222 (2016-11-21 revision 56859) [x64-mingw32]
• Operating System and version: Windows 7 Enterprise SP1
• Nexpose product version: 6.4.34

[sgreen-r7]

@0xW1sKy --- Thanks for the heads up.

First question, about how long does it take to timeout? And just so I understand, looking at the file you linked, you're just creating one DAG, which contains 600 hostnames, and 600 IPs?

Thanks again.

[0xW1sKy]

Timeout occurs within a couple of minutes. I haven't timed it so I'm not sure on the exact time-frame, but it is short enough of a wait that if you're sitting and watching the script, you'll see it timeout before you get annoyed with waiting around. Pretty certain it's <5 minutes
Also, for clarification timeout from the web request is what I assume is the issue, I believe when running the same script manually via irb the errors it returns are http/web-socket related.

And yes, this is creating a single DAG, which contains 600 hostname criterion and 600 ip address criterion.

[sgreen-r7]

@0xW1sKy Thanks, also i forgot to ask initially, but has this been happening for a while, or is this only recently?

[0xW1sKy]

This script is on the newer side for us. created within the last 30 days. Any time we've put a large number of criteria we've seen it time out. I doubt that there is a upper limit to make the timeout occur. I'm assuming its just based on the amount of time it takes for nexpose to finish the backend query needed before the asset group can finish it's creation.

So, I can't really put any time frame on when it started happening as it has always happened to us.
But only attempts we have made were within the last 30 days.

[sgreen-r7]

@0xW1sKy -- cool, thank you for the feedback.

I'll investigate what I can, and see if we can remediate the issue via the gem by editing/adding the ability to configure timeouts for the gem requests.

Sidenote: Not to be confrontational/defensive, but this sounds like an "odd" way to use a DAG; and what you've listed sounds more inline of how we envisioned Static Asset Groups to be used. Since DAGs have the ability to include a range of IP addresses, does it make more sense to determine the overlapping ranges for the 600ips? Hostnames might be a different can of worms, but recently we added the ability to create a DAG via regex for Hostnames; so that might be a route to go down as well. I'm sure there's a reason how we ended up with the script in the current format, but I just wanted to make sure everybody was aware of the different ways to configure a DAG.

Either way, thank you for the additional information, and I'll continue to update this ticket when progress is made with the custom timeouts.

[0xW1sKy]

Dude, I promise you're preaching to the choir.

IPs in the scenario that lead to this script don't overlap ranges (or in most cases aren't even in the same subnet.), machine names don't follow any common naming convention, and depending on the scan, the results may return the DNS name or netbios name that more often then not do not match each other. (In an environment that has 30k+ machines.) ),:

This whole thing is a bandaid solution for issues on our side of the equation and luckily nexpose has enough functionality and flexibility to allow us to still be successful in our efforts to manage the environment.

And the regex option I do use extensively. It's been a godsend. Please thank whatever team was responsible for implementing that feature. In the places that it was possible to use, it made my life incredibly easier.

Thanks for your help on this issue and your quick feedback!

[sgreen-r7]

@0xW1sKy -- awesome to hear, I just wanted to make sure that we weren't working with a "inherited" solution from somebody/somewhere else.

I'll continue to update this issue when I get more data/info from our internal testing. And as always feel free to update/comment if any other shenanigans begin to manifest.

Thanks again!

[sgreen-r7]

Try upgrade to >7.0.0 and editing the timeout.
https://github.com/rapid7/nexpose-client/releases/tag/v7.0.0

Feel free to comment on this issue, or reopen a new one if this is still happening after upgrading and setting a custom timeout.