Skip to content

Comments

NMAP test upgrade - service detection#935

Merged
noursaidi merged 19 commits intofaucetsdn:masterfrom
noursaidi:nmap
Dec 16, 2021
Merged

NMAP test upgrade - service detection#935
noursaidi merged 19 commits intofaucetsdn:masterfrom
noursaidi:nmap

Conversation

@noursaidi
Copy link
Collaborator

@noursaidi noursaidi commented Dec 10, 2021

This PR is still WIP but close to completion - submitting PR for any comments. Outstanding is just implementing CI tests and
and documentation. I'm also invesitgating if the HTTP test can be merged into this the services detection to save running two 65k port NMAP scans.

This PR maintains the current NMAP functionality and extends it by also check for disallowed services on all ports (test document here). in summary:

  • runs a nmap scan against all ports with service detection option (-sV) for TCP ports
  • runs an nmap scan against ports 69,161,162 for UDP and any other UDP port in module_config.json
  • For each TCP services(pop, telnet, ftp, imap, smtp), check if the service has been found in the NMAP scan or it's default port was open
  • For UDP services (tftp, SNMP), check if the default port was open
  • Output results for each test item (security.service.X)

The existing nmap scan has been retained for now (expected to be used within ATA) and is selected with a "services_scan": false

@noursaidi noursaidi requested review from grafnu and pisuke December 10, 2021 13:50
@codecov
Copy link

codecov bot commented Dec 10, 2021

Codecov Report

Merging #935 (b282a08) into master (e9e851a) will increase coverage by 0.17%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #935      +/-   ##
==========================================
+ Coverage   82.56%   82.73%   +0.17%     
==========================================
  Files          46       46              
  Lines        5862     5862              
==========================================
+ Hits         4840     4850      +10     
+ Misses       1022     1012      -10     
Flag Coverage Δ
ata 63.90% <ø> (+1.26%) ⬆️
aux 68.24% <ø> (ø)
base 66.51% <ø> (-0.02%) ⬇️
dhcp 67.50% <ø> (ø)
many 67.20% <ø> (-0.35%) ⬇️
mud 72.12% <ø> (-0.07%) ⬇️
switch 67.77% <ø> (-0.02%) ⬇️
topo 66.45% <ø> (-0.06%) ⬇️
unit 32.48% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
daq/acl_state_collector.py 82.19% <0.00%> (-1.37%) ⬇️
daq/host.py 91.35% <0.00%> (+0.28%) ⬆️
daq/runner.py 85.86% <0.00%> (+0.31%) ⬆️
daq/tcpdump_helper.py 80.95% <0.00%> (+2.38%) ⬆️
daq/stream_monitor.py 90.08% <0.00%> (+2.47%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e9e851a...b282a08. Read the comment docs.

@noursaidi
Copy link
Collaborator Author

noursaidi commented Dec 14, 2021

@grafnu @pisuke do you have any comments or questions on this PR? Except for CI fixes which I'm working my way though (test_mud may cause me some issues with unexpected long scan duration but I'm diagnosing that one - I haven't encountered very long NMAP module durations, usually <10 mins on my devices and much shorter on faux devices, but what I propose to deal with long scans if it is an issue:

  • splitting out the nmap port scan which is usually quick; and nmap service scan which takes longer depending on the number of ports found open;
  • setting a timeout for the services detection which is a function of the configured module expiry (say half of it)
  • if the service detection nmap scan times out, then set the security.services.X tests to skip or gone,
  • Still give a result for the security.nmap.ports scan) so the module results are still useful

Copy link
Collaborator

@grafnu grafnu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't worry about mud at all -- so if there's an easy way to remove/deprecate then that would be fine. The current mud stuff is all very very very very provisional and doesn't really do anything, I think.

if [ -n "${options[telnet]}" ]; then
echo Enabling mock telnet server...
(while true; do echo Telnet `hostname`; nc -nvlt -p 23 -e `which hostname`; done) &
while true; do echo -e "Telnet $(hostname)" | nc -l -w 1 23; done)&
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems like this would be a syntax error? ... stray )?

echo Running services on non standard ports and open default ports

echo Starting FTP 21514 and open default 20,21
nc -nvlt -p 20&
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

space before & (everywhere)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added spaces

(while true; do echo -e "* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN] Dovecot (Ubuntu) ready.\r\n" \
| nc -l -w 1 5361; done)&

#echo Starting TELNET and default port 23
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

restore or remove

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed and existing telnet faux option reinstated


echo Done with $image_name build.

docker build . -f subset/pentests/nmap/Dockerfile.test_nmap -t daqf/nmap No newline at end of file
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing newline

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was an accidental file change - modification removed

v.protocol == protocol and re.search(product, v.product)}


if __name__ == "__main__":
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's this for?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

redundant in this file - removed

@noursaidi
Copy link
Collaborator Author

Comments addressed - CI tests should be functional

mkdir -p local/site
# set nmap module to legacy nmap port based scan
cat resources/setups/common/base_config.json \
| jq '.modules .nmap .services_scan = false' \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit -- indent hanging lines 4 spaces

cmd/run -s interfaces.faux.opts=telnet device_specs=resources/device_specs/simple.json
echo DAQ result code $? | tee -a $TEST_RESULTS
cat inst/result.log | tee -a $TEST_RESULTS
head -20 inst/run-*/nodes/*/activate.log
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

potentially extra debugging statements

@noursaidi noursaidi merged commit 9d81f80 into faucetsdn:master Dec 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants