security.services.http Sporadic Missing HTTP Requests

[duncangreene]

Describe the bug
Sporadically (haven't been able to deduce how often), some HTTP Requests (I think just HTTP 1.1) are seen missing from the services.pcap.

This causes a false "No HTTP server found" to be recorded in the report, despite the DUT running a HTTP server.

To Reproduce
Steps to reproduce the behavior:

1. Run services test.

Expected behavior
All HTTP Requests should be present.

Error logs
allhttprequests.zip
missinghttprequests.zip

Environment (please provide the following information about your setup):

• OS: Ubuntu
• Version 24.04.1
• Testrun 2.0

Additional context
DUT is https://www.ismacontrolli.com/en/isma-b-mix18-ip.html

[jhughesoti]

The presence of HTTP traffic does not indicate an HTTP server running on the device. For that first example you provided, the traffic detected is to port 502, which is modbus traffic. I confirmed modbus traffic is present on the report in the protocol tests.

In the second example, confirming with the report, it does detect the HTTP server being open. This could be a timing issue with the device in question and the testing steps. i.e., example one tested services module before the device was in a steady state where the web server was ready vs the second where it was.

Any additional steps for testing to produce this would be helpful in determining the root cause.

[duncangreene]

Aha, I hadn't registered the port number in the first 3 telegrams.

The issue appears to be the absence of the outgoing requests from Testrun (10.10.10.9) to port 80 in the first example. What would cause Testrun to not send these outgoing requests?

[jhughesoti]

The services module uses an nmap scan of the ports of interest so there should be no difference between one run and the next. I suspect the device is not responding to that process when you are seeing this which would indicate a different testing procedure sequence between the tests (stable state vs non-stable state) or inconsistent behavior of the device. Without having the full logs and testing procedure for each outcome, it's difficult to say exactly what the reason for this result is.

[duncangreene]

The original post contains the two test .zip outputs, if the log files you're after are included in there? Let me know if there are other logs etc. to include.

I can see from the services.pcap in missinghttprequests.zip that the DUT doesn't SYN ACK to the port 80 SYN, however this device does indeed run a bona fide web server as part of its 'normal' operation (and as evidenced in the respective services.pcap in allhttprequests.zip).

My first (late night) thought, with an eye on manufacturers potentially implementing minimum viable resources in embedded devices such as these to service 'normal' communications, is that maybe it's struggling to service this quantity or type of (potentially not-normal) communication at once? Are there any configurable rate-limits to testrun available? Akin to the timing templates offered by nmap? My first thought is to slow the outgoing Testrun communications down a bit, to see whether that delivers the expected result 100% of the time?

I've tested this same device a handful of times in Testrun v1.4, I will double check tomorrow whether I saw same false negatives in those reports too.

[duncangreene]

Alas, max_device_reports = 5 has rolled over all but one of my v1.4 reports, and that report was for an NTP-only test. I did find another v1.4 report for this device in my archives, and that showed a true positive as expected.

Just for added info, the other 2 reports I've done in v2.0 showed a false negative and true positive (so a 50% hit rate across 4 tests done in v2.0 so far). I appreciate Testrun can only report what it sees (and if a web server doesn't respond first time then it might be okay to assume it's not running one), but I do wonder whether it's worth investigating a slower scan and/or a retry mechanism maybe?

I believe nmap implements some sort of dynamic timing model based on RTT's (probably more to cater for network latency which should be negligible in the case of a Testrun environment), but whether that's fully leveraged in Testrun? I'm unsure if nmap is used in isolation in the different tests (i.e. the dynamic timing 'learnt' by nmap running within the Protocol module is unknown to nmap running within the Services module). Spitballing here a bit! 😅

The other option might be to implement a retry mechanism, maybe with exponential backoff, but I imagine this might clash with or delay other tests.

[jhughesoti]

You can change the max_device_reports to zero to prevent rollover and disable the report limit per device.

nmap is run within the services module and the results of that scan are what is used to determine the results.

Can you elaborate on your testing procedure? i.e., are you fully powering your device down between each test or are you just re-testing multiple times with the device powered on the whole time? I would like to eliminate difference in device state and testing procedure as a possible issue.

[duncangreene]

I can confirm that the device is/was fully powered down between each test. Device is only powered on once 'Waiting for Device' is displayed.

There are no other external influences to the device at all during the test either (e.g. there's no HMI that could be tinkered with or temp sensors connected to the inputs, etc.). DUT is directly connected to onboard Ethernet adapter of Testrun machine (no switch etc. in between).

[duncangreene]

A potential development...

As mentioned above, in line with the below I was using the onboard Ethernet adapter on my Testrun machine to connect to the DUT in the aforementioned tests.

I swapped the interfaces round this evening so that a USB Ethernet adapter was connected to the DUT, and so far 5 out of 5 tests have all returned true positives. This would echo the results I saw in v1.4, where I'm quite sure I only ever used a USB Ethernet adapter to connect to the DUT.

I'll run a few more tests using USB Ethernet over the next few days, and I can swap back to using onboard Ethernet to see if that causes the problem to come back again, but so far the signs seem to be pointing to using the onboard Ethernet adapter to connect to the DUT as causing the problem. What's strange is that only the HTTP test seemed to exhibit unreliable results, however this device doesn't have any other 'active' services (that could trigger false negatives) so unsure whether there's any weight to this particularly.

Presumably there is a reason for allowing use of only 1 onboard Ethernet adapter, and not 2, what's the background there? Why the need for USB Ethernet adapter(s) specifically? I'd be interested to know the technicals on this if poss!

[jboddey]

To increase the accuracy of the http service test (and potentially other port tests) we should utilize nmap's rate limiting arguments to avoid port scan detection in devices and devices with low processing capacity.

[duncangreene]

Presumably there is a reason for allowing use of only 1 onboard Ethernet adapter, and not 2, what's the background there? Why the need for USB Ethernet adapter(s) specifically? I'd be interested to know the technicals on this if poss!

@jboddey @jhughesbiot Did you have any background on the USB adapter vs. onboard adapter above?

[jboddey]

Presumably there is a reason for allowing use of only 1 onboard Ethernet adapter, and not 2, what's the background there? Why the need for USB Ethernet adapter(s) specifically? I'd be interested to know the technicals on this if poss!

@jboddey @jhughesbiot Did you have any background on the USB adapter vs. onboard adapter above?

I'm not yet sure on the reason for differing results. However, the usb adapter is required for the device connection as it allows us to control the power to the adapter to simulate a port toggle event for testing.

[duncangreene]

@jboddey Aha, so the USB Ethernet adapter specifically has to be used to connect to the DUT, and the onboard Ethernet adapter (if used) must only be used for the Internet connection? If this is the case this may warrant an update to the documentation, as in the past I'd been using the two interchangeably.

Am I right in thinking the Ethernet adapter power cycle is done as part of the connection.dhcp_disconnect test? If so, I would add that when using onboard Ethernet to connect to the DUT, no Errors appeared in the report at least from a connection.dhcp_disconnect PoV. I can raise this as a separate issue if this behaviour isn't expected?

[jboddey]

@duncangreene I will definitely update the docs to state that the USB ethernet should be used for the DUT. Could you send over the ZIP file for an attempt where you have used the onboard adapter for the DUT and received no errors?