Inconsistent / incorrect ROBOT attack results (2nd issue)

[famzah]

I'm experiencing the same inconsistency in the results as in issue #1107, when repeatedly testing the same Debian server.

Here is an example for ports 465 (SSL-only) and 587 (STARTTLS):

# IP address of the server is 192.252.146.33

famzah@vbox64:~/testssl$ for i in {1..20} ; do ./testssl.sh --robot famzah.net:465|grep " ROBOT" ; done
 ROBOT                                     not vulnerable (OK)
 ROBOT                                     VULNERABLE (NOT ok)
 ROBOT                                     VULNERABLE (NOT ok) - weakly vulnerable as the attack would take too long

famzah@vbox64:~/testssl$ for i in {1..20} ; do ./testssl.sh --robot --starttls smtp famzah.net:587|grep " ROBOT" ; done
 ROBOT                                     not vulnerable (OK)
 ROBOT                                     VULNERABLE (NOT ok)
 ROBOT                                     VULNERABLE (NOT ok) - weakly vulnerable as the attack would take too long

I cannot reproduce this on port 25 (STARTTLS) where I consistently get "not vulnerable (OK)". This is the expected result as the server is using the patched OpenSSL by Debian. The server is running Debian Buster and OpenSSL is version "1.1.1d-0+deb10u7". Note: ports 465 & 587 are served by a different server implementation compared to the server for port 25. I cannot disclose more about the implementations.

You should be able to reproduce on your end easily. It takes a couple of attempts to get different results. Usually in less than 10 attempts you will encounter a different result.

It you can't reproduce, I have saved the screen output and the "/tmp" dir content for both "not vulnerable (OK)" and "VULNERABLE (NOT ok)" runs executed with debug=3. If you need the saved data, I will attach it here.

The question is -- Is "testssl.sh" encountering a bug and reports inconsistent results, or is it the server for ports 465 & 587 which does something weird and really is vulnerable.

Thank you.

---

Here is my environment:

famzah@vbox64:~/testssl$ ./testssl.sh -b 2>/dev/null | grep "from"
    testssl.sh       3.0.6 from https://testssl.sh/

famzah@vbox64:~/testssl$ ./testssl.sh -b 2>/dev/null | grep -A3 OpenSSL
 Using "OpenSSL 1.1.1f  31 Mar 2020" [~79 ciphers]
 on vbox64:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

# maybe related to: https://github.com/drwetter/testssl.sh/issues/1119
famzah@vbox64:~/testssl$ ./testssl.sh --robot --starttls smtp famzah.net:587|head -n2|tail -n1
No engine or GOST support via engine with your /usr/bin/openssl

famzah@vbox64:~/testssl$ lsb_release -a
Description:    Ubuntu 20.04.3 LTS

famzah@vbox64:~/testssl$ uname -a
Linux vbox64 5.4.0-92-generic #103-Ubuntu SMP Fri Nov 26 16:13:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[drwetter]

As far as I am concerned: this check was relatively robust the past. IIRC #1107 was something different.

Did a few tests on the target you were providing. Here are somewhat intermediate findings.

First of all it seems for me 85-95% of the tests returned "not vulnerable (OK)". Never encountered "VULNERABLE (NOT ok) - weakly vulnerable" so far.

Then when I tested from a North American host I never got "VULNERABLE (NOT ok)", other than when I checked from Europe. The target IP is located in the US.

Given that and as an educated guess: it could be a networking timing issue. But as said my statistics aren't good yet. It seems that I can't do >21 checks in a row.

@dcooper16 : do you have any idea?

[dcooper16]

My memory of how this test works isn't very good. (I wrote it more than 4 years ago, and I was just implementing a test method developed by others.) However, I tried testing the server to see if I could figure out what was happening, and got "not vulnerable (OK)" every time. If the inconsistent results don't happen if testing from North America, that would explain why I didn't see it.

Seeing the screen output from "not vulnerable (OK)" and "VULNERABLE (NOT ok)" runs executed with debug=3 may be helpful. It would at least be a starting point.

[famzah]

You can test to the following hosts which have an identical setup:

195.8.222.36   : located in Europe
192.252.146.33 : located in the USA
116.251.204.34 : located in HK

Today I launched three VPS instances @ Digital Ocean from the following locations:

• FRA: Frankfurt, Germany (close to the EU server)
• NYC: New York, USA (super close to the US server)
• SGP: Singapore (close to the HK server)

Here are the tests results:

FRA - 195.8.222.36   ( 28 ms): total=40 ; ok=40 ; weakly= 0 ; not_OK= 0
FRA - 192.252.146.33 ( 84 ms): total=40 ; ok=39 ; weakly= 0 ; not_OK= 1
FRA - 116.251.204.34 (180 ms): total=40 ; ok= 5 ; weakly= 5 ; not_OK=30

NYC - 192.252.146.33 (  8 ms): total=40 ; ok=40 ; weakly= 0 ; not_OK= 0
NYC - 195.8.222.36   (133 ms): total=40 ; ok=11 ; weakly=12 ; not_OK=17
NYC - 116.251.204.34 (210 ms): total=40 ; ok= 8 ; weakly= 8 ; not_OK=24

SGP - 116.251.204.34 ( 35 ms): total=40 ; ok=40 ; weakly= 0 ; not_OK= 0
SGP - 195.8.222.36   (205 ms): total=40 ; ok=11 ; weakly=16 ; not_OK=13
SGP - 192.252.146.33 (294 ms): total=40 ; ok= 6 ; weakly= 8 ; not_OK=26

I've done a total of 40 attempts to port 465 from each VPS location to each server. Each test result line shows the following:

VPS_LOCATION - TESTED_SERVER (ping RTT): total tests; "not vulnerable (OK)"; "VULNERABLE (NOT ok) - weakly vulnerable"; "VULNERABLE (NOT ok)" results

My black-box testing definitely shows that network latency plays a vital role in the consistency of the results.

---

Here is my environment:

root@debian-s-1vcpu-1gb-fra1-01:~/testssl.sh# ./testssl.sh -b 2>/dev/null | grep "from" # directly "git clone'd"
    testssl.sh       3.1dev from https://testssl.sh/dev/

root@debian-s-1vcpu-1gb-fra1-01:~/testssl.sh# ./testssl.sh -b 2>/dev/null | grep -A3 OpenSSL
 Using "OpenSSL 1.0.2-chacha (1.0.2k-dev)" [~179 ciphers]
 on debian-s-1vcpu-1gb-fra1-01:./bin/openssl.Linux.x86_64
 (built: "Jan 18 17:12:17 2019", platform: "linux-x86_64")

# no warning for "No engine or GOST support via engine with your /usr/bin/openssl"

root@debian-s-1vcpu-1gb-fra1-01:~/testssl.sh# lsb_release -a
Description:    Debian GNU/Linux 11 (bullseye)

root@debian-s-1vcpu-1gb-fra1-01:~/testssl.sh# uname -a
Linux debian-s-1vcpu-1gb-fra1-01 5.10.0-10-amd64 #1 SMP Debian 5.10.84-1 (2021-12-08) x86_64 GNU/Linux

[famzah]

We need to clarify the following:

• Is this a bug in the ROBOT test and is it fixable?
• Is it recommended that we run the tests from a LAN location thus eliminating any network latencies, in order for all "testssl.sh" results to be reliable?
• If the ROBOT test is hard to get fixed, maybe we could run a "ping" test before it, and issue a warning if the latency is bigger than 40 ms?

[drwetter]

Oh, great, thanks for the input!

I did 20x runs from Europe against HK with https://github.com/robotattackorg/robot-detect, which was originally used by @dcooper as a basis for porting this to testssl.sh .

The result is most often "Getting inconsistent results, aborting.", and some ",NOT VULNERABLE". Never vulnerable.

Interesting was in Hanno's/Tibor's/... python code the segment when "Getting inconsistent results, aborting." is being printed:

if (oracle_good != oracle_good_verify) or (oracle_bad1 != oracle_bad_verify1) or (oracle_bad2 != oracle_bad_verify2) or (oracle_bad3 != oracle_bad_verify3) or (oracle_bad4 != oracle_bad_verify4):
    if not args.quiet:
        print("Getting inconsistent results, aborting.")

Probably I understand either codes not good enough but

          # If the server provided the same error message for all tests, then this
          # is an indication that the server is not vulnerable.
          if [[ "${response[0]}" != "${response[1]}" ]] || [[ "${response[1]}" != "${response[2]}" ]] || \
             [[ "${response[2]}" != "${response[3]}" ]] || [[ "${response[3]}" != "${response[4]}" ]]; then
               vulnerable=true

Don't know whether this is the same as above (doubt that) but the comment here doesn't match the assignment of vulnerable=true and if there are different responses I would also expect a message like not sure.

David, can you check?

[drwetter]

robot-detect seems more robust but my statistics weren't good enough. Running robot-detect from Europe --> NY yielded 2x vulnerable, 2x not vulnerable, 16x 'Getting inconsistent results, aborting'. For testssl I got worse results than you (3x vulnerable, 2x weakly vulnerable, 1x OK, then I stopped)

So far I believe the check with both tools has problems, however it looks like testssl.sh can do better.

[dcooper16]

Hi @drwetter, @famzah,

I don't have much time to look into this today (maybe late next week), but I tried testing against 195.8.222.36, since it seemed to have the most inconsistent results when testing from North America. I ran the test using

testssl.sh --debug 3 --robot --ip 195.8.222.36 famzah.net:465

and using script to store the results in a file.

Of the 15 tests I ran, only one came back "not vulnerable (OK)":

> egrep -i "response\[|vulnerable" results.txt 
response[0] = 15030300020214
response[1] = 15030300020214
response[2] = 15030300020214
response[3] = 15030300020214
response[4] = 15030300020214
response[0] = Timeout waiting for alert
response[1] = Timeout waiting for alert
response[2] = Timeout waiting for alert
response[3] = Timeout waiting for alert
response[4] = Timeout waiting for alert
not vulnerable (OK)

The other tests all failed in the first iteration of testing. Some examples are:

> egrep -i "response\[|vulnerable" results.txt
response[0] = 15030300020214
response[1] = 15030300020214
response[2] = 15030300020214
response[3] = 15030300020214
response[4] = 1503030002021434353420544
VULNERABLE (NOT ok) - weakly vulnerable as the attack would take too long

> egrep -i "response\[|vulnerable" robot/test1.txt 
response[0] = 15030300020214
response[1] = 15030300020214
response[2] = 1503030002021434353420544
VULNERABLE (NOT ok)

> egrep -i "response\[|vulnerable" results.txt 
response[0] = 1503030002021434353420544
response[1] = 15030300020214
response[2] = 1503030002021434353420544
VULNERABLE (NOT ok)

> egrep -i "response\[|vulnerable" results.txt
response[0] = 15030300020214
response[1] = 15030300020214
response[2] = 15030300020214
response[3] = 1503030002021434353420544
VULNERABLE (NOT ok)

In every case, the server either responded with "15030300020214" or "1503030002021434353420544". The ROBOT test is just checking whether the responses are all the same or not, it isn't considering the contents of the response.

"15030300020214" is the TLS alert message indicating "bad record mac". "3435342054" is "454 T", where "454" is an SMTP authentication failed error. (I can not account for the "4" at the end of the longer message.

So, it seems that this server is always sending the same TLS response, but it sometimes adds an SMTP error message and sometimes does not. Perhaps it would be acceptable to parse the response and remove anything that comes after the TLS response before comparing, but I'm not certain. I would have to test the other servers mentioned to see whether something similar is happening with them.

[digininja]

Three years later I'm having the same problem. Don't ask why, but I just scanned 12k hosts and about about 400 of them came back vulnerable. I manually checked a couple with SSLLabs and it said they weren't vulnerable, I also checked with sslyze and it agreed. I've just rescanned the 400 vulnerable with testssl.sh and it is now saying only some of them are vulnerable. That was in parallel mode so a few manual checks along side that have confirmed inconsistent results. For the same host I've had:

VULNERABLE
Server does not support any cipher suites that use RSA key transport
Fixme: Conversion of public key failed around line 20817
not vulnerable

The hosts are up and when I hit them with curl between tests I've not seen any blocking or missing responses.

I'm on the latest version freshly pulled this morning:

$ git remote -v
origin  https://github.com/testssl/testssl.sh.git (fetch)
origin  https://github.com/testssl/testssl.sh.git (push)
$ git branch
* 3.3dev
* $ ./testssl.sh -v | grep from
  testssl.sh version 3.3dev from https://testssl.sh/dev/

[drwetter]

Hi @digininja , thanks for digging this up .

The host you displayed with four different results: that it quite awkward. I assume that was a single example and the others had different consistency problems?

The host above shows still the same symptoms. As @dcooper16 said Perhaps it would be acceptable to parse the response and remove anything that comes after the TLS response before comparing, that would be a start. But then one receives Timeout waiting for alert and 15030300020214 which would be kind of consistent with robot-detect from the authors of ROBOT: running it 9 times returned 8 times

Modulus size: 2048 bits, 256 bytes
Identical results (TLS alert 20 of length 7), retrying with changed messageflow
Identical results (Timeout waiting for alert), no working oracle found
NOT VULNERABLE!

and one time

Modulus size: 2048 bits, 256 bytes
Getting inconsistent results, aborting.

[digininja]

I scanned about 12k hosts and testssl.sh found about 900 with the issue. I manually checked a handful with Qualys and they were all false positives. I rescanned them with testssl.sh and less than half came back as vulnerable. I then picked a few of them and did them individually and got this type of result. Scanning with sslalyze (or something like that) found none vulnerable but did hit three it couldn't scan. I decided that ROBOT isn't serious enough to spend any more time on so moved on from there and didn't write it up for any.

…

On Sun, 14 Dec 2025, 12:13 Dirk Wetter, ***@***.***> wrote: *drwetter* left a comment (testssl/testssl.sh#2083) <#2083 (comment)> Hi @digininja <https://github.com/digininja> , thanks for digging this up . The host you displayed with four different results: that it quite awkward. I assume that was a single example and the others had different consistency problems? The host above shows still the same symptoms. As @dcooper16 <https://github.com/dcooper16> said *Perhaps it would be acceptable to parse the response and remove anything that comes after the TLS response before comparing*, that would be a start. But then one receives Timeout waiting for alert and 15030300020214 — Reply to this email directly, view it on GitHub <#2083 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA4SWJDD45MP3XIQXPCDJ34BVH5XAVCNFSM6AAAAACOWZVCJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMNJQHAZTSMBUGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>