Certain ssh sessions seem to crash my instance of Gitblit

[Saxman255]

I have been digging into this and at a loss where to go next. I am running v1.9.3.
for the most part everything works great. My repository is there and I can pull from it and update great.
We used to run in Linux on SLES11 and now our client is setting up new servers for linux on SLES15. And taking the same stable copy I used before on the new server, it runs for maybe a day or two and then crashes. I'll share the logs I have.
I should add that I an running gitblit as a nohup started up by a custom script that we have used for years with no issues.
Google seems to lend that there are https requests or something coming in, but I don't know what to do about it. Can I change ports? There are also suggestions that an upgraded Jetty version could provide more logging but not sure I can just use later jetty jar, lol. Any thoughts or help would be appreciated.

2024-02-10 08:41:33 [INFO ] creating ssh session from /0:0:0:0:0:0:0:1:44621
2024-02-10 08:41:33 [WARN ] exceptionCaught(SshServerSession[null@/0:0:0:0:0:0:0:1:44621])[state=Opened] IllegalStateException: Incorrect identification (null characters not allowed) - at line 1 character #7 after '������'
2024-02-10 08:41:33 [INFO ] closed ssh session from /0:0:0:0:0:0:0:1:44621
2024-02-10 08:41:33 [INFO ] creating ssh session from /0:0:0:0:0:0:0:1:34413
2024-02-10 08:41:33 [WARN ] exceptionCaught(SshServerSession[null@/0:0:0:0:0:0:0:1:34413])[state=Opened] IllegalStateException: Incorrect identification (null characters not allowed) - at line 1 character #7 after '������'
2024-02-10 08:41:33 [INFO ] closed ssh session from /0:0:0:0:0:0:0:1:34413
2024-02-10 08:41:33 [INFO ] creating ssh session from /0:0:0:0:0:0:0:1:45033
2024-02-10 08:41:33 [WARN ] exceptionCaught(SshServerSession[null@/0:0:0:0:0:0:0:1:45033])[state=Opened] IllegalStateException: Incorrect identification (null characters not allowed) - at line 1 character #7 after '������'
2024-02-10 08:41:33 [INFO ] creating ssh session from /0:0:0:0:0:0:0:1:39851
2024-02-10 08:41:33 [INFO ] closed ssh session from /0:0:0:0:0:0:0:1:45033
2024-02-10 08:41:33 [WARN ] exceptionCaught(SshServerSession[null@/0:0:0:0:0:0:0:1:39851])[state=Opened] IllegalStateException: Incorrect identification (null characters not allowed) - at line 1 character #3 after '��'
2024-02-10 08:41:33 [INFO ] closed ssh session from /0:0:0:0:0:0:0:1:39851
2024-02-10 08:41:33 [INFO ] creating ssh session from /0:0:0:0:0:0:0:1:42347
2024-02-10 08:41:33 [WARN ] exceptionCaught(SshServerSession[null@/0:0:0:0:0:0:0:1:42347])[state=Opened] IllegalStateException: Incorrect identification (null characters not allowed) - at line 1 character #4 after '�>�'
2024-02-10 08:41:33 [INFO ] closed ssh session from /0:0:0:0:0:0:0:1:42347
2024-02-10 08:41:33 [WARN ] Illegal character 0x16 in state=START for buffer HeapByteBuffer@2f393bca[p=1,l=475,c=8192,r=474]={\x16<<<\x03\x03\x01\xD6\x01\x00\x01\xD2\x03\x030\xB1\xCa\xC5W\x1ff...\x03\x08\x04\x04\x01\x05\x03\x05\x01\x08\x05\x08\x06\x06\x01>>>.git\r\nAccept-Enco...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2024-02-10 08:41:33 [WARN ] badMessage: 400 Illegal character 0x16 for HttpChannelOverHttp@6926155e{r=0,c=false,a=IDLE,uri=}
2024-02-10 08:41:33 [WARN ] Illegal character 0x16 in state=START for buffer HeapByteBuffer@2f393bca[p=1,l=475,c=8192,r=474]={\x16<<<\x03\x02\x01\xD6\x01\x00\x01\xD2\x03\x02\xBa\x02\xA5.^m?...\x03\x08\x04\x04\x01\x05\x03\x05\x01\x08\x05\x08\x06\x06\x01>>>.git\r\nAccept-Enco...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2024-02-10 08:41:33 [WARN ] badMessage: 400 Illegal character 0x16 for HttpChannelOverHttp@3e355672{r=0,c=false,a=IDLE,uri=}
2024-02-10 08:41:33 [WARN ] Illegal character 0x16 in state=START for buffer HeapByteBuffer@2f393bca[p=1,l=475,c=8192,r=474]={\x16<<<\x03\x01\x01\xD6\x01\x00\x01\xD2\x03\x01 b\x11\xBc>=\xCa...\x03\x08\x04\x04\x01\x05\x03\x05\x01\x08\x05\x08\x06\x06\x01>>>.git\r\nAccept-Enco...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2024-02-10 08:41:33 [WARN ] badMessage: 400 Illegal character 0x16 for HttpChannelOverHttp@52bc611b{r=0,c=false,a=IDLE,uri=}
2024-02-10 08:41:33 [WARN ] Illegal character 0x16 in state=START for buffer HeapByteBuffer@2f393bca[p=1,l=429,c=8192,r=428]={\x16<<<\x03\x00\x01\xA8\x01\x00\x01\xA4\x03\x000\xEc^\xEe\x1f\xFb\xE2...\xCc\xA9\xCc\xAa\xCc\xAb\xCc\xAc\xCc\xAd\xCc\xAe\x02\x00\x01>>>\x00,\x00\n\x00\n\x00\x08\x00\x17\x00\x19\x00\x18\x00\x16\x00...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2024-02-10 08:41:33 [WARN ] badMessage: 400 Illegal character 0x16 for HttpChannelOverHttp@343df7ed{r=0,c=false,a=IDLE,uri=}
2024-02-10 08:41:33 [WARN ] badMessage: 400 for HttpChannelOverHttp@6d6888d1{r=0,c=false,a=IDLE,uri=}
2024-02-10 08:41:33 [INFO ] *****************************************************************
2024-02-10 08:41:33 [INFO ] Stopping Gitblit
2024-02-10 08:41:33 [INFO ] *****************************************************************
2024-02-10 08:41:33 [INFO ] Stopped ServerConnector@7ac2e39b{SSL-HTTP/1.1}{0.0.0.0:8442}
2024-02-10 08:41:33 [INFO ] Stopped ServerConnector@78365cfa{HTTP/1.1}{0.0.0.0:8070}
2024-02-10 08:41:34 [INFO ] Gitblit context destroyed by servlet container.
2024-02-10 08:41:34 [INFO ] Git Daemon stopping...
2024-02-10 08:41:34 [INFO ] Git Daemon stopped.
2024-02-10 08:41:34 [INFO ] SSH Daemon stopping...
2024-02-10 08:41:34 [INFO ] Stopped o.e.j.w.WebAppContext@293a5f75{/,file:/u/ant/schfrwms/ACXGITBLIT_2.8test/data/temp/webapp/,UNAVAILABLE}{file:/u/ant/schfrwms/ACXGITBLIT_2.8test/gitblit.jar}

[flaix]

Hi!

Did I understand that correctly, that you don't know of any traffic caused by you internally that is causing this? But you expect some external traffic that you don't know of as a cause?

As for changing ports, that is possible in the gitblit.properties file. It is settings server.httpPort and server.httpsPort for the HTTP transport. You will find other ports in the documentation, too.

The second problem looks like in #1451. I haven't quite found out how to reproduce this.

[Saxman255]

So something I found comparing our old and new servers is that the new server had IPv6 enabled. When I look at the log it is a local host IPv6 address. So we will try to disable IPv6 and see if that works. I'll let you know if we get good results.

[Saxman255]

So, no luck there. I'll try to get with our client's server team to track down where these requests are coming from. Everything I've read around Jetty sounds like an https request coming to the http port or something like this.

I don't know if an updated version of Jetty would be able to better handle the exception? I don't know enough about Jetty, I just happened to find this regarding Jetty 9 and similar error messages. I would just be nice if there was more details or something helpful to direct where to check.

https://stackoverflow.com/questions/25625410/jetty-9-warning-badmessage-400-illegal-character

[msohn]

most of this project's dependencies are very ancient and outdated:

java 8 (latest LTS is 21)
jetty.version : 9.4.49.v20220914 (latest is 12.0.9)
slf4j.version : 1.7.36
wicket.version : 1.4.22
lucene.version : 5.5.2 (latest is 9.10)
jgit.version : 4.11.9.201909030838-r (latest is 6.9.0.202403050737-r)
groovy.version : 2.4.4
bouncycastle.version : 1.69 (latest is 1.78.1)
selenium.version : 2.28.0
wikitext.version : 1.4
sshd.version: 1.7.0 (latest is 2.12.1)
mina.version: 2.0.25 (latest is 2.2.3)
guice.version : 5.1.0 (latest is 7.0)
guice-servlet.version : 5.1.0-gb2

[Saxman255]

Yeah all my other research has pointed to the main issue being Jetty and handling there in version 9. I don't know who currently still works on this but I'm wondering if they could target upgrading the Jetty version to start maybe.

[shoelzer]

I was getting this same error and Gitblit would crash anywhere from 4 to 20 hours after launch. I didn't fix the error, but I was able to avoid it in my case by disabling unused protocols. This may be overkill, but I only need https, so I disabled all the rest in gitblit.properties:

server.httpPort = 0
git.sshPort = 0
git.daemonPort = 0
server.shutdownPort = 0

[Saxman255]

Yeah that's basically what I did. I just disabled the shutdown port and updated my script to just kill the PID directly. A little crude, but works. Whatever was going on was somehow triggering the shutdown port.