Crash：Failed to decrypt a packet

[cnkedao]

[May 16 08:53:48.554] [ET_NET 13] DEBUG: <QUICPacketPayloadProtector_openssl.cc:135 (_unprotect)> (quic_ppp) Failed to decrypt -- the first 4 bytes decrypted are 35 9c 4d 26
[May 16 08:53:48.554] [ET_NET 13] DEBUG: <QUICPacketPayloadProtector.cc:90 (unprotect)> (quic_ppp) Failed to decrypt a packet #258
traffic_server: received signal 11 (Segmentation fault)
traffic_server - STACK TRACE:
./traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xc3)[0x6aeb67]
/usr/lib/../lib64/libpthread.so.0(+0xf5d0)[0x7f2640d765d0]
./traffic_server(_ZN13IOBufferBlock4sizeEv+0xc)[0x6b21c8]
./traffic_server(_ZN17QUICPacketFactory6createE10IpEndpointSt10unique_ptrIA_h22ats_unique_buf_deleterEmmR24QUICPacketCreationResult+0xb04)[0x9ae2be]
./traffic_server(_ZN22QUICPacketReceiveQueue7dequeueER24QUICPacketCreationResult+0x6e8)[0x9e69d8]
./traffic_server(_ZN18QUICNetVConnection20_dequeue_recv_packetER24QUICPacketCreationResult+0x50)[0x9512fa]
./traffic_server(_ZN18QUICNetVConnection44_state_connection_established_receive_packetEv+0x65)[0x94df0f]
./traffic_server(_ZN18QUICNetVConnection28state_connection_establishedEiP5Event+0xe0)[0x94bcca]
./traffic_server(_ZN12Continuation11handleEventEiPv+0xc9)[0x6b2b4b]
./traffic_server(_ZN18QUICNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x73)[0x94c9c3]
./traffic_server(_ZN10NetHandler18process_ready_listEv+0xac)[0x931b02]
./traffic_server(_ZN10NetHandler15waitForActivityEl+0x5de)[0x9322ce]
./traffic_server(_ZN7EThread15execute_regularEv+0x3ec)[0x984bbc]
./traffic_server(_ZN7EThread7executeEv+0x107)[0x984dbd]
./traffic_server[0x9837af]
/usr/lib/../lib64/libpthread.so.0(+0x7dd5)[0x7f2640d6edd5]
/usr/lib/../lib64/libc.so.6(clone+0x6d)[0x7f264037aead]
Segmentation fault (core dumped)

#server: ats-quic latest
records:
CONFIG proxy.config.udp.threads INT 10
CONFIG proxy.config.diags.debug.enabled INT 1
CONFIG proxy.config.diags.debug.tags STRING quic
CONFIG proxy.config.quic.initial_max_streams_bidi_in INT 10000
CONFIG proxy.config.quic.initial_max_streams_bidi_out INT 10000

#client: ngtcp2 draft-20: https://github.com/ngtcp2/ngtcp2/tree/draft-20

command:

./client -n 1000 127.0.0.1 4433 http://wellcdn.com/10m

CTRL+C ,try again:

./client -n 1000 127.0.0.1 4433 http://wellcdn.com/10m

[maskit]

We had two days interop session as apart of QUIC WG interim meeting, and we didn't see crashes.

Could you try the latest code and let us know whether it still crashes on your environment? Also, if it still crashes, please let us know whether packet numbers seem reasonable.

[cnkedao]

Thank you for your attention,after git pull the latest code both ats and ngtcp2,this problem seems ok.
But,I found an other problem now.
I use the same command to test it and use CTRL+C to cancel soon,then try it many times.
After stopping the test for a long time, you can find that the memory is skyrocketing until the system kills it

top status:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
30932 ats 20 0 38.0g 34.6g 8436 S 299.7 43.6 6:29.39 [TS_MAIN]

[maskit]

Could you give us a bit more details?

• How many times did you ran the command? 10? 100?
• How long time did you leave ATS running? 10min? 1hours?

[cnkedao]

Ran the command about 20 times.
Then leave it running for about 5 minutes, the system will kill it due to memory problems.

[maskit]

Thanks for the information. I'll look into it next week.

[maskit]

I was able to see slowly increasing (not skyrocketing) memory usage, but not only after QUIC requests.
Do you see it only after QUIC requests?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. Marking it stale to flag it for further consideration by the community.