Simple TCP Proxy: Segmentation fault and error: can only marshal stream with unsafe flag

[Geo-7]

I'm testing Janet, and so far, it's going really well, I'm currently creating a simple proxy in Janet, Hare, Erlang, and Rust to compare their speed. The results I've seen with Janet have been awesome, especially considering that I'm using a single thread, However, I have encountered two problems with my TCP proxy.

proxy/init.janet:

(def listener (net/listen "127.0.0.1" "10000"))

(defn handler [conn]
  (def upstream (net/connect "127.0.0.1" "9000"))
  (forever
    (->> (ev/read conn 2048)
        (net/write upstream ))
    (->> (ev/read upstream 2048)
         (net/write conn))))

(defn start-server []
  (forever
    (def conn (net/accept listener))
    (ev/call handler conn)))

(defn main [&args]
  (start-server))

First: I can not build exe with jpm build I'm getting this error:

generating executable c source build/jfp.c from proxy/init.janet...
error: can only marshal stream with unsafe flag
error: build fail
  in pdag [/usr/local/lib/janet/jpm/dagbuild.janet] (tailcall) on line 79, column 23
  in run [/usr/local/lib/janet/jpm/cli.janet] (tailcall) on line 88, column 9
  in run-main [boot.janet] on line 3907, column 16
  in cli-main [boot.janet] on line 4060, column 17

Second: after sending some packets with simple client(running it multiple times), my proxy server segfaults
I trided gdb (Janet has been compiled without symbols) It crashes on

in janet_mark ()

Simple Client:

(defn make-array [a b] 
  [0 85 32 127 138 0 0 129 0 211 12 215 57 36 19 80 34 41
   64 50 53 115 33 170 153 3 18 1 135 17 23 55 135 40 10 5
   a b 223 100 224 125 116 238 118 22 12 23 38 98 35 48 81 6
   54 7 4 20 153 0 217 4 5 85 85 85 0 5 178 60 156 202 10 50
   49 48 61 49 49 46 48 55 59 175 52 202 125])


(defn send [number dummy-buffer]
  (def proxy (net/connect "127.0.0.1" 10000))
  (each x (range 0 number)
    (net/write proxy dummy-buffer)
    (ev/read proxy 1024))
  (:close proxy))

(defn main [&args]
  (print "Client started")
  (var dummy-buffer @"")
  #Make 100 clients TODO: Should be multi thread 
  (each x (range 0 100)
    (each b (make-array 0 x)
      (buffer/push-byte dummy-buffer b))
    #Each Client sends N number of packet
    (ev/call send 1000 dummy-buffer)
    ))

Simple Echo server:

(def listener (net/listen "127.0.0.1" 9000))

(defn handler [conn]
  (forever
    (->> (ev/read conn 2048)
         (net/write conn))
    (print "Got message")))

(defn start-server []
  (forever
    (->> (net/accept listener)
         (ev/call handler))))



(defn main [&args]
  (print "Server Started")
  (start-server))

Any Advice?

[CosmicToast]

Regarding the marshalling issue:
You are making a connection as a top-level binding (listener).
When you build an executable, all top-level forms are executed prior to runtime, and then marshalled.
As the compiler says, you cannot marshal a stream (an existing connection), as you're creating it build-time.
Try creating it during runtime, using dynamics or any other method of moving that over into main (or another function).

[CosmicToast]

As for the other one, I'm not too familiar with the ev/ family in janet, but it appears to be crashing during garbage collection.
If I had to guess, you're actually spawning infinite listening collections, or something along those lines, and perhaps running out of usable memory on an attempt to collect (thus allocating NULL and not catching it).
Alternatively, it's possible that it's a use-after-free.

Regardless, I'd recommend building Janet with symbols to figure this out better :)

[Geo-7]

Thank you I moved it to the main and my first problem has been solved.

[bakpakin]

The segfault with the mark-phase of the garbage collector looks like a bug. My guess it is related to the newish code making ev/connect non-blocking.

[bakpakin]

Yup, here is a full stacktrace from valgrind:

Invalid read of size 4
==3127305==    at 0x13DE02: janet_mark_string (gc.c:100)
==3127305==    by 0x13DCF9: janet_mark (gc.c:66)
==3127305==    by 0x135A00: ev_machine_write (ev.c:2529)
==3127305==    by 0x130964: janet_stream_mark (ev.c:384)
==3127305==    by 0x1310FC: janet_ev_mark (ev.c:534)
==3127305==    by 0x13ED65: janet_collect (gc.c:450)
==3127305==    by 0x17F2E9: run_vm (vm.c:992)
==3127305==    by 0x181D0D: janet_continue_no_check (vm.c:1515)
==3127305==    by 0x181EEA: janet_continue_signal (vm.c:1547)
==3127305==    by 0x133884: janet_loop1 (ev.c:1316)
==3127305==    by 0x133C4D: janet_loop (ev.c:1385)
==3127305==    by 0x163FE0: janet_loop_fiber (run.c:127)
==3127305==  Address 0xffffffffffffffe8 is not stack'd, malloc'd or (recently) free'd
==3127305== 
==3127305== 
==3127305== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==3127305==  Access not within mapped region at address 0xFFFFFFFFFFFFFFE8
==3127305==    at 0x13DE02: janet_mark_string (gc.c:100)
==3127305==    by 0x13DCF9: janet_mark (gc.c:66)
==3127305==    by 0x135A00: ev_machine_write (ev.c:2529)
==3127305==    by 0x130964: janet_stream_mark (ev.c:384)
==3127305==    by 0x1310FC: janet_ev_mark (ev.c:534)
==3127305==    by 0x13ED65: janet_collect (gc.c:450)
==3127305==    by 0x17F2E9: run_vm (vm.c:992)
==3127305==    by 0x181D0D: janet_continue_no_check (vm.c:1515)
==3127305==    by 0x181EEA: janet_continue_signal (vm.c:1547)
==3127305==    by 0x133884: janet_loop1 (ev.c:1316)
==3127305==    by 0x133C4D: janet_loop (ev.c:1385)
==3127305==    by 0x163FE0: janet_loop_fiber (run.c:127)
==3127305==  If you believe this happened as a result of a stack
==3127305==  overflow in your program's main thread (unlikely but
==3127305==  possible), you can try to increase the size of the
==3127305==  main thread stack using the --main-stacksize= flag.
==3127305==  The main thread stack size used in this run was 8388608.

looking into this for this fix

[bakpakin]

Yeah, so the issue is that the ev/connect makes a write with the internal machinery with a NULL pointer. During garbage collection, the garbage collector tries to mark the NULL pointer. the symptom is segfaulting if garbage collection is run during a slow connect.

This fix is simply protect the the marking code against NULL pointers during connect.

Should be fixed in 8df7364

[Geo-7]

Thanks, that was fast :)

[zevv]

Oh oh. I'm on a holiday trip with no computer with me, so I can't join the debug fun, sorry.

[CosmicToast]

Exact opposite situation of mine, huh

[zevv]

Well, I warned you that I had no clue what I was doing, right :)

#1139 (comment)