Full Go "net" package port, WIP #4273

scottfeldman · 2024-05-23T04:22:48Z

[This is a resurrection of #4187, which I accidentally closed by deleting the fork it was based on.]

This PR is WIP to port the full Go "net" package to TinyGo.

With this PR, I can compile and link a simple example:

package main

import (
        "fmt"
        "net"
)

func main() {
        conn, err := net.Dial("tcp", "localhost")
        if err != nil {
                fmt.Println(err)
        }
        conn.Close()
}

tinygo build -target nano-rp2040 -tags netgo main.go

Notes:

This PR uses Go's full "net", "net/http", and "crypto/tls" packages, unmodified.
Need to compile with -tags netgo to pick Go's DNS resolver.
We no longer need the repo/submodule tinygo-org/net.
None of the network drivers are hooked up; we'll need to redefine the network driver interfaces at the syscall level.
I've copied in the stock Go src/syscall/ files for linux,arm. This defines all of the syscall types and data structures for linux,arm. We can copy in other arch files as needed. I'm only testing with nano-rp2040.
There are new files in sys/syscall:
- src/syscall/env_tinygo.go
- src/syscall/exec_tinygo.go // can go away if issue # 1 is addressed
- src/syscall/netlink_tinygo.go // can go away if issue # 1 is addressed
- src/syscall/syscall_tinygo.go
syscall_tinygo.go has the stubbed out syscalls. These all println("not implemented") and return EOPNOTSUPP.

The idea is this: with the netdev work in the last release, we truncated the "net", "net/http", and "crypto/tls" packages and inserted our own stubs to call into the network wifi drivers. With this PR, we use the full Go packages, but this time the insertion point is at the syscall level. Syscall is now where we define the interfaces to the network stack, and the network stack calls into the network drivers. It makes sense...if we were a full OS, syscall is where we'd have our OS-specific code.

Issues:

Issue # 1

Import cycle with "sync" package. Unfortunately, syscall package imports "sync" in some places, which causes an import cycle:

package tinygo.org/x/drivers/examples/net/http-get
       imports bytes
       imports io
       imports sync
       imports internal/task
       imports runtime/interrupt
       imports device/arm
       imports syscall
       imports sync: import cycle not allowed
(See src/syscall/netlink_linux.go for an example).

I've worked around this issue by stubbing out any imports of "sync", but that's not a workable solution in the long term. We'll need "sync" in the implementation of the stubbed out syscalls for the network stack and drivers. So we'll need to revisit "sync" so as to not import "syscall". Is it possible?

Use full Go "net" package. This includes "net/http". We no longer need tinygo-org/net repo, assuming we can get the full Go "net" package working with Tinygo. Also, use full Go "crypto/tls" package. This breaks existing netdev linkage to wifi drivers. A new interface to the drivers will need to be created at the syscall level.

Other archs can be added? I'm not sure. I'm testing with -target=nano-rp2040. I haven't tried other targets.

Remove duplicated linux/arch defines. Remove syscall_linux.go and syscall_unix.go. We'll replace these with syscall_tinygo.go in the next commit.

Stub out all the syscall entry points. Basically replace src/syscall_{linux|unix}.go with src/syscall_tinygo.go. It'll take some experimentation to discover which syscalls need to be implemented. For networking, we can explore with the examples/net examples. The two files exec_tinygo.go and netlink_tinygo.go are copies of {exec|netlink}_linux.go files, and have stubbed out functions. These files are copied only to work around an import cycle. The original files import "sync", but that creates an import cycle: package tinygo.org/x/drivers/examples/net/tcpclient imports bytes imports io imports sync imports internal/task imports runtime/interrupt imports device/arm imports syscall imports sync: import cycle not allowed If we can solve the import cycle, we don't need these two new files.

Remove the non-standard files src/syscall/syscall_nonhosted.go src/syscall/tables_nonhosted.go And move contents to _tinygo.go files: src/syscall/syscall_tinygo.go src/syscall/env_tinygo.go src/syscall/exec_tinygo.go

We pulling in the full "crypto/tls" package, so we don't need these custom files anymore.

Add these just to get a clean compile. Not sure if these are going to be called for normal networking?

leongross · 2024-05-28T13:51:51Z

@scottfeldman could you maybe elaborate on the process of how the board-specific drivers are abstracted here? As my understanding of tinygos network driver model goes, drivers are represented as the a combination of Netdever and Netlinker interfaces. But I cannot find any reference to that in the inserted syscall code, so how are the bare metal board specifics picked up here?

And to enable network support for linux platforms, couldn't the functions in syscall_tinygo be linked against the according and already existing interfaces provided bey the unix package?

scottfeldman · 2024-05-30T19:01:52Z

@scottfeldman could you maybe elaborate on the process of how the board-specific drivers are abstracted here? As my understanding of tinygos network driver model goes, drivers are represented as the a combination of Netdever and Netlinker interfaces. But I cannot find any reference to that in the inserted syscall code, so how are the bare metal board specifics picked up here?

@leongross your're right, the netdev/netlink interfaces are the current TinyGo driver model for embedded net devices. In this PR, these interfaces will be replaced with a TBD interface at the syscall level, since all the "net" package calls ultimately resolve into syscalls. The next step in this PR is to discover which syscalls are needed by "net" (and crypto/tls), and let those define the interface to the device drivers. The goal is to support both raw-MAC devices (i.e. Pico-W) as well as devices with an embedded stack (i.e. wifinina, rtl8720n). Some ASCII pics:

raw-MAC stack:

your app
"net" package
-------------           <-- syscall interface TBD
network stack
device driver
-------------           <-- hw interface
device

embedded stack:

your app
"net" package
-------------           <-- syscall interface TBD
device driver
-------------           <-- embedded fw interface
embedded network stack
-------------           <-- hw interface
device

And to enable network support for linux platforms, couldn't the functions in syscall_tinygo be linked against the according and already existing interfaces provided bey the unix package?

Enabling full OS support for "net" wasn't the goal of this PR, but it seems we could make it work by linking in the OS syscalls when compiling against a full OS, bypassing the driver interface mentioned above. I suspect the work to do this is around loader/goroot.go and some built tag magic.

ydnar · 2024-06-13T21:43:16Z

Import cycle with "sync" package. Unfortunately, syscall package imports "sync" in some places, which causes an import cycle:

Can you work around this with //go:linkname?

scottfeldman · 2024-06-13T23:32:48Z

Import cycle with "sync" package. Unfortunately, syscall package imports "sync" in some places, which causes an import cycle:

Can you work around this with //go:linkname?

Yes!

I say that with excitement as I discovered that work-around last week and it's working great. So Issue #1 is not an issue.

scottfeldman · 2024-06-13T23:54:20Z

Update: I'm not ready to post commits, but I do have the full "net" pkg calling into wifinina driver via a custom TinyGo syscall interface. syscalls in the interface so far:

src/syscall/system.go

//go:build tinygo

package syscall

type systemer interface {
        Socket(domain, typ, proto int) (fd int, err error)
        CloseOnExec(fd int)
        SetNonblock(fd int, nonblocking bool) (err error)
        SetsockoptInt(fd, level, opt int, value int) (err error)
        Connect(fd int, ip []byte, port uint16) (err error)
        Write(fd int, buf []byte) (n int, err error)
        Read(fd int, buf []byte) (n int, err error)
}

Wifinina implements this interface. So far I have net.Dial("tcp", "foobar.com") attempting to connect. Since I'm compiling with -tags netgo, the Go DNS client will attempt to resolve "foobar.com" by opening a UDP socket on 127.0.0.1:53. So the first socket to open is the UDP socket. I'm working thru intercepting the Reads and Writes to fake a DNS server response to resolve "foobar.com". I'll have more details on this DNS business when I commit, but that's where I'm at right now.

The net.Dial() test app needs greater than -stack-size=16KB and less than -stack-size=32KB. I haven't figured out the minimum, but 32KB is good so far. The test image is ~350K flash, 8k ram.

Use go:linkname to provide a system interface to syscalls. //go:linkname UseSystem syscall.useSystem func UseSystem(s Systemer) For example, the wifinina driver implements Systemer interface and plugs itself in to handle syscall calls: nina := wifinina.New(&wifinina.Config{...}) UseSystem(nina) syscalls will call into the Systemer, usually 1:1: func Socket(domain, typ, proto int) (fd int, err error) { return system.Socket(domain, typ, proto) } Systemer interface methods have basic Go type arguments because otherwise you'd have a dependency cycle with higher-level types. For example, if Systemer used net.Addr, "net" imports "syscall" which imports "net"...no go.

Add all of the necessary syscall func stubs to get a clean compile against "net" pkg. The stubs println a "not implemented" msg for those unimplememnted. Some of the syscall funcs are implemented and call into the Systemer interface. More will be added, but the set defined in Systemer interface are enough to run the examples/net/tcpclient example app.

The "net" pkg hits these low-level runtime funcs, so replace the panic() with a TODO note so the app can continue. No ill effects have been noted with the funcs as NOPS, with very limited testing... I have no idea what these functions should do. Maybe NOP is OK?

scottfeldman · 2024-06-16T05:54:05Z

Update: I am making some commits to capture where I'm at so far. Not done, but I now have the first test (examples/net/tcpclient) working with wifinina using the full "net" pkg.

$ tinygo flash -monitor -tags netgo -target nano-rp2040 -size short -stack-size 32KB -ldflags="-X 'main.ssid=test' -X 'main.pass=testtest'" ./examples/net/tcpclient/

   code    data     bss |   flash     ram
 297888    5280    4232 |  303168    9512
Connected to /dev/ttyACM0. Press Ctrl-C to exit.

Tinygo ESP32 Wifi network device driver (WiFiNINA)

Driver version           : 0.27.0
ESP32 firmware version   : 1.4.8
MAC address              : 34:94:54:26:a7:cc

Connecting to Wifi SSID 'test'...CONNECTED

DHCP-assigned IP         : 10.0.0.113
DHCP-assigned subnet     : 255.255.255.0
DHCP-assigned gateway    : 10.0.0.1

---------------
Dialing TCP connection 
Sending data 

Wrote 133780 bytes in 3548 ms
 
Disconnecting TCP...
---------------
Dialing TCP connection 
Sending data

deadprogram · 2024-06-16T06:36:10Z

This is extremely exciting @scottfeldman please let us know how we can help out!

scottfeldman · 2024-06-16T15:38:18Z

This is extremely exciting @scottfeldman please let us know how we can help out!

Thank you. I'm not sure how to break this up and share, but here's my short list of what still needs to be done:

need to create another PR in drivers to capture the wifinina changes so far...
get tcp/udp examples/net tests working with wifinina (nano-rp2040)
get "crypto/tls" examples working with wifinina
get all examples/net tests working with rtl8720n (wioterminal)
don't forget about espat
replace netdev/netlink interfaces with something cleaner, that also works with...
get seqs/cyw43 working under Systemer and pass all examples/net tests

I would really like help with 5 and 6. Work on those should probably wait until we get thru 1 and 2, just to prove the new full "net" pkg solution is going to work, especially the "crypto/tls" part.

scottfeldman · 2024-06-16T18:01:49Z

One more comment: I noticed the "net" pkg trying to open system files like /etc/resolve.conf and /etc/hosts for DNS resolution. Opening those files fail, of course, but I do see those calls working their way down to the syscall level so I had a thought: could we put a tinyfs behind these file i/o syscalls?

scottfeldman · 2024-06-20T21:49:21Z

@deadprogram I need some help with getting "crypto/tls" working...who's my contact for "crypto/tls" for big Go? I'm trying to get examples/net/tlsclient working, and it's doing the full TLS handshake with the server and then failing trying to verify the server certificate:

Connection failed: tls: failed to verify certificate: x509: certificate signed by unknown authority

What is working since last update is DNS resolution and UDP connections. For the TLS connection, I had to first use UDP to get NTP time and then call runtime.AdjustTimeOffset() to set system time, otherwise the server certificate fails due to being out-of-date wrt system time.

Oh, also, since I'm using nano-rp2040, I had to hack "crypto/rand" with this code to provide a custom rand.Reader:

func init() {
        rand.Reader = &reader{}
}

type reader struct{}

func (r *reader) Read(b []byte) (n int, err error) {
        if len(b) == 0 {
                return
        }
        var randomByte uint32
        for i := range b {
                if i%4 == 0 {
                        randomByte, err = machine.GetRNG()
                        if err != nil {
                                return n, err
                        }
                } else {
                        randomByte >>= 8
                }
                b[i] = byte(randomByte)
        }
        return len(b), nil
}

I think I got that code snippet from @deadprogram a while back, and have just been carrying it around with my projects. But perhaps this should get moved into TinyGo somehow? Anyway, without overriding rand.Reader, the program gets a nil-pointer dereference panic. "crypto/tls" uses rand to generate the TLS Client msg for the handshake.

Ok, that's it for now. If someone can help me get passed the server certificate verification, I think full "crypto/tls" is just going to work. From wifinina's perspective, it's just a simple TCP connection, so a lot of the code we had in there to program mbedTLS is no longer needed. Yay, bunch of code deletions!

scottfeldman · 2024-06-21T08:48:01Z

Ok, I'm passed the server cert validation issue I was having. I created a root CA cert to pass into tls.Dial() by go:embed'ing a PEM file containing the CA certs. Now the validator is happy.

Next problem is OOM:

panic: runtime error at 0x100407ef: out of memory
[tinygo: panic at /usr/local/go/src/crypto/internal/bigmod/nat.go:71:15]

The -print-allocs=. output is overwhelming. Not sure where to start there.

I sprinkled some runtime.GC() calls in the Connect() and Read() driver paths, but still hitting OOM:

panic: runtime error at 0x1003281b: out of memory
[tinygo: panic at /usr/local/go/src/crypto/internal/nistec/p256.go:235:11]

Maybe it's not possible for "crypto/tls" to fit? That would be a bummer.

I don't suppose there is a way to dump annotated memory to see what's in the heap at OOM?

leongross · 2024-06-28T11:30:12Z

I did some work on enabling the unix.Syscall wrapper for tinygo 1. When the syscal interface for the network stack is done I think we have a good chance to make networking finally work on linux systems.

deadprogram · 2024-08-14T17:34:37Z

I don't suppose there is a way to dump annotated memory to see what's in the heap at OOM?

I think something like this is possible using gdb?

leongross · 2024-08-22T11:07:04Z

Ok, I'm passed the server cert validation issue I was having. I created a root CA cert to pass into tls.Dial() by go:embed'ing a PEM file containing the CA certs. Now the validator is happy.

Next problem is OOM:
panic: runtime error at 0x100407ef: out of memory
[tinygo: panic at /usr/local/go/src/crypto/internal/bigmod/nat.go:71:15]
The -print-allocs=. output is overwhelming. Not sure where to start there.

I sprinkled some runtime.GC() calls in the Connect() and Read() driver paths, but still hitting OOM:
panic: runtime error at 0x1003281b: out of memory
[tinygo: panic at /usr/local/go/src/crypto/internal/nistec/p256.go:235:11]
Maybe it's not possible for "crypto/tls" to fit? That would be a bummer.

I don't suppose there is a way to dump annotated memory to see what's in the heap at OOM?

With the new go1.23 release (updated crypto libraries) and the according tinygo release it might be worth the effort to have another look at this.

leongross · 2024-08-22T13:07:52Z

This PR has been quite inactive in recent history. We at u-root are quite interested in getting this working, so we can make the remaining commands build for the initramfs busybox.
If there is anything that you can outsource to me @scottfeldman I would be more than happy to help you with that.

deadprogram · 2024-08-22T17:56:56Z

@scottfeldman same goes for me, I am willing to help! 😸

scottfeldman · 2024-08-28T20:39:08Z

Hi Everyone, sorry for the silence/absence on this issue. I've been away.

My personal motivation for this work was to get pico-w running with the full net and crypto/tls packages and Patricios' cwy43 + seqs TCP/IP stack. I was sucessful in getting TCP/UDP traffic working with the full net package on Arduino rp2040. When I tried TLS traffic, I ran into the OOM issues.

I suspect TLS just by itself is going to take more RAM than we have available on these little embedded processors with ~256KB for both stack and heap. And then we have to add the net package and the seqs package which will need socket buffer space for each socket connection. So adding this all up, it's not going to fit. I wanted to investigate adding SPI PSRAM chip to pico-w to see if we could get 8MB of RAM. But I don't know the upper-limit for TLS (is 8MB enough?), and I don't know what's required from the compiler to map stack/heap to PSRAM. Probably a lot, I suspect.

So that's where I got stuck and gave up.

In any case, @leongross if this work can be used for Linux/u-boot, awesome, please take it over. I was surprised how little actually needs to be implemented at the syscall level to support the full net package. Most of the syscall .go files copied over as-is. It really boils down to a hand full of functions defined in the Systemer interface.

scottfeldman added 8 commits March 12, 2024 09:46

copy in linux/arm syscall defines

3025bc0

Other archs can be added? I'm not sure. I'm testing with -target=nano-rp2040. I haven't tried other targets.

fixup definitions

80e6aab

Remove duplicated linux/arch defines. Remove syscall_linux.go and syscall_unix.go. We'll replace these with syscall_tinygo.go in the next commit.

cleanup

3cf9cf7

Remove the non-standard files src/syscall/syscall_nonhosted.go src/syscall/tables_nonhosted.go And move contents to _tinygo.go files: src/syscall/syscall_tinygo.go src/syscall/env_tinygo.go src/syscall/exec_tinygo.go

remove src/crypto/tls

6e49da6

We pulling in the full "crypto/tls" package, so we don't need these custom files anymore.

remove tinygo-org/net from submodules

462a9d5

add stubbed out runtime go:linkname funcs

a4efa34

Add these just to get a clean compile. Not sure if these are going to be called for normal networking?

scottfeldman requested review from deadprogram, aykevl, bgould, sago35 and soypat May 23, 2024 04:22

leongross mentioned this pull request May 28, 2024

add NOP netdev as default tinygo-org/net#29

Merged

scottfeldman added 4 commits June 15, 2024 22:09

remove generated linux files, to be replaced by custom tinygo equiv

f5c6a7f

archie2x mentioned this pull request Aug 9, 2024

tinygo: -compiler command line argument (e.g. tinygo) u-root/gobusybox#121

Draft

stopped working on it

921f0a9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Full Go "net" package port, WIP #4273

Full Go "net" package port, WIP #4273

scottfeldman commented May 23, 2024

leongross commented May 28, 2024

scottfeldman commented May 30, 2024 •

edited

Loading

ydnar commented Jun 13, 2024

scottfeldman commented Jun 13, 2024

scottfeldman commented Jun 13, 2024

scottfeldman commented Jun 16, 2024

deadprogram commented Jun 16, 2024

scottfeldman commented Jun 16, 2024

scottfeldman commented Jun 16, 2024

scottfeldman commented Jun 20, 2024

scottfeldman commented Jun 21, 2024

leongross commented Jun 28, 2024 •

edited

Loading

deadprogram commented Aug 14, 2024

leongross commented Aug 22, 2024

leongross commented Aug 22, 2024

deadprogram commented Aug 22, 2024

scottfeldman commented Aug 28, 2024

Full Go "net" package port, WIP #4273

Are you sure you want to change the base?

Full Go "net" package port, WIP #4273

Conversation

scottfeldman commented May 23, 2024

Notes:

Issues:

Issue # 1

leongross commented May 28, 2024

scottfeldman commented May 30, 2024 • edited Loading

raw-MAC stack:

embedded stack:

ydnar commented Jun 13, 2024

scottfeldman commented Jun 13, 2024

scottfeldman commented Jun 13, 2024

scottfeldman commented Jun 16, 2024

deadprogram commented Jun 16, 2024

scottfeldman commented Jun 16, 2024

scottfeldman commented Jun 16, 2024

scottfeldman commented Jun 20, 2024

scottfeldman commented Jun 21, 2024

leongross commented Jun 28, 2024 • edited Loading

deadprogram commented Aug 14, 2024

leongross commented Aug 22, 2024

leongross commented Aug 22, 2024

deadprogram commented Aug 22, 2024

scottfeldman commented Aug 28, 2024

scottfeldman commented May 30, 2024 •

edited

Loading

leongross commented Jun 28, 2024 •

edited

Loading