BSD Sockets API layer

[zephyrbot]

Reported by Paul Sokolovsky:

Support for BSD Sockets like API for Zephyr was one of the most common request among stakeholders and parties interested in Zephyr, and for a long time. At the Zephyr meeting at OpenSummit Portland 2017 and Zephyr Mini-summit at Budapest 2017 it was confirmed that there're no stakeholders who oppose the idea, and then Linaro Zephyr team would like to proceed with prototyping and implementing this API.

This Epic is created to collect initial ideas/requirements, as well to keep interested parties in loop. It may be somewhat informal at this time, and may be rewritten to follow a more formal pattern if requested.

The current basic ideas are:

1. Implementation of BSD Sockets would be a pilot project of adding POSIX-compatible APIs to Zephyr, which is another popular requested from parties interested in Zephyr.

2. However, the initial target is to implement "BSD Socket compatible API", without pledging for full/almost full POSIX compliance from the start. The underlying requirements is that the new API still followed Zephyr paradigm of being lightweight. As a specific example, BSD Sockets compatible API may be implemented similarly to how Winsock is (was) implemented on Windows systems - while the underlying system isn't POSIX compliant, Wisock API is still largely BSD Sockets compatible, allowing to easily port existing apps.

3. The new API is a separate layer, not intended to supersede or replace native Zephyr networking API. This is because it's already clear that BSD Sockets API will require more resources (e.g. it's not zero-copy friendly, while native Zephyr is). Implementation-wise, there however should be work on making native API behaviors more POSIX compliant, to avoid cases when BSD Sockets compatible functions need to be coded "from scratch" or require a lot of adaptation code due to differences in behavior (this is a specific point from [~anasnashif] and how we already tried to do it for 1.7.0 release).

4. The initial prototyping would be done out of tree (using a sample application requiring BSD Sockets APII), with an existing native networking API. Experience gathered during this process will be used to submit an RFC how to implement the needed functionality in a more optimal way natively in Zephyr.

More technical points/details:

5. The main difference between native Zephyr API and BSD Sockets is "push" vs "pull" API paradigm: on reception, Zephyr API "pushes" packets to an application via a callback, while BSD Sockets (and POSIX API in general) lets an app "pull" data from an OS when the app needs it. So, majority of work (and the main initial focus) will be on "impedance matching" two API models. This will require a buffering queue for incoming packets. Zephyr has such a queue, but it is global, for all interface and network contexts, while BSD Sockets will require it per-socket.

6. The initial focus is just on connection/sending/receiving calls. More advanced features like select()/poll() (waiting for multiple sockets) is subject for next stage of work (it will require more deep thinking on how to architect POSIX API layer, as those calls aren't specific to sockets, but are generic for file descriptor objects).

7. Initial prototyping is to happen using MicroPython, as it already has a module which implements push-style API in terms of pull-style API, and allows for easy testing as it provides scripting language to exercise these APIs.

• BSD Sockets API: Offloading support #3706: BSD Sockets API: Offloading support
• BSD Sockets API: Select/poll API #3664: BSD Sockets API: Select/poll API
• BSD Sockets API: Non-blocking API #3663: BSD Sockets API: Non-blocking API
• BSD Sockets API: Basic blocking API #3662: BSD Sockets API: Basic blocking API

(Imported from Jira ZEP-1921)

[zephyrbot]

by xiaorui hu:

Hi, Paul, it seems that majority of work will be on the BSD socket API Layer. Is there any detailed plan on the specific APIs ? for example, in V1.8 the BSD socket like API socket()/recv()/send() will be supported, in V1.9 the advanced APIs select()/listen()/poll() will be considered ? We are very concerned about this. Thanks.

[zephyrbot]

by Paul Sokolovsky:

xiaorui hu , the plan is pretty much like you write, except that I target basic blocked sockets support for the initial implementation, that includes both client and server support calls: socket, recv, send, bind, connect, listen, accept. Only after such patch is done, looking into more advanced, non-blocking socket operation with select/poll.

Unfortunately, I can't call specific Zephyr versions. I definitely hope to have at least an initial patchset within 1.8 timeframe, but whether it would be suitable for merging into 1.8 doesn't depend on me, but more on the Zephyr maintainers. For example, recently a considerable patch for IP stack refactoring was posted, I may imagine that would be a major topic for 1.8, and any work I'm doing will need to be rebased onto this pretty massive changes. In either case, currently implementation is in prototyping phase, to see what issues the current IP stack may have wrt to socket-like operation and what need to be fixed first (some of the issues were already identified and linked to this issue).

Hope this clarifies the situation, and I will share updates going forward.

[zephyrbot]

by Paul Sokolovsky:

Prototyping progress report #1: https://lists.zephyrproject.org/pipermail/zephyr-devel/2017-April/007492.html

Prototyping progress report #2: https://lists.zephyrproject.org/pipermail/zephyr-devel/2017-April/007494.html

Prototyping progress report #3: https://lists.zephyrproject.org/pipermail/zephyr-devel/2017-May/007638.html

[zephyrbot]

by Paul Sokolovsky:

The initial prototyping phase is effectively complete. The subset of BSD Sockets API has been found viable to be implemented for Zephyr. However, a number of issues were identified which lie on a critical path to Sockets API to be able to work properly (majority of these issues also affect native Zephyr API). These are/will be set as blockers of this ticket.

A conversion of the out-of-tree prototype implementation into a native, in-tree implementation has started, interested parties may track progress via #153 . However, this feature is still targeted for 1.9 merge window and release (the pull request above is to collect early feedback).

[zephyrbot]

by Paul Sokolovsky:

However, a number of issues were identified which lie on a critical path to Sockets API to be able to work properly

To clarify, they were identified, and usually a workaround patch was prepared which affects the issue identified. However, these patch and workarounds and drafts at best, and require further research and analysis, to find the most appropriate way to resolve them.

[zephyrbot]

by Paul Sokolovsky:

I prepared instructions for people who would like to test the current prototype implementation in MicroPython, while in-tree patches for Zephyr are being worked on: https://github.com/pfalcon/micropython/wiki/ZephyrSocket

[zephyrbot]

by Paul Sokolovsky:

Based on Mark Linkmeyer 's request, added proposed user stories breakdown for this epic.

[zephyrbot]

by Paul Sokolovsky:

xiaorui hu : The initial implementation of this feature (GH-3662 story) has now been merged, would be nice to receive a feedback on it. Thanks!

[zephyrbot]

Related to GH-3425

[zephyrbot]

Related to GH-3699

[pfalcon]

The only reason why this ticket is kept open is because of GH-3706: "BSD Sockets API: Offloading support". However, there was no work on that for awhile - since the PR to implement such an offloading support was rejected.

So, given that core work on implementing BSD Sockets API is done, I would like to close this ticket, and treat offloading support as a separate task.

[tbursztyka]

I agree. BSD socket offloading will be possible when we'll be using socket API only (hopefully in 1.13).