CDCInterface vs UART - irq method

[surfermarty]

CDCInterface has a comment:

    # USB CDC serial device class, designed to resemble machine.UART
    # with some additional methods.

but trying to use it with PPP over serial exposes that UART has an 'irq()' method that isn't implemented in CDCInterface and (as far as I can work out) in the subordinate classes. PPP class needs it to be there to use it.

Now I may be missing something dumb - can I wrapper the CDCInterface in another object to get the missing irq method?

Have been looking at a subclass that implements the missing irq function, but the UART docs say 'Returns an irq object.' and I'm struggling to find out how to create one of those currently (new to python and micropython). I've got a _thread based task to check for input availability and call the handler, but also an unhandled exception and stack trace when I attach it to the PPP class, so getting to grips with MP debugging too.

If anyone is able to throw light onto the missing 'irq' method or how I can work around it, that would be great.

Another area of confusion is that in the UART class 'irq' definition:

handler is an optional function to be called when the interrupt event triggers. The handler must take exactly one argument which is the UART instance.

but in the PPP class, the connect method suggests it will be called as such

stream.irq(ppp.poll, stream.IRQ_RXIDLE)

and the poll method is listed with out arguments

PPP.poll()

[projectgus]

Hi @surfermarty,

but trying to use it with PPP over serial exposes that UART has an 'irq()' method that isn't implemented in CDCInterface and (as far as I can work out) in the subordinate classes. PPP class needs it to be there to use it.

That's a good point, something we could look at extending in CDCInterface as well.

Now I may be missing something dumb - can I wrapper the CDCInterface in another object to get the missing irq method?

Yes, that should work fine.

implents the missing irq function, but the UART docs say 'Returns an irq object.' and I'm struggling to find out how to create one of those currently (new to python and micropython).

The IRQ class is implemented in C not Python, and there's no way to construct one from Python code. You could possibly implement a "works-alike" dummy IRQ class in Python, however you probably don't even need to do this:

[...] in the PPP class, the connect method suggests it will be called as such

stream.irq(ppp.poll, stream.IRQ_RXIDLE)

and the poll method is listed with out arguments

PPP.poll()

The docs for ppp.poll() don't specify it (will fix), but there's an optional extra argument accepted here which is then ignored. Behaviour which exists only for compatibility with IRQs. So the UART IRQ will calll ppp.poll(uart_obj), but the PPP class implementation ignores that (optional) argument.

Because these parts of the IRQ interface are totally ignored, it should be safe for your custom irq() function to return anything (i.e. None or True) to the caller (as the PPP caller also ignores the .irq() result). Then if your thread detects "rx idle" state and calls the registered handler, you can also pass anything as the handler argument because the PPP class will ignore that as well.

[projectgus]

Quick docs fix submitted at #18740

[surfermarty]

Thanks - I'm gettting a stack trace even with a simple wrapper class, so suspect that I need to spend some more time looking at the micropython diagnostic pages. Needless to say if I substitute the CDC for a real UART, it works - so it's either something dumb in my dummy class or with CDCInterface. (Note, I'm not expecting this to work - just not crash immediately!)

class CDC(CDCInterface):

  def __init__(self):
    super().__init__()

  def irq(self, handler, flags):
    pass

Rest of code:

cdc = maui.CDC()
cdc.init()
usb.device.get().init(cdc,builtin_driver=False)
ppp=network.PPP(cdc)
ppp.connect(security=ppp.AUTH_CHAP,user="12345678", key="12345678")

[projectgus]

Oh, this is an interesting crash!

0x4205870f: mp_cstack_usage at /home/gus/ry/george/micropython/py/cstack.c:46
 (inlined by) mp_cstack_check at /home/gus/ry/george/micropython/py/cstack.c:52
0x3fcb9760: ?? ??:0
0x4201d262: pppos_write at /home/gus/ry/george/esp-idf-v5/components/lwip/lwip/src/netif/ppp/pppos.c:242
0x3fcb98e0: ?? ??:0
0x4200f4a1: tcpip_thread_handle_msg at /home/gus/ry/george/esp-idf-v5/components/lwip/lwip/src/api/tcpip.c:166
 (inlined by) tcpip_thread at /home/gus/ry/george/esp-idf-v5/components/lwip/lwip/src/api/tcpip.c:148
0x00000000: ?? ??:0

0x00000000: ?? ??:0

The PPP callback is running in the context of the ESP-IDF LWIP task, which isn't actually initialised with Python support. So when the Python-side stack checks runs, it crashes. This isn't a problem when using the hardware UART because the hardware UART is written in C, and when the hardware UART calls Python it does it via a "soft" irq type mechanism (on esp32).

(If the above sounds technical, it's because it is... this is quite a low-level limitation on implementing PPP where the underlying protocol is in Python. 🤔.)

[surfermarty]

I'm old enough to say I spent my formative years writing assembly code, so yes it does make sense. discussion #9263 about virtual serial/uarts also found the same but the code for a stripped back uart in C was only alluded to.

Being new to python & MP for 3 weeks now, apologies if what follows isn't actually practical - it makes sense in my head but I'm not familiar enough with MP enough to spot the pitfalls that maybe obvious to others.

If as in #9263, a 'machine_vuart' class was created in C that exhibited the functionallity of the regular UART class, but instead of being tied to the h/w pins or UART, was tied to the RingIO/RingBuffer from @andrewleech, then the CDCInteface (or indeed anything that comes along which needs to look like a UART) could use that as the underlying core driver code. Including the use case of needing a 'debug' serial port, where it's just tied to print statements etc, so you can see what's being sent.

At the moment, this is my itch to scratch - I can work around my problem in hardware, by adding a second UART-USB bridge and tie it to the h/w UART, but it's still an itch.

If someone else who is proficient with MP thinks the above is viable, has use for MP in general and wishes to just knock it out quickly, I'm not precious about who does it - but at the same time I don't expect people to scratch my itch. If it's a corner case and MP really wouldn't include it, then just say so and I'll roll up my sleeves and see if I can hack my own solution.

My initial ponderings are around whether such a machine_vuart could be done in a machine agnostic way using existing MP functions, or whether it needs to follow the current machine_uart implementation of breaking out functions to the individual ports; and clearly my lack of MP knowledge is then the hinderance.

[surfermarty]

So to follow up, I've written a VirtualUART class in C that is h/w agnostic. Presents itself as a regular stream and has the irq method, so it's acceptable to be passed to the PPP class.

Works fine as a standalone class and doesn't suffer the crashes as found previously. As noted above, at least on ESP32, lwip task isn't in the same context as MP, so when you subclass it, the lwip task is trying to work up the object stack and it crashes again. Providing you don't subclass it, it work great.

To overcome the issue with context and callbacks - because it's up to the developer to do something with the VirtualUART class - i.e. tie it to a CDCInterface object - I've use the mp_sched_schedule function to instigate callbacks, which then means the function is called in the context of the main MP, rather than lwip. Don't think I can ever get a solution to the lwip context and subclassing, but that is now a documented limitation.

Now have PPP connecting to the CDCInterface and passing bytes back and forward. It won't bring up the IP stack just yet - but now it's not crashing, I can go look at why the ESP32 is ignoring the 'chap' or 'pap' items in the PPP request. Gone from not 'working-crashing-difficult to debug' to 'not working-not crashing-can now debug'

I may have to tweak the VirtualUART code depending on what I find, but if it's of interest, I'm happy to submit as a PR. It's inclusion is currently controlled by a platform #define, so only going to be included on custom builds.

Thanks @projectgus for the line about MP contexts and lwip, that would have taken me a long while to work out.