Custom build for Eden Cocktail Lounge in San Marcos. 22 esp32s, 10k leds, 1 stressed out network

[woodward54]

The ceiling has a grid of LED squares controlled by 22 esp32 microcontrollers, all networked over Wifi/TCP to a companion media server app I built in Unity. The user can upload any video they like and change the speed, contrast, and color in realtime.

Still struggling to push the display past 15fps. After that, the network bandwidth starts to get congested and we start losing frames. The TCP sockets can't push the data fast enough, leading to the esp buffers not filling up in time. Does anyone have thoughts to address this? Do we need a better router? My other thought is maybe the computer (M1 macbook air) can't handle the bandwidth of that many TCP connections. We have the MacBook hardwired to the router via ethernet.

It's been a pleasure building on the work done by the NightDriver community!

Video

[davepl]

FWIW, Nightdriver uses lzcompress on the data packets, which isn’t the “smartest” way to compress video of course but it does work to reduce bandwidth quite a bit. There’s only so much CPU on the client, but it can easily decompress a full 64x32 on a single ESP32 at 24 fps.

…

On Dec 13, 2024, at 4:54 PM, David Woodward ***@***.***> wrote: The ceiling has a grid of LED squares controlled by 22 esp32 microcontrollers, all networked over Wifi/TCP to a companion media server app I built in Unity. The user can upload any video they like and change the speed, contrast, and color in realtime. Still struggling to push the display past 15fps. After that, the network bandwidth starts to get congested and we start losing frames. The TCP sockets can't push the data fast enough, leading to the esp buffers not filling up in time. Does anyone have thoughts to address this? Do we need a better router? My other thought is maybe the computer (M1 macbook air) can't handle the bandwidth of that many TCP connections. We have the MacBook hardwired to the router via ethernet. It's been a pleasure building on the work done by the NightDriver community! Video <https://youtu.be/B7P4akEZtL0> — Reply to this email directly, view it on GitHub <#673>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4HCF3FJTHCOPJPI5GH2MD2FN6VBAVCNFSM6AAAAABTS7W2QCVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZXGY3TMOBWGY>. You are receiving this because you are subscribed to this thread.

[woodward54]

I have compression turned on which helps. I've noticed that playing 'simpler' videos that can be compressed better have a more stable fps (i.e. videos with high color grouping). Currently, Im compressing & sending 1 frame at a time, but I think you're right using a video compression algo and batching multiple frames over time would probably work better in this case.

Each esp32 drives 480 ws2812b pixels. The esp32 CPU sits around 10% utilization, so I have some headroom there.

Do you have any recommendations for video compression algorithms to look into?

[robertlipe]

Nice! You're kind of asking for help, so help us understand more about your setup for the question asked. My guess is that you're not running the effects on the ESP32; you're running something like XLights and feeding video frames into the ESP32's doing color rendering. So there are two halves to performance: how fast you can put stuff to the cip and how fast you an get it out. Espressif claims the parts will deliver 20Mbps <https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-guides/wifi.html#:~:text=Up%20to%2020%20MBit/s,UDP%20throughput%20over%20the%20air> over TCP and 30 over UDP. I'll just be kind and wish my doctor would prescribe what they're taking. You can do wifi testing and just have the ESP32's eat colordata packets, memcpy them to themselves just to simulate reading them later, but not actually putting them on the ESP32 bus. That would help determine if the ESP32's can guzzle the data down quickly enough. So where is the effect generation done and what't the actual path from the animator through the blinkies? 30 * 22 = more than you're going to get from a single router in one room. If it's environmental, you'd see a difference where a lone ESP32 might saturate the bus but if you're sending to 22, they get groggy. If you think this is a problem, wired ethernet ESP32's are not that expensive compared to the setup you already have. Might be an easy experiment. You didn't list what LED type or topology you're using. WS281x uses a self-clocking signal of 800Khz that limits each WS281x bus (strip) to approximately 800LEDs at about 60fps. (I could do the math if asked, but I'm rushing to the door.) If you're using something like APA102, you can get a WAY higher frame rate, but you also need more data pins. There's code in the project to allow this, but I've seen no evidence that anyone has used it. (If you are, I/we would like to hear more...) You can make a strand infinitely long and "merely" the frame rate will stutter. So you can test with any length of strip and any number of pins. It is possible to run multiple pins of data. Depending on the ESP32's used, the real number might be between 4 and 11(?). You definitely want to use the pin numbers that result in DMA-fed transfers and not bit-banged ports. There's been some noise in the FastLED library in recent months that have changed which combinations get the low-overhead version and which require the hardware hand-holding, For testing, you can configure a strip to be any length. There's no acknowledgmeent. It doesn't know if the bits just fall onto the floor right at the plug on the chip or sail off the end of a 10 led strip or only light the first 100 of a 1000 led stirp. It doesn't know. Let that guide some thought experiments. I have to run to the airport and have some more follow-up ideas and questions, but wanted o hammer out some blabberings just to get the conversation going. MOre later.

…

On Fri, Dec 13, 2024 at 6:54 PM David Woodward ***@***.***> wrote: The ceiling has a grid of LED squares controlled by 22 esp32 microcontrollers, all networked over Wifi/TCP to a companion media server app I built in Unity. The user can upload any video they like and change the speed, contrast, and color in realtime. Still struggling to push the display past 15fps. After that, the network bandwidth starts to get congested and we start losing frames. The TCP sockets can't push the data fast enough, leading to the esp buffers not filling up in time. Does anyone have thoughts to address this? Do we need a better router? My other thought is maybe the computer (M1 macbook air) can't handle the bandwidth of that many TCP connections. We have the MacBook hardwired to the router via ethernet. It's been a pleasure building on the work done by the NightDriver community! Video <https://youtu.be/B7P4akEZtL0> — Reply to this email directly, view it on GitHub <#673>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCSD3YKCWII6J32NP74KWD2FN6VFAVCNFSM6AAAAABTS7W2QCVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZXGY3TMOBWGY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[woodward54]

Each ESP32 drives 480 ws2812b pixels. Using MAX_BUFFERS=45. I've tried increasing the buffer size but that's about the limit I can get with onboard memory- our boards are stock ESP32 and don't have PSRAM.

I've thought about replacing the esp32 boards with ethernet or adding PSRAM. In hindsight, we probably should have done this from the get-go. The tricky part is the esp boards are buried in the ceiling and hard to access.

I'm confident its a bandwidth issue because when we run fewer esp boards (i.e. 10), the connection can run at 24fps with 0 frame drops. As we add more boards to the network and/or play more complex (less compressible) videos, the network starts to get bogged up and we loose frames.

I'm a software guy, so networking is a bit out of my domain. Is there any way to monitor WiFi bandwidth? I've tried using the asus router traffic monitor and we only push around 1.6mbps which is way under the advertised capability of the router.

[davepl]

I’ll be honest, I haven’t run video on a non-PSRAM board in quite some time! The Mesmerizer PCB is the easiest approach, as it as 4MB of PSRAM. For individual models, the Feather TFT S3 has 2M. With that, I’ve got 500 frames of buffer which keeps things rock solid. You only likely need 3 seconds of buffer though I’d say. - Dave

…

On Dec 14, 2024, at 11:22 AM, David Woodward ***@***.***> wrote: Each ESP32 drives 480 ws2812b pixels. Using MAX_BUFFERS=45. I've tried increasing the buffer size but that's about the limit I can get with onboard memory- our boards are stock ESP32 and don't have PSRAM. I've thought about replacing the esp32 boards with ethernet or adding PSRAM. In hindsight, we probably should have done this from the get-go. The tricky part is the esp boards are buried in the ceiling and hard to access. I'm confident its a bandwidth issue because when we run fewer esp boards (i.e. 10), the connection can run at 24fps with 0 frame drops. As we add more boards to the network and/or play more complex (less compressible) videos, the network starts to get bogged up and we loose frames. I'm a software guy, so networking is a bit out of my domain. Is there any way to monitor WiFi bandwidth? I've tried using the asus router traffic monitor and we only push around 1.6mbps which is way under the advertised capability of the router <https://www.asus.com/us/networking-iot-servers/wifi-routers/asus-gaming-routers/rt-ax82u/techspec/>. — Reply to this email directly, view it on GitHub <#673 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4HCF4PB4Z36PTE7UX7YP32FSAPFAVCNFSM6AAAAABTS7W2QCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNJWHAZTIOA>. You are receiving this because you commented.

[woodward54]

Yeah, we might need to upgrade to PSRAM boards.

Where can I order a Mesmerizer PCB?

[robertlipe]

Cheap Trick: take last year's router (for testing it doesn't have to be an awesome one) and split the network load across multiple routers, on multiple channels. This will give you multiple collision domains and a different kind (not typing "less" want a typo) of interference. Putting them o different channels is key. Hopefully everything is DHCP so that doesn't require a laptop on a ladder and you can define this group of MACs to.on router and the other to the other. Actually, you might have to play a game if letting half boot and grab leases on one, disable login in that router, then turn on the other. We can at least defer that trip to the ladder. Well, unless you implemented kerberos type auth where all MACs ask a central authority for their own passwords. That's pretty quaint in modern times,.but for an hour looking at the ladder in the corner, you'd get to feel pretty smug. Don't extrapolate to the absurd and run 24 radios in a 2000 sq foot area. WiFi doesn't work like that. Everyone will be trying to about over everyone else. Is your sender one process sending to all or is it 24 processes maintaining synchronization in some other way. Dave,. Is a mesmerizer really your. Recommendation for a strip? Something like a yulc or gledopto with an onboard regulator may have advantages.

…

On Sat, Dec 14, 2024, 1:22 PM David Woodward ***@***.***> wrote: Each ESP32 drives 480 ws2812b pixels. Using MAX_BUFFERS=45. I've tried increasing the buffer size but that's about the limit I can get with onboard memory- our boards are stock ESP32 and don't have PSRAM. I've thought about replacing the esp32 boards with ethernet or adding PSRAM. In hindsight, we probably should have done this from the get-go. The tricky part is the esp boards are buried in the ceiling and hard to access. I'm confident its a bandwidth issue because when we run fewer esp boards (i.e. 10), the connection can run at 24fps with 0 frame drops. As we add more boards to the network and/or play more complex (less compressible) videos, the network starts to get bogged up and we loose frames. I'm a software guy, so networking is a bit out of my domain. Is there any way to monitor WiFi bandwidth? I've tried using the asus router traffic monitor and we only push around 1.6mbps which is way under the advertised capability of the router <https://www.asus.com/us/networking-iot-servers/wifi-routers/asus-gaming-routers/rt-ax82u/techspec/> . — Reply to this email directly, view it on GitHub <#673 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCSD33DXVM7UZXFVTJ2GF32FSAPFAVCNFSM6AAAAABTS7W2QCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNJWHAZTIOA> . You are receiving this because you commented.Message ID: <PlummersSoftwareLLC/NightDriverStrip/repo-discussions/673/comments/11568348 @github.com>

[davepl]

You can get a Mesmerizer kit here, complete with matrix and cables and remote and the works… https://www.ebay.com/itm/335713383751

…

On Dec 14, 2024, at 12:50 PM, Robert Lipe ***@***.***> wrote: Cheap Trick: take last year's router (for testing it doesn't have to be an awesome one) and split the network load across multiple routers, on multiple channels. This will give you multiple collision domains and a different kind (not typing "less" want a typo) of interference. Putting them o different channels is key. Hopefully everything is DHCP so that doesn't require a laptop on a ladder and you can define this group of MACs to.on router and the other to the other. Actually, you might have to play a game if letting half boot and grab leases on one, disable login in that router, then turn on the other. We can at least defer that trip to the ladder. Well, unless you implemented kerberos type auth where all MACs ask a central authority for their own passwords. That's pretty quaint in modern times,.but for an hour looking at the ladder in the corner, you'd get to feel pretty smug. Don't extrapolate to the absurd and run 24 radios in a 2000 sq foot area. WiFi doesn't work like that. Everyone will be trying to about over everyone else. Is your sender one process sending to all or is it 24 processes maintaining synchronization in some other way. Dave,. Is a mesmerizer really your. Recommendation for a strip? Something like a yulc or gledopto with an onboard regulator may have advantages. On Sat, Dec 14, 2024, 1:22 PM David Woodward ***@***.***> wrote: > Each ESP32 drives 480 ws2812b pixels. Using MAX_BUFFERS=45. I've tried > increasing the buffer size but that's about the limit I can get with > onboard memory- our boards are stock ESP32 and don't have PSRAM. > > I've thought about replacing the esp32 boards with ethernet or adding > PSRAM. In hindsight, we probably should have done this from the get-go. The > tricky part is the esp boards are buried in the ceiling and hard to access. > > I'm confident its a bandwidth issue because when we run fewer esp boards > (i.e. 10), the connection can run at 24fps with 0 frame drops. As we add > more boards to the network and/or play more complex (less compressible) > videos, the network starts to get bogged up and we loose frames. > > I'm a software guy, so networking is a bit out of my domain. Is there any > way to monitor WiFi bandwidth? I've tried using the asus router traffic > monitor and we only push around 1.6mbps which is way under the advertised > capability of the router > <https://www.asus.com/us/networking-iot-servers/wifi-routers/asus-gaming-routers/rt-ax82u/techspec/> > . > > — > Reply to this email directly, view it on GitHub > <#673 (reply in thread)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACCSD33DXVM7UZXFVTJ2GF32FSAPFAVCNFSM6AAAAABTS7W2QCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNJWHAZTIOA> > . > You are receiving this because you commented.Message ID: > <PlummersSoftwareLLC/NightDriverStrip/repo-discussions/673/comments/11568348 > @github.com> > — Reply to this email directly, view it on GitHub <#673 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4HCF33VRTGADTBAPJD4XT2FSK2XAVCNFSM6AAAAABTS7W2QCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNJWHA3TONQ>. You are receiving this because you commented.