Add configurations to send data over wifi

[davidex720]

I think it could be useful to have the possibility to choose between sending the payload over lora or wifi.
In many countries lora has still some problems and this option could really benefit this project

[cyberman54]

I will not implement this, because i am not convinced that this makes sense, since there will be a huge trade off in using the 2,4Ghz airtime for sniffing versus transmitting data.

But you're welcome to bring a pull request for this feature, i will be lucky reviewing it.

[davidex720]

I see your point
This feature surely is not fast to implememet but probably scanning every minute is not strictly necessary for most of applications

[cyberman54]

You can reuse existing pieces of code to implement this:

• connecting to wifi network -> see OTA.cpp
• transferring counter data to the target host -> implement additional sendqueue, see SPISLAVE.cpp
• cycle between sniffing/transferring -> use time-of-day (see TIMEKEEPER.cpp) or use Ticker.h to kickoff task

Downsides will be:

• loading esp32 wifi & tcp/ip stack will have (probably massive) impact on RAM, sniffing performance and power consumption
• using the wifi stack in coexistence for sniffing and transferring probably will have additional impact on sniffing performance

Thus i would expect a trade-off.

[cyberman54]

What you can do to decrease the tradeoff: use compiler directives to compile the code without the LoRa LMIC stack, set HAS_LORA to not defined (= comment out) in the board's hal file.

This will free up resources on Core 1 and RAM.

If you implement coexist sniffing and transferring via Wifi be aware of race conditions when accessing the wifi subsystem. The wifi logic of paxcounter is running on Core 0.

[davidex720]

this could be a good occasion to develop this point too:
#250

I'm on the 250, it is somehow required

[cyberman54]

Unfortunately the whole code needs some refactoring, because it was grown without a roadmap. Thus the #includes got confusing.

But the code works :-)

[valentt]

If you scan wifi for 5 minutes, then you need around 5 seconds to switch from monitor mode to client mode, connect to wifi, get ip address via dhcp and send the data. You loose 5 seconds of scanning time, but in 5 minute window this is totally acceptable. For some even 5 seconds of 1 minute would be acceptable.

[cyberman54]

@valentt It's not necessary to manage a wifi connection while sniffing MACs. Sniffing operates in the background, even while a wifi connection is established and in use.

But of course it is necessary to stop wifi channel rotation while a wifi connection is established. This could easyly be implemented using ESP32 wifi events, calling xTimerStop(WifiChanTimer, 0); before wifi connection is established, and xTimerStart(WifiChanTimer, 0); after it is released.

Stopping wifi channel rotation will, however, reduce the MAC discovery rate, theoretically by ~90% (1/13).

Besides this, an established wifi connection could perhaps impact the sniffer performance, causing packet loss. This is an assumption, i never tested this. A way to compensate this effect could be increasing sniffing performance by using a queue for MAC processing.

You're welcome to create a PR for the wifi data transfer feature.

[cyberman54]

@davidex720 we have some news here. In development branch there is an enhancement with mqtt client, sending pax telemetrics data over ethernet interface.

[holgerkoch]

Hi,
i have a question for sending the pax telemetrics over WiFi.
In paxcounter.conf is a config item:
#define MQTT_ETHERNET 0 // select PHY: set 0 for Wifi, 1 for ethernet
and i interpret it that the paxcounter connect to wifi with the credentials from the ota.conf settings and will be send the telemetry to a mqtt server.
Unfortunatelly, he doesn't connect to wifi. If i activate the bootmenu, he connect and start the webserver successful.

Should that work?

Best regards

[cyberman54]

as stated above: wifi connect during runtime is not yet implemented. MQTT only works via Ethernet PHY.

[holgerkoch]

@cyberman54,
thank you very much for your quick response.
So i will try to use lora.
Maybe it is a good idea to add a comment, that the config item

#define MQTT_ETHERNET 0 // select PHY: set 0 for Wifi, 1 for ethernet

will not work for wifi, because it is not implemented now.

Best regards, Holger

[cyberman54]

Comment added. Thanks for the hint.

[raomin]

Hello, I would also need this feature. Before considering a PR, I would like to know if the approach described by @cyberman54 here is still the right one.

[cyberman54]

@raomin Yes, it is. An approach would be like this:

• if send cycle is due, stop wifi sniffing, e.g. by calling set_wifiscan 0 from rcommand.cpp
• start wifi stack in station mode
• connect to wifi ap and send data (e.g. using start_ota_update() from ota.cpp as boiler template)
• stop wifi stack
• restart wifi sniffing,, e.g. by calling set_wifiscan 1 from rcommand.cpp

[raomin]

I see I cannot/should not call set_wifiscan but go through

void rcommand(const uint8_t *cmd, const size_t cmdlength)

There are no implementation of this function call for a specific parameter and the implementation is not obvious at first glance...
Can you save me some time and explain how I should send one call for {0x17, set_wifiscan, 1} to 0?

In the meantime, I exposed set_wifiscan() in rcommand.h

Thanks

[cyberman54]

set_wifiscan(0) stops wifi sniffing, (1) starts

Note: of you use set_wifiscan, you don't need to take care of the channel rotation. This is done in libpax, called by set_wifiscan.

[raomin]

This I saw (and did). It's just that only rcommand() was declared in rcommand.h as this looks like the entry point of commands that are redirected to other private functions eg set_wifiscan.
But if it's ok to you to add a declaration of set_wifiscan in rcommand.h it's fine with me! :)

[cyberman54]

yes, use the private function and declare it as public, that should work.

[cyberman54]

don't touch ota.cpp and ota.h, both are not needed here.
Try to unload wifi driver from RAM, after you finished you comunication, there is a ESP-IDF API call for this.
Try using a board with additional PSRAM.

[raomin]

See my fork here.
https://github.com/raomin/ESP32-Paxcounter
I though this was a good idea to implement a WifiManager that would centralize the Wifi switches for OTA and MQTT. But if you prefer, I just duplicate code.

On your advice, I switched to an esp32s3. That's so much better.
The code works fine now on an esp32s3. I guess the ble stack just takes too much on an esp32.
No more short of memory or weird reboots. Thanks!

Now the MQTT message is a byte version encoded in base64.
I guess a json message would be more appropriate.
Are you OK with defining a MQTT_ENCODER configuration item with 0 for base64 and 1 for JSON?

[cyberman54]

/ping @davidex720 i made some edits in this comment

• Wifi:
  Wifi.begin() in arduino-esp32 was not reliable over a long time, which is fatal for OTA. Didn't catch up with latest versions. If you tested it against several Wifi APs / environments and you found it now connecting reliable it would make sense to sanitize and centralize wifi code, using same functions for OTA and MQTT.

• RAM:
  Note: Not all ESP32-S3 modules have internal PSRAM, depends on type of module.
  I think it would be possible to have code with less RAM footprint, if you unload wifi stack from RAM, right after using it. There is an ESP32 api call for this.
  There is no BLE stack in paxcounter, neither in libpax. The code is using the low level VHCI API of ESP32 to control BLE, thus does not have a significant RAM footprint.

• MQTT:
  Feel free to change MQTT encoding/protocol, but please provide an appropriate update/addendum of documentation. Thanks.
  In my opinion MQTT has the goal of keeping telemetry payloads small. Thus, conversion of payload bytes to json output should be a task of an MQTT broker, not of the MQTT client.

[raomin]

• Wifi
  I was not refering to the arduino project WifiManager but merely to the wifiManager.cpp code I've put centralizing the wifi connections and disconnections from ota.cpp and mqttclient.cpp
  Let me know if you prefer I leave ota.cpp untouched, I would remove wifiManager.cpp then.

• RAM
  Yes, I'm using an esp32-s3, with psram :) But I'll also unload wifi stack. I guess calling esp_wifi_stop() then esp_wifi_deinit() should do the job...

• MQTT/JSON
  Most MQTT implementation I've worked with do not include message handler (eg nodered) that would unpack binary. It still remains within a tcp packet ;)
  Anyway, that remains an option in my implementation:
  I have the json encoding working if you want to have a look.
  Now one caveat of on demand wifi/mqtt connection is that it's unlikely that we receive a rcommand mqtt message at the same time on the subscribed topic. Unless the message is send in retain but then it would be read each time.

paxout paxcounter_9131294e
paxout/1 {'total':1, 'ble':0, 'wifi':1}

I'll put a a quick doc update to clarify MQTT use and this last point. Also a bit of code cleanup and it should be ready for PR.

[raomin]

So, I'm trying to get something working on esp32 (no psram).
Thing is, I cannot even initialize the interrupt: no more memory.

Here are my findings on memory consumption:

Sure, I could remove display, and it would work, or just stick to psram... But is it hopeless to slim up libpax memory consumption?

[cyberman54]

That's strange. Libpax assigns 2KB of DRAM as bitmap for storing identifiers. It does not load a wifi stack, neither a bluetooth stack. It does not include any external libraries. How did you create the memory map?

[raomin]

ESP.getFreeHeap() at every step of the setup.
Going deeper in this rabbit hole, that's the wifi_sniffer_init that consumes most of the 98kB. Nothing much we can do here I think...

[cyberman54]

Thanks for investigating this. Wifi_sniffer_init in libpax calls esp32 idf functions, i wouldn't expect that these cause such big impact on RAM. Maybe it's the promiscous mode callback which is called with every wifi packet arriving. How many wifi devices are counted in your test environment by paxcounter?

[raomin]

That's memory usage during setup(). No packets are received at this level.

[cyberman54]

Libpax is using static memory. It's assigned to a bitmap array, at this point in code:
https://github.com/dbSuS/libpax/blob/master/lib/libpax/libpax.cpp#L33
The calculation of array size should result to 2KB. But maybe it isn't for some reason?

[cyberman54]

@raomin
Could you please try to switch off wifi sniffer, only use BLE sniffer, then again check free ram?

1. Set WIFICOUNTER 0 and BLECOUNTER 1 in paxcounter.conf
2. Delete NVRAM by pio run -t erase
3. Rebuild and reflash software
4. Run and inspect RAM
   Thanks.

[raomin]

Libpax is using static memory. It's assigned to a bitmap array, at this point in code: https://github.com/dbSuS/libpax/blob/master/lib/libpax/libpax.cpp#L33 The calculation of array size should result to 2KB. But maybe it isn't for some reason?

Just to clarify what I'm doing for memory measurement, at every step of setup() I'm printing ESP.getFreeHeap() and deduce what is consumed by the function just called. So static memory is consumed before I get anything printed.
Also, because of the use of multithreading, there could be a side impact on the memory that I wrongly assign to Wifi_sniffer_init...
Let's see with WIFICOUNTER 0...

[cyberman54]

Maybe a false positive?

ESP.getFreeHeap() boils down to RTOS heap_caps_get_free_size(MALLOC_CAP_INTERNAL).
This shows only a part of the whole memory.

Found here: https://esp32.com/viewtopic.php?t=27780

[raomin]

Maybe a false positive?

Not really, after my init on a tdisplay, with DISP, BLE, WIFI and MQTT, ESP.getFreeHeap() return less than 4Kb and it crashes when it tries to initiate the 4Kb of irqHandlers for the state machine. If I disable DISP for ex, getFreeHeap is higher and I pass the irqHandlers. So that's pretty representative of what memory is left.

I tried with WIFICOUNTER to 0 and free heap is now 74kb more than when WIFICOUNTER is to 1.

[cyberman54]

Did some investigation. The esp32 wifi driver consumes around 71 KB of RAM. It's memory footprint can be lowered by tuning some buffer sizes, see https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/wifi.html#why-buffer-configuration-is-important

I found that this can free up 10 - 15 KB of RAM.

[raomin]

So I guess we should see with static void useStaticBuffers(bool bufferMode); to true...

  WiFi.useStaticBuffers(true);

... in main.cpp
I'll have a look

[cyberman54]

Paxcounter generally does not use Wifi class, except for OTA and Boot-Menu, but it then reboots before to stop libpax and free RAM, before running ota.cpp / boot.cpp.

[raomin]

But then, is the sniffer actually using these buffers?

[cyberman54]

Not sure. I did not find a documentation source yet which explains the sniffer mode of esp32 api.

[cyberman54]

@raomin You should not use Arduino WIFI class function calls from within paxcounter code, because paxcounter delegates control of wifi to the libpax library. Libpax is based on esp32 api wifi function calls, not on Arduino WIFI class.

Exceptions are ota.cpp and boot.cpp which are using Arduino WIFI class to establish a tcp/ip connect over wifi, but this happenes in a dedicated runmode, after the module is rebooted for, thus libpax is not active while ota.cpp or boot.cpp is running.

Regarding memory buffers for wifi usage, what Arduino WIFI class does is setting the RX/TX buffer values to it's own configuration, found here. In libpax code buffer adjustments would need to happen here. Libpax lib does not expose this to it's api, so there won't be a straight forward way to modify wifi buffers from within paxcounter code.

Regarding your feature, establishing a tcp/ip connection over wifi to send payload, the flow would be:
0 - paxcounter is running libpax, Arduino Wifi stack is not initialized
1 - stop libpax by calling libpax_counter_stop(), this causes libpax to deinit and unload esp32 wifi stack (freeing ~70KB)
2 - initialize Arduino Wifi stack with RX/TX buffer configuration of you flavour and do your tranmission work
3- deinit Arduino Wifi stack and reload libpax by calling init_libpax()

That's an awful workaround. Reason is that paxcounter and libpax originally were development to transmit payload not on Wifi, but on other channels like Lorawan or SPI or logging to SD card. Since Wifi is used for sniffing. But with the above scheme both could be combined. You're welcome to post a PR for this!

[raomin]

Thanks for the clarification. That's more or less what I did already with set_wifiscan(0) which calls libpax_counter_stop().
It's just that I'm troubled with the few memory we have left which is not enough for the wifi stack, the MQTT stack, the display and the sniffing of BLE and WIFI. That leaves us with either unloading everything (which I find challenging in a nice way) or... using psram :)

[cyberman54]

A preliminary version of this feature can be found in branch mqtt-over-wifi