ble scan does not work after driver update

[alexiionescu]

After the driver update on 27 June the ble scan will crash after hci start scan command.
Last know working version is revision abf6d4b (before driver updates)
To reproduce use project esp-ble-scan
There is commented lines in Cargo.toml for last known working configuration.

INFO - scan started
Exception 'Load access fault' mepc=0x400587d2, mtval=0x00000003
0x400587d2 - r_lld_init_evt_end_type_check_state_get
    at ??:??
0x00000003 - _max_hart_id
    at ??:??
TrapFrame
PC=0x400587d2         RA/x1=0x400051d0      SP/x2=0x3fc85ff8      GP/x3=0x3fccf2f0      TP/x4=0x00000000
0x400587d2 - r_lld_init_evt_end_type_check_state_get
    at ??:??
0x400051d0 - r_lld_ext_scan_dynamic_pti_get
    at ??:??
0x3fc85ff8 - __global_pointer$
    at ??:??
0x3fccf2f0 - _heap_start
    at ??:??
0x00000000 - _max_hart_id
    at ??:??
T0/x5=0xf6422a88      T1/x6=0x4000151e      T2/x7=0x00000000      S0/FP/x8=0x3fc88520   S1/x9=0x3fce0000
0xf6422a88 - _rtc_fast_data_start
    at ??:??
0x4000151e - r_sch_plan_interval_req
    at ??:??
0x00000000 - _max_hart_id
    at ??:??
0x3fc88520 - _ZN8esp_wifi9HEAP_DATA17hb399b07eb12ef028E
    at ??:??
0x3fce0000 - ets_ops_table_ptr
    at ??:??
A0/x10=0x3fc88520     A1/x11=0x00000003     A2/x12=0x3fccf304     A3/x13=0x3fc8b240     A4/x14=0x7f957824
0x3fc88520 - _ZN8esp_wifi9HEAP_DATA17hb399b07eb12ef028E
    at ??:??
0x00000003 - _max_hart_id
    at ??:??
0x3fccf304 - _heap_start
    at ??:??
0x3fc8b240 - _ZN8esp_wifi9HEAP_DATA17hb399b07eb12ef028E
    at ??:??
0x7f957824 - _rtc_fast_data_start
    at ??:??
A5/x15=0x3fc88520     A6/x16=0x00000000     A7/x17=0x000000a7     S2/x18=0x3fc8be16     S3/x19=0x3fc8b214
0x3fc88520 - _ZN8esp_wifi9HEAP_DATA17hb399b07eb12ef028E
    at ??:??
0x00000000 - _max_hart_id
    at ??:??
0x000000a7 - _max_hart_id
    at ??:??
0x3fc8be16 - _ZN8esp_wifi9HEAP_DATA17hb399b07eb12ef028E
    at ??:??
0x3fc8b214 - _ZN8esp_wifi9HEAP_DATA17hb399b07eb12ef028E
    at ??:??
S4/x20=0x3fcdffac     S5/x21=0x00000002     S6/x22=0x3fccf364     S7/x23=0x00007805     S8/x24=0x3fce0000
0x3fcdffac - lld_scan_env
    at ??:??
0x00000002 - _max_hart_id
    at ??:??
0x3fccf364 - _heap_start
    at ??:??
0x00007805 - _hart_stack_size
    at ??:??
0x3fce0000 - ets_ops_table_ptr
    at ??:??
S9/x25=0x40001418     S10/x26=0x3ff1b000    S11/x27=0x3fccf438    T3/x28=0x01001000     T4/x29=0x00000000
0x40001418 - r_rf_rssi_convert
    at ??:??
0x3ff1b000 - ets_ops_table_ptr
    at ??:??
0x3fccf438 - _heap_start
    at ??:??
0x01001000 - _hart_stack_size
    at ??:??
0x00000000 - _max_hart_id
    at ??:??
T5/x30=0x00000000     T6/x31=0x00007e97
0x00000000 - _max_hart_id
    at ??:??
0x00007e97 - _hart_stack_size
    at ??:??

MSTATUS=0x00001801
0x00001801 - _hart_stack_size
    at ??:??
MCAUSE=0x00000005
0x00000005 - _max_hart_id
    at ??:??
MTVAL=0x00000003
0x00000003 - _max_hart_id
    at ??:??

[bjoernQ]

Oh, that's bad

Maybe it's "just" something like in cda6146

There were some ROM functions which should not get used and it should use the replacement contained in the driver

[bjoernQ]

Seems like the driver contains a lot of replacements for ROM functions. I tried to remove most of these (not the ones which are unlikely to be involved) but it didn't help unfortunately

r_hci_register_vendor_desc_tab
r_bt_bb_isr
r_ke_task_schedule
r_llc_loc_con_upd_proc_continue
r_llc_rem_con_upd_proc_continue
r_llc_hci_command_handler
r_lld_llcp_rx_ind_handler
r_lld_con_sched
r_lld_con_stop
r_lld_per_adv_sched
r_lld_scan_process_pkt_rx
r_lld_scan_process_pkt_rx_adv_rep
r_llm_le_features_get
r_bt_rf_coex_conn_phy_coded_data_time_limit_en_get
r_rf_txpwr_cs_get
r_rf_txpwr_dbm_get
r_bt_rtp_get_txpwr_idx_by_act
r_rwble_isr
r_sch_arb_event_start_isr
r_sch_plan_set
r_sch_prog_end_isr
r_btdm_task_post
r_btdm_task_post_from_isr
r_btdm_task_recycle
r_register_esp_vendor_cmd_handler

rom_phy_xpd_rf
rom_pbus_xpd_tx_on
read_hw_noisefloor
rom_index_to_txbbgain
rom_txbbgain_to_index

coex_bt_release
coex_bt_request
coex_event_duration_get
coex_status_get
coex_wifi_release
coex_core_pti_get
coex_core_release
coex_core_request
coex_core_status_get
coex_hw_timer_disable
coex_hw_timer_enable
coex_hw_timer_set
coex_schm_interval_set
coex_schm_lock
coex_schm_unlock



esf_buf_alloc
esf_buf_alloc_dynamic
esf_buf_recycle
ic_ebuf_alloc
ic_ebuf_recycle_rx
ic_ebuf_recycle_tx
ic_get_trc
ic_interface_enabled
ic_mac_deinit
ic_mac_init
ic_reset_rx_ba
GetAccess
is_lmac_idle
lmacAdjustTimestamp
lmacDiscardAgedMSDU
lmacDiscardMSDU
lmacEndFrameExchangeSequence
lmacIsIdle
lmacIsLongFrame
lmacMSDUAged
lmacPostTxComplete
lmacProcessAllTxTimeout
lmacProcessCollisions
lmacProcessRxSucData
lmacReachLongLimit
lmacReachShortLimit
lmacRecycleMPDU
lmacReleaseTxopQueue
lmacRequestTxopQueue
lmacRxDone
lmacSetTxFrame
lmacTxFrame
pm_allow_tx
pm_check_state
pm_disable_dream_timer
pm_disable_sleep_delay_timer
pm_dream
pm_enable_active_timer
pm_enable_sleep_delay_timer
pm_is_in_wifi_slice_threshold
pm_is_waked
pm_keep_alive
pm_local_tsf_process
pm_mac_sleep
pm_mac_wakeup
pm_on_beacon_rx
pm_on_data_rx
pm_on_tbtt
pm_parse_beacon
pm_process_tim
pm_rx_beacon_process
pm_rx_data_process
pm_set_beacon_filter
pm_sleep
pm_sleep_for
pm_tbtt_process
pm_tx_data_done_process
pp_hdrsize
pp_post
pp_process_hmac_waiting_txq
ppAMPDU2Normal
ppAssembleAMPDU
ppCalFrameTimes
ppCalSubFrameLength
ppCalTxAMPDULength
ppCheckTxAMPDUlength
ppDequeueRxq_Locked
ppDequeueTxDone_Locked
ppDequeueTxQ
ppDisableQueue
ppEmptyDelimiterLength
ppEnqueueRxq
ppEnqueueTxDone
ppGetTxframe
ppGetTxQFirstAvail_Locked
ppMapTxQueue
ppMapWaitTxq
ppProcessRxPktHdr
ppProcessTxQ
ppProcessWaitingQueue
ppProcTxDone
ppProcTxSecFrame
ppRecordBarRRC
ppRecycleAmpdu
ppRecycleRxPkt
ppResortTxAMPDU
ppResumeTxAMPDU
ppRxFragmentProc
ppRxPkt
ppRxProtoProc
ppSearchTxframe
ppSearchTxQueue
ppSelectNextQueue
ppSubFromAMPDU
ppTask
ppTxPkt
ppTxProtoProc
ppTxqUpdateBitmap
RC_GetBlockAckTime
rc_get_trc
rc_get_trc_by_index
rcAmpduLowerRate
rcClearCurAMPDUSched
rcClearCurSched
rcClearCurStat
rcGetAmpduSched
rcGetSched
rcLowerSched
rcSetTxAmpduLimit
rcTxUpdatePer
rcUpdateAckSnr
rcUpdateRate
rcUpdateRxDone
rcUpdateTxDone
rcUpdateTxDoneAmpdu2
rcUpSched
rssi_margin
rx11NRate2AMPDULimit
TRC_AMPDU_PER_DOWN_THRESHOLD
TRC_AMPDU_PER_UP_THRESHOLD
trc_calc_duration
trc_isTxAmpduOperational
trc_onAmpduOp
TRC_PER_IS_GOOD
trc_SetTxAmpduState
trc_tid_isTxAmpduOperational
trcAmpduSetState
config_is_cache_tx_buf_enabled
wDev_AppendRxBlocks
wdev_bank_load
wdev_bank_store
wdev_csi_len_align
wDev_DiscardFrame
wDev_GetNoiseFloor
wDev_IndicateAmpdu
wDev_IndicateFrame
wdev_mac_reg_load
wdev_mac_reg_store
wdev_mac_sleep
wdev_mac_special_reg_load
wdev_mac_special_reg_store
wdev_mac_wakeup
wDev_ProcessFiq
wDev_ProcessRxSucData
wDevCheckBlockError
wdevProcessRxSucDataAll
hal_mac_is_dma_enable
hal_mac_is_low_rate_enabled
hal_mac_tx_get_blockack
hal_mac_tx_set_ppdu
mac_tx_set_duration
mac_tx_set_plcp0
mac_tx_set_htsig
mac_tx_set_plcp1
mac_tx_set_plcp2



ieee80211_crypto_decap
ieee80211_crypto_encap
ampdu_dispatch
ampdu_dispatch_all
ampdu_dispatch_as_many_as_possible
ampdu_dispatch_movement
ampdu_dispatch_upto
ieee80211_ampdu_reorder
ieee80211_ampdu_start_age_timer
ieee80211_decap
wifi_get_macaddr
wifi_rf_phy_disable
ieee80211_align_eb
ieee80211_classify
ieee80211_copy_eb_header
ieee80211_encap_esfbuf
ieee80211_is_tx_allowed
ieee80211_output_pending_eb
ieee80211_output_process
ieee80211_recycle_cache_eb
ieee80211_search_node
ieee80211_set_tx_desc
ieee80211_set_tx_pti
wifi_is_started
scan_start
sta_input
chm_is_at_home_channel
cnx_node_is_existing
cnx_node_search

aes_encrypt
md5_vector
MD5Final
MD5Init
MD5Update
hmac_md5
hmac_md5_vector

[alexiionescu]

So if I remove the function from rom_functions.x and it links, it should be OK?

[bjoernQ]

Probably those functions could be removed from rom_functions.x ..... but in my tests with your code it didn't change anything unfortunately.

From the crash it seems that the crash is in the rom's memcpy which got called from r_GF_Jacobian_Point_Addition256 (also rom) which probably got called from something in libbtdm_app.a

[alexiionescu]

But in my stack

Probably those functions could be removed from rom_functions.x ..... but in my tests with your code it didn't change anything unfortunately.

From the crash it seems that the crash is in the rom's memcpy which got called from r_GF_Jacobian_Point_Addition256 (also rom) which probably got called from something in libbtdm_app.a

On my esp32c3 I get a different stack (see above).
I do not know very well to decode that stack, so maybe I am wrong.

[bjoernQ]

it's also in your exception message .... unfortunately, espflash doesn't know about the rom code and decodes the addresses wrong - I get the exact same crash

I really have no idea what is going on there. The r_GF_Jacobian_Point_Addition256 is most likely used in crypto but I don't see why any crypto is needed for a BLE scan.

I also tested on ESP32-C2 and there it works fine

[alexiionescu]

it's also in your exception message .... unfortunately, espflash doesn't know about the rom code and decodes the addresses wrong - I get the exact same crash

I really have no idea what is going on there. The r_GF_Jacobian_Point_Addition256 is most likely used in crypto but I don't see why any crypto is needed for a BLE scan.

I also tested on ESP32-C2 and there it works fine

I have my main project with a lot more code and esp-now,gpio where it does not crash just freeze after BLE scan start.
So my opinion is that the PC is corrupt and jumps to incorrect address.

For now I have made a branch on my esp-wifi fork without the driver update so I can continue working.
Let's hope we can figure out what is wrong at some point, until then I will merge manually on that branch any new updates on the esp-wifi.
What are the major fixes/improvements that the driver update brings?

[bjoernQ]

I have my main project with a lot more code and esp-now,gpio where it does not crash just freeze after BLE scan start.

That is interesting information - might help when looking deeper into it.

What are the major fixes/improvements that the driver update brings?

We try to update the drivers from time to time mainly to enable new chips (we keep all the drivers at the same version).
That last update was really causing headaches especially for C3 and S3 BLE and it seems there is even more. Definitely good you came up with that example repo

I will try to look deeper into the drivers as soon as I get to it. There is also hope that the next driver update might fix it magically

[alexiionescu]

What are the major fixes/improvements that the driver update brings?

We try to update the drivers from time to time mainly to enable new chips (we keep all the drivers at the same version). That last update was really causing headaches especially for C3 and S3 BLE and it seems there is even more. Definitely good you came up with that example repo

I will try to look deeper into the drivers as soon as I get to it. There is also hope that the next driver update might fix it magically

Thanks.
I see a lot of activity recently on here esp32-wifi-lib.
But I do not see all the libs you are using on this repository.

[bjoernQ]

I think I know what is wrong and how to fix it ..... Hopefully will be able to do the fix on Monday

[alexiionescu]

I think I know what is wrong and how to fix it ..... Hopefully will be able to do the fix on Monday

That will be great, thanks.