Concurrent grammar handling

[danslavetskiy]

Thread 1 (Thread 0x7f1ce164f6c0 (LWP 25648)):
#0  0x00007f1d500a7602 in hash (h=0x0, k=0x7f1b98027340) at ./src/include/private/switch_hashtable_private.h:53
#1  switch_hashtable_insert_destructor (h=h@entry=0x0, k=k@entry=0x7f1b98027340, v=v@entry=0x7f1a5810e1f8, flags=flags@entry=(HASHTABLE_FLAG_FREE_KEY | HASHTABLE_DUP_CHECK), destructor=destructor@entry=0x0) at src/switch_hashtable.c:197
#2  0x00007f1d4ffa3944 in switch_core_hash_insert_destructor (hash=0x0, key=<optimized out>, data=data@entry=0x7f1a5810e1f8, destructor=destructor@entry=0x0) at src/switch_core_hash.c:148
#3  0x00007f1d1da10fe7 in recog_channel_load_grammar (data=0x7f1b980bbf00 "session:8b69b015-d4a4-445c-84a6-5fbe268e054a", type=GRAMMAR_TYPE_URI, name=0x7f1a5810e1d0 "8b69b015-d4a4-445c-84a6-5fbe268e054a", schannel=0x7f1b980b8be8) at mod_unimrcp.c:2433
#4  recog_asr_load_grammar (ah=<optimized out>, grammar=<optimized out>, name=0x7f1a5810e1d0 "8b69b015-d4a4-445c-84a6-5fbe268e054a") at mod_unimrcp.c:3298
#5  0x00007f1d4ff99874 in switch_core_asr_load_grammar (ah=0x7f1b980b8b40, grammar=grammar@entry=0x7f1a5810dd5a "{ speech-language=pl-PL, session-id=11311b8d-a5af-4282-8c57-921fcd424821, ai.lekta.asr.id=dictation, ai.lekta.asr.trace-id=11311b8d-a5af-4282-8c57-921fcd424821, ai.lekta.asr.external-id=5ce0abde-8263-"..., name=name@entry=0x7f1d502e4de7 "") at src/switch_core_asr.c:156
#6  0x00007f1d5006dbbb in switch_ivr_detect_speech (session=session@entry=0x7f1b981507f8, mod_name=mod_name@entry=0x7f1a5810dd43 "unimrcp:mrcp-asr-proxy", grammar=grammar@entry=0x7f1a5810dd5a "{ speech-language=pl-PL, session-id=11311b8d-a5af-4282-8c57-921fcd424821, ai.lekta.asr.id=dictation, ai.lekta.asr.trace-id=11311b8d-a5af-4282-8c57-921fcd424821, ai.lekta.asr.external-id=5ce0abde-8263-"..., name=name@entry=0x7f1d502e4de7 "", dest=dest@entry=0x0, ah=ah@entry=0x0) at src/switch_ivr_async.c:5482
#7  0x00007f1d5006de2b in switch_ivr_play_and_detect_speech (session=session@entry=0x7f1b981507f8, file=file@entry=0x7f1a5810dd10 "/usr/local/freeswitch/ivona/tts/silence.wav", mod_name=0x7f1a5810dd43 "unimrcp:mrcp-asr-proxy", grammar=0x7f1a5810dd5a "{ speech-language=pl-PL, session-id=11311b8d-a5af-4282-8c57-921fcd424821, ai.lekta.asr.id=dictation, ai.lekta.asr.trace-id=11311b8d-a5af-4282-8c57-921fcd424821, ai.lekta.asr.external-id=5ce0abde-8263-"..., result=result@entry=0x7f1ce164de08, input_timeout=5000, input_timeout@entry=0, args=0x0) at src/switch_ivr_async.c:4900
#8  0x00007f1d1e63aa54 in play_and_detect_speech_function (session=0x7f1b981507f8, data=<optimized out>) at mod_dptools.c:547

Thread 75 (Thread 0x7f1ce07c66c0 (LWP 22634)):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f1ce07c36d0, op=393, expected=0, futex_word=0x7f1b980b8cf8) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f1b980b8cf8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f1ce07c36d0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f1d4fdbef7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f1b980b8cf8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f1ce07c36d0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f1d4fdc18bc in __pthread_cond_wait_common (abstime=0x7f1ce07c36d0, clockid=0, mutex=0x7f1b980b8c80, cond=0x7f1b980b8cd0) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=cond@entry=0x7f1b980b8cd0, mutex=mutex@entry=0x7f1b980b8c80, abstime=abstime@entry=0x7f1ce07c36d0) at ./nptl/pthread_cond_wait.c:643
#5  0x00007f1d502cba7f in fspr_thread_cond_timedwait (cond=0x7f1b980b8cc8, mutex=0x7f1b980b8c78, timeout=timeout@entry=5000000) at locks/unix/thread_cond.c:89
#6  0x00007f1d4ff7e359 in switch_thread_cond_timedwait (cond=<optimized out>, mutex=<optimized out>, timeout=timeout@entry=5000000) at src/switch_apr.c:385
#7  0x00007f1d1da09e1d in speech_channel_destroy (schannel=schannel@entry=0x7f1b980b8be8) at mod_unimrcp.c:933
#8  0x00007f1d1da0a8ba in recog_asr_close (ah=0x7f1b980b8b40, flags=<optimized out>) at mod_unimrcp.c:3421
#9  0x00007f1d4ff993c8 in switch_core_asr_close (ah=0x7f1b980b8b40, flags=flags@entry=0x7f1ce07c3784) at src/switch_core_asr.c:233
#10 0x00007f1d50062701 in speech_callback (bug=<optimized out>, user_data=0x7f1b98139e58, type=<optimized out>) at src/switch_ivr_async.c:5171
#11 0x00007f1d4ff98257 in switch_core_media_bug_close (bug=bug@entry=0x7f1ce07c58a8, destroy=destroy@entry=SWITCH_FALSE) at src/switch_core_media_bug.c:1301
#12 0x00007f1d4ff98440 in switch_core_media_bug_remove_all_function (session=session@entry=0x7f1b981507f8, function=function@entry=0x0) at src/switch_core_media_bug.c:1269
#13 0x00007f1d4ffb605a in switch_core_session_hangup_state (session=session@entry=0x7f1b981507f8, force=force@entry=SWITCH_TRUE) at src/switch_core_state_machine.c:835
#14 0x00007f1d4ffb7655 in switch_core_session_run (session=<optimized out>) at src/switch_core_state_machine.c:612
#15 0x00007f1d4ffb20f0 in switch_core_session_thread (thread=<optimized out>, obj=0x7f1b981507f8) at src/switch_core_session.c:1727
#16 0x00007f1d4ffad7d7 in switch_core_session_thread_pool_worker (thread=0x7f1ad0008b80, obj=<optimized out>) at src/switch_core_session.c:1791
#17 0x00007f1d4fdc21c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#18 0x00007f1d4fe41ac0 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

The situation is the following:

1. ASR starts to recognize (Thread 1) and creates grammar https://github.com/freeswitch/mod_unimrcp/blob/main/mod_unimrcp.c#L2433
2. In the mean time there is hangup (Thread 75), so the grammar structures get destroyed https://github.com/freeswitch/mod_unimrcp/blob/main/mod_unimrcp.c#L3413

so grammar creation/destruction occurs in a unsynchronized manner.

[morbit85]

It seems recog_channel_enable_grammar() also requires a lock. It tries to access grammars/enabled_grammars and we invoke it right after recog_channel_disable_all_grammars() in recog_asr_load_grammar().

[danslavetskiy]

It seems recog_channel_enable_grammar() also requires a lock. It tries to access grammars/enabled_grammars and we invoke it right after recog_channel_disable_all_grammars() in recog_asr_load_grammar().

hi @morbit85 thx for review, added

[andywolk]

How about protecting recog_channel_disable_grammar as well?

[danslavetskiy]

How about protecting recog_channel_disable_grammar as well?

also added for recog_channel_unload_grammar

[danslavetskiy]

Hi @andywolk @morbit85 thx for review, could you plz help with CI?