Proxy writer deletion fails with assertion in entity index

[dpotman]

Deletion of a proxy writer sometimes fails with an assertion when running the test ddsc_listener_matched under high load. A stack trace of the failing delete:

#3  0x00007f6bf4ae6f36 in __GI___assert_fail (assertion=0x7f6bf520f140 <str> "ddsrt_avl_lookup (&all_entities_treedef, &ei->all_entities, e) != NULL",
    file=0x7f6bf520e440 <str> "/dev/cyclonedds/src/core/ddsi/src/ddsi_entity_index.c", line=267,
    function=0x7f6bf520f1c0 <__PRETTY_FUNCTION__.remove_from_all_entities> "void remove_from_all_entities(struct entity_index *, struct entity_common *)")
    at assert.c:101
#4  0x00007f6bf4f727f5 in remove_from_all_entities (ei=0x606000151340, e=0x6140001da840) at /dev/cyclonedds/src/core/ddsi/src/ddsi_entity_index.c:267
#5  0x00007f6bf4f6e4cd in entity_index_remove (ei=0x606000151340, e=0x6140001da840) at /dev/cyclonedds/src/core/ddsi/src/ddsi_entity_index.c:284
#6  0x00007f6bf4f6e5fd in entidx_remove_proxy_writer_guid (ei=0x606000151340, pwr=0x6140001da840)
    at /dev/cyclonedds/src/core/ddsi/src/ddsi_entity_index.c:365
#7  0x00007f6bf504e89e in delete_proxy_writer (gv=0x62300000c7b8, guid=0x7fff04c4cfc0, timestamp=..., isimplicit=1)
    at /dev/cyclonedds/src/core/ddsi/src/q_entity.c:6318
#8  0x00007f6bf5047463 in delete_ppt (proxypp=0x6130000814c0, timestamp=..., isimplicit=1) at /dev/cyclonedds/src/core/ddsi/src/q_entity.c:5489
#9  0x00007f6bf504652e in delete_proxy_participant_by_guid (gv=0x62300000c7b8, guid=0x61600000a2d8, timestamp=..., isimplicit=1)
    at /dev/cyclonedds/src/core/ddsi/src/q_entity.c:5607
#10 0x0000000000624450 in dodeaf (ctx=0x7fff04c4d820) at /dev/cyclonedds/src/core/ddsc/tests/test_oneliner.c:1694
#11 0x0000000000617cda in dispatchcmd (ctx=0x7fff04c4d820) at /dev/cyclonedds/src/core/ddsc/tests/test_oneliner.c:1856
#12 0x00000000006147b9 in test_oneliner_step1 (ctx=0x7fff04c4d820) at /dev/cyclonedds/src/core/ddsc/tests/test_oneliner.c:1884
#13 0x0000000000614654 in test_oneliner_step (ctx=0x7fff04c4d820,
    ops=0x678e00 <str> "sm da r pm w' ; ?sm r ?pm w' ; r'' ?pm w' deaf P'' ; wr w' 1   ; ?da r take{(1,0,0)} r ?ack(r) w' ; deaf P    ; ?sm(1,0,0,-1,w') r ?da r take{d1} r ; wr w' 2 ; hearing P ; ?sm(2,1,1,1,w') r  ?da r sle"...) at /dev/cyclonedds/src/core/ddsc/tests/test_oneliner.c:1920
#14 0x0000000000617433 in test_oneliner (
    ops=0x678e00 <str> "sm da r pm w' ; ?sm r ?pm w' ; r'' ?pm w' deaf P'' ; wr w' 1   ; ?da r take{(1,0,0)} r ?ack(r) w' ; deaf P    ; ?sm(1,0,0,-1,w') r ?da r take{d1} r ; wr w' 2 ; hearing P ; ?sm(2,1,1,1,w') r  ?da r sle"...) at /dev/cyclonedds/src/core/ddsc/tests/test_oneliner.c:1974

However, before deleting, the same (memory address and guid) proxy writer was added:

0  entidx_insert_proxy_writer_guid (ei=0x606000151340, pwr=0x6140001da840) at /dev/cyclonedds/src/core/ddsi/src/ddsi_entity_index.c:335
#1  0x00007f6bf504b92e in new_proxy_writer (gv=0x62300000c7b8, ppguid=0x7f6be54d5a38, guid=0x7f6be54d4320, as=0x607000075730, plist=0x7f6be54d3fa0,
    dqueue=0x60d000001970, evq=0x610000000f40, timestamp=..., seq=0) at /dev/cyclonedds/src/core/ddsi/src/q_entity.c:6153
#2  0x00007f6bf506459b in create_proxy_builtin_endpoint_impl (gv=0x62300000c7b8, timestamp=..., ppguid=0x7f6be54d5a38, proxypp=0x6130000814c0, ep_guid=0x7f6be54d4320,
    plist=0x7f6be54d3fa0, topic_name=0x7f6bf522a300 <str> "DCPSSubscription") at /dev/cyclonedds/src/core/ddsi/src/q_entity.c:4810
#3  0x00007f6bf5063ffd in create_proxy_builtin_endpoints (gv=0x62300000c7b8, bestab=0x7f6bf5293e00 <add_proxy_builtin_endpoints.bestab_default>, nbes=9,
    ppguid=0x7f6be54d5a38, proxypp=0x6130000814c0, timestamp=..., xqos_wr=0x62300000d5c0, xqos_rd=0x62300000d438)
    at /dev/cyclonedds/src/core/ddsi/src/q_entity.c:4846
#4  0x00007f6bf50451af in add_proxy_builtin_endpoints (gv=0x62300000c7b8, ppguid=0x7f6be54d5a38, proxypp=0x6130000814c0, timestamp=...)
    at /dev/cyclonedds/src/core/ddsi/src/q_entity.c:4884
#5  0x00007f6bf5044e29 in new_proxy_participant (gv=0x62300000c7b8, ppguid=0x7f6be54d5a38, bes=64575, privileged_pp_guid=0x7f6be54d4ca0, as_default=0x6070000756c0,
    as_meta=0x607000075730, plist=0x7f6be54d57f0, tlease_dur=10000000000, vendor=..., custom_flags=0, timestamp=..., seq=1)
    at /dev/cyclonedds/src/core/ddsi/src/q_entity.c:5265
#6  0x00007f6bf500215b in handle_spdp_alive (rst=0x7f6be1acd640, seq=1, timestamp=..., datap=0x7f6be54d57f0)
    at /dev/cyclonedds/src/core/ddsi/src/q_ddsi_discovery.c:935

[thijsmie]

Hi @dpotman, I think this was fixed by your merge request? If so, you may close this issue.

[dpotman]

No I don't think that #900 fixes this issue. I spend some time to find a cause for this issue, but did not find anything yet. So lets keep this issue open for now.