3.8.0: quorum queue does not emit stats, logs exceptions in rabbit_fifo:dead_letter_effects/4

[sFrenkie]

What:

• RabbitMQ was not able to show any queue stats about metrics-processor-json-input queue
• queue does not habe leader node

We observed log:

** Last event = {cast,go}
** When server state  = [{id,{'%2F_metrics-processor-json-input','rabbit@rabbitmq-ha-2.rabbitmq-ha-discovery.ingest.svc.cluster.local'}},{opt,terminate},{raft_state,recover},{ra_server_state,#{cluster => #{{'%2F_metrics-processor-json-input','rabbit@rabbitmq-ha-0.rabbitmq-ha-discovery.ingest.svc.cluster.local'} => #{commit_index_sent => 0,match_index => 0,next_index => 1,query_index => 0},{'%2F_metrics-processor-json-input','rabbit@rabbitmq-ha-1.rabbitmq-ha-discovery.ingest.svc.cluster.local'} => #{commit_index_sent => 0,match_index => 0,next_index => 1,query_index => 0},{'%2F_metrics-processor-json-input','rabbit@rabbitmq-ha-2.rabbitmq-ha-discovery.ingest.svc.cluster.local'} => #{commit_index_sent => 0,match_index => 0,next_index => 1,query_index => 0}},commit_index => ...,...}}]
** Reason for termination = error:{case_clause,{1,{'$prefix_msg',#{delivery_count => 2,size => 262}},[]}}
** Callback mode = [state_functions,state_enter]
** Stacktrace =
**  [{rabbit_fifo,'-dead_letter_effects/4-anonymous-0-',4,[{file,"src/rabbit_fifo.erl"},{line,1117}]},{maps,fold_1,3,[{file,"maps.erl"},{line,232}]},{rabbit_fifo,dead_letter_effects,4,[{file,"src/rabbit_fifo.erl"},{line,1117}]},{rabbit_fifo,apply,3,[{file,"src/rabbit_fifo.erl"},{line,185}]},{ra_server,apply_with,2,[{file,"src/ra_server.erl"},{line,1887}]},{ra_server,'-recover/1-fun-0-',2,[{file,"src/ra_server.erl"},{line,297}]},{ra_server,'-apply_to/5-lists^foldl/2-0-',3,[{file,"src/ra_server.erl"},{line,1845}]},{ra_server,apply_to,5,[{file,"src/ra_server.erl"},{line,1845}]}]
2021-02-01 13:29:31.553 [error] <0.29710.0> CRASH REPORT Process '%2F_metrics-processor-json-input' with 0 neighbours crashed with reason: no case clause matching {1,{'$prefix_msg',#{delivery_count => 2,size => 262}},[]} in rabbit_fifo:'-dead_letter_effects/4-anonymous-0-'/4 line 1117
2021-02-01 13:29:31.553 [error] <0.552.0> Supervisor {<0.552.0>,ra_server_sup} had child '%2F_metrics-processor-json-input' started with ra_server_proc:start_link(#{await_condition_timeout => 30000,broadcast_time => 100,cluster_name => '%2F_metrics-processor-json-input',...}) at <0.29710.0> exit with reason no case clause matching {1,{'$prefix_msg',#{delivery_count => 2,size => 262}},[]} in rabbit_fifo:'-dead_letter_effects/4-anonymous-0-'/4 line 1117 in context child_terminated
2021-02-01 13:29:38.741 [info] <0.29782.0> queue 'metrics-processor-json-input' in vhost '/': terminating with {case_clause,{1,{'$prefix_msg',#{delivery_count => 2,size => 262}},[]}} in state recover
2021-02-01 13:29:38.773 [error] <0.29782.0> ** State machine '%2F_metrics-processor-json-input' terminating
** Last event = {cast,go}
** When server state  = [{id,{'%2F_metrics-processor-json-input','rabbit@rabbitmq-ha-2.rabbitmq-ha-discovery.ingest.svc.cluster.local'}},{opt,terminate},{raft_state,recover},{ra_server_state,#{cluster => #{{'%2F_metrics-processor-json-input','rabbit@rabbitmq-ha-0.rabbitmq-ha-discovery.ingest.svc.cluster.local'} => #{commit_index_sent => 0,match_index => 0,next_index => 1,query_index => 0},{'%2F_metrics-processor-json-input','rabbit@rabbitmq-ha-1.rabbitmq-ha-discovery.ingest.svc.cluster.local'} => #{commit_index_sent => 0,match_index => 0,next_index => 1,query_index => 0},{'%2F_metrics-processor-json-input','rabbit@rabbitmq-ha-2.rabbitmq-ha-discovery.ingest.svc.cluster.local'} => #{commit_index_sent => 0,match_index => 0,next_index => 1,query_index => 0}},commit_index => ...,...}}]
** Reason for termination = error:{case_clause,{1,{'$prefix_msg',#{delivery_count => 2,size => 262}},[]}}

Quorum status:

bash-5.0$ rabbitmqctl quorum_status metrics-processor-json-input                        
Status of quorum queue metrics-processor-json-input on node rabbit@rabbitmq-ha-2.rabbitmq-ha-discovery.ingest.svc.cluster.local ...
┌─────────────────────────────────────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────┬───────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name                                                           │ Raft State                                                                   │ Log Index │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├─────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@rabbitmq-ha-2.rabbitmq-ha-discovery.ingest.svc.cluster.local │ {{case_clause,{1,{'$prefix_msg',#{delivery_count => 2,size => 262}},[]}},    │           │              │                │      │                 │
│                                                                     │  [{rabbit_fifo,'-dead_letter_effects/4-anonymous-0-',4,                      │           │              │                │      │                 │
│                                                                     │                [{file,"src/rabbit_fifo.erl"},{line,1117}]},                  │           │              │                │      │                 │
│                                                                     │   {maps,fold_1,3,[{file,"maps.erl"},{line,232}]},                            │           │              │                │      │                 │
│                                                                     │   {rabbit_fifo,dead_letter_effects,4,                                        │           │              │                │      │                 │
│                                                                     │                [{file,"src/rabbit_fifo.erl"},{line,1117}]},                  │           │              │                │      │                 │
│                                                                     │   {rabbit_fifo,apply,3,[{file,"src/rabbit_fifo.erl"},{line,185}]},           │           │              │                │      │                 │
│                                                                     │   {ra_server,apply_with,2,[{file,"src/ra_server.erl"},{line,1887}]},         │           │              │                │      │                 │
│                                                                     │   {ra_server,'-recover/1-fun-0-',2,[{file,"src/ra_server.erl"},{line,297}]}, │           │              │                │      │                 │
│                                                                     │   {ra_server,'-apply_to/5-lists^foldl/2-0-',3,                               │           │              │                │      │                 │
│                                                                     │              [{file,"src/ra_server.erl"},{line,1845}]},                      │           │              │                │      │                 │
│                                                                     │   {ra_server,apply_to,5,[{file,"src/ra_server.erl"},{line,1845}]}]}          │           │              │                │      │                 │
├─────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@rabbitmq-ha-1.rabbitmq-ha-discovery.ingest.svc.cluster.local │ {{case_clause,{1,{'$prefix_msg',#{delivery_count => 2,size => 262}},[]}},    │           │              │                │      │                 │
│                                                                     │  [{rabbit_fifo,'-dead_letter_effects/4-anonymous-0-',4,                      │           │              │                │      │                 │
│                                                                     │                [{file,"src/rabbit_fifo.erl"},{line,1117}]},                  │           │              │                │      │                 │
│                                                                     │   {maps,fold_1,3,[{file,"maps.erl"},{line,232}]},                            │           │              │                │      │                 │
│                                                                     │   {rabbit_fifo,dead_letter_effects,4,                                        │           │              │                │      │                 │
│                                                                     │                [{file,"src/rabbit_fifo.erl"},{line,1117}]},                  │           │              │                │      │                 │
│                                                                     │   {rabbit_fifo,apply,3,[{file,"src/rabbit_fifo.erl"},{line,185}]},           │           │              │                │      │                 │
│                                                                     │   {ra_server,apply_with,2,[{file,"src/ra_server.erl"},{line,1887}]},         │           │              │                │      │                 │
│                                                                     │   {ra_server,'-recover/1-fun-0-',2,[{file,"src/ra_server.erl"},{line,297}]}, │           │              │                │      │                 │
│                                                                     │   {ra_server,'-apply_to/5-lists^foldl/2-0-',3,                               │           │              │                │      │                 │
│                                                                     │              [{file,"src/ra_server.erl"},{line,1845}]},                      │           │              │                │      │                 │
│                                                                     │   {ra_server,apply_to,5,[{file,"src/ra_server.erl"},{line,1845}]}]}          │           │              │                │      │                 │
├─────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@rabbitmq-ha-0.rabbitmq-ha-discovery.ingest.svc.cluster.local │ timeout                                                                      │           │              │                │      │                 │
└─────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────┴───────────┴──────────────┴────────────────┴──────┴─────────────────┘

Queue: Durable + Quorum + DLX
We were not able purge queue in UI or CMD with rabbitmqctl. Tip rabbitmqctl eval '{ok, Q} = rabbit_amqqueue:lookup(rabbit_misc:r(<<"/">>, queue, <<"metrics-processor-json-input">>)), rabbit_amqqueue:delete_crashed(Q).' from link worked. We create queue again.

RabbtiMQ version: 3.8.0
Log from admin UI after try to purge queue:

Request to node rabbit@rabbitmq-ha-0.rabbitmq-ha-discovery.ingest.svc.cluster.local failed with {'EXIT',
{{{case_clause,
{1,
{'$prefix_msg',
#{delivery_count =>
2,
size =>
262}},
[]}},
[{rabbit_fifo,
'-dead_letter_effects/4-anonymous-0-',
4,
[{file,
"src/rabbit_fifo.erl"},
{line,
1117}]},
{maps,
fold_1,
3,
[{file,
"maps.erl"},
{line,
232}]},
{rabbit_fifo,
dead_letter_effects,
4,
[{file,
"src/rabbit_fifo.erl"},
{line,
1117}]},
{rabbit_fifo,
apply,
3,
[{file,
"src/rabbit_fifo.erl"},
{line,
185}]},
{ra_server,
apply_with,
2,
[{file,
"src/ra_server.erl"},
{line,
1887}]},
{ra_server,
'-recover/1-fun-0-',
2,
[{file,
"src/ra_server.erl"},
{line,
297}]},
{ra_server,
'-apply_to/5-lists^foldl/2-0-',
3,
[{file,
"src/ra_server.erl"},
{line,
1845}]},
{ra_server,
apply_to,
5,
[{file,
"src/ra_server.erl"},
{line,
1845}]}]},
{gen_statem,
call,
[{'%2F_metrics-processor-json-input',
'rabbit@rabbitmq-ha-0.rabbitmq-ha-discovery.ingest.svc.cluster.local'},
{leader_call,
{command,
normal,
{'$usr',
{purge},
await_consensus}}},
5000]}}}

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

There have been quite a few fixes specifically to quorum queues but also its underlying Raft implementation and RabbitMQ in general since October 2019 when 3.8.0 shipped. Please upgrades.

[michaelklishin]

The stack trace looks identical to #2256 addressed in Feb 2020 and shipped in 3.8.3.

[sFrenkie]

Thank you. I was not to able to pick up right things from stacktrace to google it. I will update RabbitMQ in our project.