The retention is not working due to non closing ledger

[alexku7]

Describe the bug

I have been researching the retention problem and high disk usage.
Currently the retention is defined to 2 hours, ttl 600 sec.
"managedLedgerMaxLedgerRolloverTimeMinutes" : "240",
"managedLedgerCursorRolloverTimeInSeconds" : "14400"
"managedLedgerCursorMaxEntriesPerLedger" : "50000"
"managedLedgerMaxEntriesPerLedger" : "50000",

pulsar version is 2.5.0

if i run bin/pulsar-admin topics stats-internal persistent://internal/eyecloud/test591-InputEvents i get the following output

{
"entriesAddedCounter" : 75562,
"numberOfEntries" : 75562,
"totalSize" : 8430787149,
"currentLedgerEntries" : 75562,
"currentLedgerSize" : 8430787149,
"lastLedgerCreatedTimestamp" : "2020-05-09T15:19:16.289Z",
"waitingCursorsCount" : 3,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "1073:75561",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 1073,
"entries" : 0,
"size" : 0,
"offloaded" : false
} ],
"cursors" : {
"InputEvents-SourceReader" : {
"markDeletePosition" : "1073:75561",
"readPosition" : "1073:75562",
"waitingReadOp" : false,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 75562,
"cursorLedger" : 1075,
"cursorLedgerLastEntry" : 428,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2020-05-09T15:19:16.292Z",
"state" : "Open",
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"properties" : { }
}

If i understand the output correctly , the ledger was opened almost 1 day ago, it has more then 50000 entries and according to the policies should be closed.

This is relatively a big topic and allocated lot of disk space. i suspect that it's not deleted because the ledger is still opened. I see the same things on the other similar topics .
Is my assumption correct ?

[sijie]

Which version of Pulsar are you using? Before 2.5.1, if there is no incoming traffic, the ledger rollover will not be triggered.

[alexku7]

Hi
We are using 2.5.0

What do you mean no incoming traffic ?
I think the limit of 50k entries and/or the limit of 240 minutes should cause to rollover. Am i right ?

Anyway, what the solution in this case? The topic has 8 Gb data bas doesn't have any incoming traffic , so the ledger is the open state. So the retention will not occur ever ?

[alexku7]

Hi @sijie
Recently We upgraded to 2.5.1 but unfortunately it seems that the bug is non resolved.

The ledgers are still in open state and not deleted :(

We have some scenario when a producer send huge amount of messages in a short period to the topic and "closes" it and opens a new one.

such topic contains a lot of messages and because of the ledger is remained opened forever , the topic is stored forever. After a while ( few days) we found ourselves with full disks and failed pulsar.

Actually I succeeded to reproduce this bug . If i take one of the problematic topics and simply simulate a little activity by sending messages during few minutes, the ledgers is closed and recycled.

We need help:) That very critical bug for us and i think for other customers

[sijie]

@alexku7 I see. It seems that we need to make sure a ledger to be closed if there are no more incoming traffic to that topic.

[sijie]

@alexku7 we will pick this item up to fix it for 2.6.0 release.

[alexku7]

Thank you @sijie
Do you know when the ver 2.6.0 is expected to be released ?

[sijie]

@alexku7 it is coming this month. @codelipenghui is the release manager and is already starting on wrapping up the 2.6.0 release.

[trexinc]

Simple instruction to replicate issue (replicates in 2.5.2 as well):
Create namespace internal/test with the following retention and TTL:

bin/pulsar-admin namespaces get-retention internal/test
{
"retentionTimeInMinutes" : 60,
"retentionSizeInMB" : 5120
}

bin/pulsar-admin namespaces get-message-ttl internal/test
600

Then run in parallel for about five minutes the following (you can add -m 60000 to the producer):

bin/pulsar-perf produce -r 40000 internal/test/test1
bin/pulsar-client consume -s test internal/test/test1

The topic will then remain like this forever:

bin/pulsar-admin topics stats-internal internal/test/test1
{
"entriesAddedCounter" : 64431,
"numberOfEntries" : 64431,
"totalSize" : 6548489946,
"currentLedgerEntries" : 64431,
"currentLedgerSize" : 6548489946,
"lastLedgerCreatedTimestamp" : "2020-05-23T04:42:56.047Z",
"waitingCursorsCount" : 0,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "182338:64430",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 182338,
"entries" : 0,
"size" : 0,
"offloaded" : false
} ],
"cursors" : {
"sub" : {
"markDeletePosition" : "182338:64430",
"readPosition" : "182338:64431",
"waitingReadOp" : false,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 64431,
"cursorLedger" : 182339,
"cursorLedgerLastEntry" : 108,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2020-05-23T04:42:56.049Z",
"state" : "Open",
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"properties" : { }
}
}
}

[govardhan1194]

Hey,

I am working on a project with pulsar and I am facing the same issue when it comes to offloading the data to s3. The ledger doesnt seem to close.
Is there any update on when the fixed version would be released?

Thanks in advance!

[alexku7]

Hello @sijie
Recently, we have upgraded to pulsar 2.6.0 but unfortunately the issue still exists :(
It seems that the topic still is not deleted because of opened ledger.

I am just wondering if the fix has been merged into 2.6.0 ?
I