Fix Lucene text index realtime to offline conversion

[SabrinaZhaozyf]

We have observed java.io.FileNotFoundException after #12744 when converting the mutable lucene text index.

Specifically, the path generated by getMutableIndexDir does not point to the existing mutable index.

ERROR [RealtimeSegmentDataManager_RealtimeFeatureTest1__0__0__20240423T1743Z] [RealtimeFeatureTest1__0__0__20240423T1743Z] [pinot-server] [] Could not build segment
java.lang.RuntimeException: Caught exception while instantiating the LuceneTextIndexCreator for column: textDim1
        at org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator.<init>(LuceneTextIndexCreator.java:147
...
Caused by: java.lang.RuntimeException: Failed to convert the mutable lucene index: java.io.FileNotFoundException: Directory '<path-to-dataDir>/dataDir/RealtimeFeatureTest1_REALTIME/consumers/RealtimeFeatureTest1__0__0__20240423T1743Z/textDim1.lucene.v9.index' does not exist.
    at org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator.convertMutableSegment(LuceneTextIndexCreator.java:190)
    at org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator.<init>(LuceneTextIndexCreator.java:137)

getMutableIndexDir works for paths where there's no consumer dir configured. However, it does not work when consumer dir is configured. Ideally, the path should align with getConsumerDir

cc @itschrispeck @chenboat

[stationeros]

Can this be assigned to me ?

[itschrispeck]

Hey @stationeros - sorry, but I felt a bit responsible for introducing this bug and have raised a fix to unblock others.

[hpvd]

@SabrinaZhaozyf does this merged bugfix solve this issue for you?
If so, we could close this issue...