HBASE-22749: Distributed MOB compactions

[VladRodionov]

Initial patch for HBASE-22748: Distributed MOB compactions

[busbey]

still reviewing, but trying to periodically get feedback published.

[busbey]

please don't include unrelated formatting changes.

this is also the wrong import order for our checkstyle rules. the o.a.h.h.shaded stuff is purposefully grouped later than other things. for reference see the ImportOrder entry under hbase-checkstyle/src/main/resources/hbase/checkstyle.xml:

    <module name="ImportOrder">
      <property name="groups" value="*,org.apache.hbase.thirdparty,org.apache.hadoop.hbase.shaded"/>
      <property name="option" value="top" />
      <property name="ordered" value="true"/>
      <property name="sortStaticImportsAlphabetically" value="true"/>
    </module>

Depending on what IDE you're using, I believe we have some helper configs in dev-support for getting things in line with what we check.

[busbey]

We're assuming here that GEN0 will sort before other generations based on assuming it must be for data from before we had the additional information in the mob filename, right? What if folks have been specifying timestamps?

[VladRodionov]

Not sure, why they have to do this?

[busbey]

We should prioritize compacting GEN0 because we can't clean up any mob files for a table until all the pre-mob-ref-tracking hfiles are gone.

[VladRodionov]

GEN0 files will have, by default, oldest timestamps, so they will come first. The one major assumption in new compaction also - is nobody touches MOB files except HBase system. If I got you correctly, you meant that somebody could touch MOB files and change their modification time? IN this case the system will fail, of course in several places, especially during cleaning MOB files, where we rely on correct timestamps.

[busbey]

I don't see any tests around Generation partitioning or the generational compaction. maybe I missed them?

[VladRodionov]

Created:
https://issues.apache.org/jira/browse/HBASE-23189

[busbey]

still reviewing, but trying to periodically get feedback published.

I believe I've gotten through everything for the current patch.

[busbey]

Looks like a revert was pushed instead an update to the changeset?

[busbey]

please push the current state of the work to the PR branch. it's much harder to follow conversation and track the code off of an outdated commit.

[busbey]

if we get to this point and it's taken us longer than MOB_MINIMUM_FILE_AGE_TO_ARCHIVE_KEY then we need to do something, since we know that there can easily be a disconnect between the view we had of regions while building the set of active mob files and current.

1. We could log an ERROR and abort. Good because we know we won't delete anything we need. Bad because it requires an operator noticing and making a configuration change. The cost of them not noticing is we stop removing unneeded things from HDFS, so the cluster could die by exhausting available space.
2. We could use MOB_MINIMUM_FILE_AGE_TO_ARCHIVE_KEY as a floor. Say we use an archive threshold of 2x the time to build the list of active mobs or MOB_MINIMUM_FILE_AGE_TO_ARCHIVE_KEY, whichever is greater. Good because don't risk cluster death if no one notices. Maybe bad because of how large of a window we end up with? Oh! also bad because if it takes us a long time to get through iterating on the mob directory we might exhaust the min age window still.
3. Instead of using now when we start iterating over the mob directory, we use now as of before we start iterating over the hfile directories. Good because again no risk of cluster death if no one notices. Also easier to explain/reason about than the variable look-behind of number 2.

what do you think?

[VladRodionov]

We can record the time needs to build the set of all active MOB files as T and use 2xT as minimum age to archive.

[VladRodionov]

It means we do not need MOB_MINIMUM_FILE_AGE_TO_ARCHIVE_KEY anymore.

[VladRodionov]

This is new algorithm:

      long maxCreationTimeToArchive = EnvironmentEdgeManager.currentTime() - 3600000;
      LOG.info("Only MOB files whose creation time less than {} will be archived", 
        maxCreationTimeToArchive);

The idea is to set maxCreationTimeToArchive as now() -1h. now() - when cleaning chore starts. 1h gap gives us guarantee that all corresponding store files exist at the time we start cleaning. It is hard to imagine, that MOB file and store creation are more that 1h apart.
Then I just check MOB file creation time and if its less than this threshold - it can be archived.