Add a HoodieStorageFactory to provide HoodieStorage instance

[yihua]

Task Description

What needs to be done:

Avoid and disallow using the constrcutor of HoodieStorage implementation classes directly.

Why this task is needed:

Streamline the HoodieStorage instantiation

Task Type

Code improvement/refactoring

Related Issues

Parent feature issue: (if applicable )
Related issues:
NOTE: Use Relationships button to add parent/blocking issues after issue is created.

[yihua]

@CTTY could you add your thoughts here too?

[KiteSoar]

@yihua Hi，May I ask if this task still needs to be continued? If it does, I am a new contributor to Hudi and would like to take over this work.

[yihua]

@KiteSoar Thanks for your interest! Yes, there should be work to be done on this improvement story. @CTTY could you provide more concrete tasks that @KiteSoar can take up?

[CTTY]

Hi @KiteSoar , thanks for jumping on this! We do need some help on this issue. I don't have exact steps in mind right now but here is what we want to achieve:

• Forbid using the constructor of concrete Storage implementation(e.g. HoodieHadoopStorage) to create storage instances across Hudi code base.
• As a replacement, we can add HoodieStorageFactory to help users/developers construct a Storage instance when needed based on StoragePath, StorageConfiguration, and HOODIE_STORAGE_CLASS config.

The motive is to avoid usages like this: https://github.com/CTTY/hudi/blob/0c0c402a9a03061155b5c43f1f4cdd37726eacfd/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala#L236

Why?

• HoodieStorage is supposed to be an interface that allows downstream users to inject customized storage, so they have more control over the IO operations.
• Using HoodieHadoopStorage directly across the code base will cause Hudi to step over downstream customized storage implementations

Please let me know if you want to work on this or have further questions, I'd be available on Slack as well. Thanks again!

[KiteSoar]

@CTTY Thanks for the guidance! I'd like to work on this. I plan to place the HoodieStorageFactory inside the hudi-io module (under org.apache.hudi.storage) , while strictly defaulting to HoodieHadoopStorage for backward compatibility, and I will proceed to refactor the codebase to replace the direct constructor calls with this factory method. How does this plan sound to you?

[CTTY]

I plan to place the HoodieStorageFactory inside the hudi-io module (under org.apache.hudi.storage) , while strictly defaulting to HoodieHadoopStorage for backward compatibility,

StorageFactory should not be put in hudi-io because you need HOODIE_STORAGE_CLASS(in hudi-common) to determine the storage type. HOODIE_STORAGE_CLASS is set to HoodieHadoopStorage by default, so no need to worry about the backward compatibility here

[KiteSoar]

I plan to place the HoodieStorageFactory inside the hudi-io module (under org.apache.hudi.storage) , while strictly defaulting to HoodieHadoopStorage for backward compatibility,

StorageFactory should not be put in hudi-io because you need HOODIE_STORAGE_CLASS(in hudi-common) to determine the storage type. HOODIE_STORAGE_CLASS is set to HoodieHadoopStorage by default, so no need to worry about the backward compatibility here

Ah, good catch! I missed that the config is defined in hudi-common. I also checked HoodieStorageConfig and saw that HOODIE_STORAGE_CLASS already has HoodieHadoopStorage as the default value, so backward compatibility is indeed automatically handled.I'll place the factory in hudi-common then and start working on it.