projections + compaction + catalog + msq #17803
Open
+2,805
−1,287
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
changes: * `CompactionTask` now accepts a `projections` property which will cause classic and MSQ auto-compaction to build segments with projections * `DataSourceCompactionConfig` has been turned into an interface, with the existing implementation renamed to `InlineSchemaDataSourceCompactionConfig` * Added projections list to `InlineSchemaDataSourceCompactionConfig` to allow explicitly defining projections in an inline schema compaction spec * if not explicitly defined, compaction tasks will now preserve existing projections when processing segments, combining all named projections across the segments being processed - different projections with the same name are not checked for equivalence, rather one will be chosen dependent on segment processing order. * Added ability to define projections as a property of a datasource in the catalog * If projections are defined in a catalog, they will be automatically used by MSQ insert and replace queries * Added new experimental `CatalogDataSourceCompactionConfig` which allows populating much of a `CompactionTask` using information stored in the catalog. Currently this has some feature gaps compared to `InlineSchemaDataSourceCompactionConfig`, but will be improved in follow-up work to eventually become much more powerful than what can be expressed via a `InlineSchemaDataSourceCompactionConfig` * Moved `MetadataCatalog` to druid-server from the catalog extension * Added method to get `MetadataCatalog` from `CatalogResolver` * Added `CatalogCoreModule` to provide a null binding for `MetadataCatalog`, overridden if the catalog extension is loaded * Overlord added as a watcher for catalog like the Broker so that it can have `CatalogResolver` and `MetadataCatalog` available * Added binding for `MetadataCatalog` to Coordinator to have `MetadataCatalg` available
github-actions
bot
added
Area - Batch Ingestion
Area - Querying
Area - Segment Format and Ser/De
Area - Dependencies
Area - Ingestion
Area - MSQ
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
extensions-core/multi-stage-query/src/test/java/org/apache/druid/msq/exec/MSQInsertTest.java
Dismissed
Show dismissed
Hide dismissed
server/src/test/java/org/apache/druid/client/indexing/ClientCompactionRunnerInfoTest.java
Dismissed
Show dismissed
Hide dismissed
server/src/test/java/org/apache/druid/server/compaction/CompactionStatusTest.java
Dismissed
Show dismissed
Hide dismissed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
changes:
CompactionTasknow accepts aprojectionsproperty which will cause classic and MSQ auto-compaction to build segments with projectionsDataSourceCompactionConfighas been turned into an interface, with the existing implementation renamed toInlineSchemaDataSourceCompactionConfigInlineSchemaDataSourceCompactionConfigto allow explicitly defining projections in an inline schema compaction specCatalogDataSourceCompactionConfigwhich allows populating much of aCompactionTaskusing information stored in the catalog. Currently this has some feature gaps compared toInlineSchemaDataSourceCompactionConfig, but will be improved in follow-up work to eventually become much more powerful than what can be expressed via aInlineSchemaDataSourceCompactionConfigMetadataCatalogto druid-server from the catalog extensionMetadataCatalogfromCatalogResolverCatalogCoreModuleto provide a null binding forMetadataCatalog, overridden if the catalog extension is loadedCatalogResolverandMetadataCatalogavailableMetadataCatalogto Coordinator to haveMetadataCatalgavailableRelease note
todo
This PR has: