[Feature] Support information schema for external catalog

[letian-jiang]

This PR does the following

• Add information schema connector for external catalogs
• Introduce CatalogConnector to aggregate normal connector and information schema connector with a single catalog

What type of PR is this:

• BugFix
• Feature
• Enhancement
• Refactor
• UT
• Doc
• Tool

Checklist:

• I have added test cases for my bug fix or my new feature
• This pr will affect users' behaviors
• This pr needs user documentation (for new or modified features or behaviors)
  • I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

• I have checked the version labels which the pr will be auto-backported to the target branch
  • 3.1
  • 3.0
  • 2.5
  • 2.4

[DorianZheng]

We shouldn't let every connector to know about information schema. We can implement it transparently. Let's talk offline.

[dirtysalt]

a few questions about design:

• why two connector metadata: normal connector metadata and information connector metadata? What's the difference between them?
• What's difference between ConnectorService and Connector?

[dirtysalt]

The code is well written. I have no questions about the code.

[letian-jiang]

a few questions about design:

• why two connector metadata: normal connector metadata and information connector metadata? What's the difference between them?
• What's difference between ConnectorService and Connector?

Q1

• normal metadata provides information about normal tables (e.g. native iceberg/hudi table)
• infoschema metadata provides information about catalog metadata (e.g. columns table)

Q2

• ConnectorService is an intermediate layer between MetadataMgr and user-defined Connector.
• Thus, common logic (information schema and query-level metadata cache in our case) across all connectors can be put in ConnectorService, not MetadataMgr or Connector.

[stephen-shelby]

Please give the clear definition of the ConnectorService.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
31 Code Smells

0.0% Coverage
0.0% Duplication

The version of Java (11.0.20) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more here

[wanpengfei-git]

[FE PR Coverage Check]

😍 pass : 246 / 290 (84.83%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/planner/HudiScanNode.java	0	1	00.00%	[79]
🔵	com/starrocks/planner/PaimonScanNode.java	0	1	00.00%	[87]
🔵	com/starrocks/connector/hive/HiveMetastoreApiConverter.java	0	1	00.00%	[223]
🔵	com/starrocks/catalog/ScalarType.java	0	3	00.00%	[556, 557, 559]
🔵	com/starrocks/planner/DeltaLakeScanNode.java	0	1	00.00%	[95]
🔵	com/starrocks/service/FrontendServiceImpl.java	17	30	56.67%	[328, 401, 402, 406, 412, 413, 1010, 1011, 1012, 1015, 1016, 1021, 1031]
🔵	com/starrocks/planner/SchemaScanNode.java	4	6	66.67%	[166, 343]
🔵	com/starrocks/service/InformationSchemaDataSource.java	16	21	76.19%	[89, 310, 326, 327, 331]
🔵	com/starrocks/connector/ConnectorMgr.java	4	5	80.00%	[91]
🔵	com/starrocks/connector/informationschema/InformationSchemaMetadata.java	12	14	85.71%	[56, 69]
🔵	com/starrocks/server/MetadataMgr.java	25	29	86.21%	[72, 73, 126, 135]
🔵	com/starrocks/connector/CatalogConnectorMetadata.java	67	74	90.54%	[104, 158, 159, 189, 190, 194, 195]
🔵	com/starrocks/catalog/system/information/InfoSchemaDb.java	47	50	94.00%	[122, 123, 125]
🔵	com/starrocks/planner/IcebergTableSink.java	1	1	100.00%	[]
🔵	com/starrocks/connector/ConnectorFactory.java	3	3	100.00%	[]
🔵	com/starrocks/catalog/system/information/SchemataSystemTable.java	1	1	100.00%	[]
🔵	com/starrocks/catalog/system/information/EnginesSystemTable.java	1	1	100.00%	[]
🔵	com/starrocks/catalog/system/information/ReferentialConstraintsSystemTable.java	1	1	100.00%	[]
🔵	com/starrocks/catalog/system/information/ColumnPrivilegesSystemTable.java	1	1	100.00%	[]
🔵	com/starrocks/catalog/system/information/CollationsSystemTable.java	1	1	100.00%	[]
🔵	com/starrocks/sql/plan/PlanFragmentBuilder.java	2	2	100.00%	[]
🔵	com/starrocks/catalog/system/information/SchemaPrivilegesSystemTable.java	1	1	100.00%	[]
🔵	com/starrocks/catalog/system/information/KeyColumnUsageSystemTable.java	1	1	100.00%	[]
🔵	com/starrocks/catalog/system/information/TablePrivilegesSystemTable.java	1	1	100.00%	[]
🔵	com/starrocks/server/CatalogMgr.java	3	3	100.00%	[]
🔵	com/starrocks/catalog/system/information/EventsSystemTable.java	1	1	100.00%	[]
🔵	com/starrocks/catalog/system/information/ColumnsSystemTable.java	1	1	100.00%	[]
🔵	com/starrocks/catalog/system/information/TableConstraintsSystemTable.java	1	1	100.00%	[]
🔵	com/starrocks/planner/IcebergScanNode.java	1	1	100.00%	[]
🔵	com/starrocks/catalog/system/information/TriggersSystemTable.java	1	1	100.00%	[]
🔵	com/starrocks/connector/CatalogConnector.java	13	13	100.00%	[]
🔵	com/starrocks/catalog/system/information/ViewsSystemTable.java	1	1	100.00%	[]
🔵	com/starrocks/connector/informationschema/InformationSchemaConnector.java	5	5	100.00%	[]
🔵	com/starrocks/catalog/system/information/UserPrivilegesSystemTable.java	1	1	100.00%	[]
🔵	com/starrocks/catalog/system/information/CharacterSetsSystemTable.java	1	1	100.00%	[]
🔵	com/starrocks/catalog/system/information/RoutinesSystemTable.java	1	1	100.00%	[]
🔵	com/starrocks/catalog/system/information/TablesSystemTable.java	1	1	100.00%	[]
🔵	com/starrocks/sql/optimizer/transformer/SqlToScalarOperatorTranslator.java	3	3	100.00%	[]
🔵	com/starrocks/planner/HdfsScanNode.java	1	1	100.00%	[]
🔵	com/starrocks/catalog/system/information/StatisticsSystemTable.java	1	1	100.00%	[]
🔵	com/starrocks/catalog/system/SystemTable.java	4	4	100.00%	[]