Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Refactor] Refactor GlobalStateMgr #6268

Merged
merged 6 commits into from
May 19, 2022

Conversation

stephen-shelby
Copy link
Contributor

@stephen-shelby stephen-shelby commented May 18, 2022

Signed-off-by: stephen-shelby stephen5217@163.com

What type of PR is this:

  • bug
  • feature
  • enhancement
  • others

Which issues of this PR fixes :

related with: #6272

related with #5062

Problem Summary(Required) :

Goals:

  • Refactoring GlobalStateMgr (the original Fe catalog) is to prepare for building the connector framework.
  • Reduce globalstatemgr redundant code, The original code is reduced from 8000 lines to 3000 lines.

Most of the code in globalstatemgr is transferred to three parts:

  • NodeMgr:
    It is mainly used to manage Fe/Be/Broker, heartbeat and Fe start BDB nodes.
  • LocalMetastore:
    It is mainly responsible for managing metadata related to internal catalog. It includes Cluster/Database/Table/Partition/Tablet/ Replica, and implements the API of ConnectorMetadata
  • XXXMgr in GlobalStateMgr
    Some methods, such as loadxxx and savexxx, transfer them to the corresponding manager instance. For example, transfer loadExportJob() and saveExportJob() in GlobalStateMgr to ExportMgr

@stephen-shelby
Copy link
Contributor Author

run starrocks_fe_unittest

@stephen-shelby stephen-shelby changed the title [WIP] Refactor GlobalStateMgr [Refactor] Refactor GlobalStateMgr May 18, 2022
@stephen-shelby
Copy link
Contributor Author

run starrocks_fe_unittest

Copy link
Contributor

@DorianZheng DorianZheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

result.add(frontend);
}
}
private LocalMetastore localMetastore;
Copy link
Contributor

@caneGuy caneGuy May 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: how about moving this into metadatamgr?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This global instance is used in this stage, and the usage method will be uniformly rectified later

@gengjun-git gengjun-git merged commit a502fb5 into StarRocks:main May 19, 2022
@stephen-shelby stephen-shelby deleted the Refactor-GlobalStateMgr branch May 19, 2022 16:14
@wanpengfei-git
Copy link
Collaborator

[FE PR Coverage check]

😞 fail : 41 / 310 (13.23%)

file detail

path covered line new line coverage
🔵 com/starrocks/load/routineload/RoutineLoadManager.java 0 6 00.00%
🔵 com/starrocks/load/ExportMgr.java 0 23 00.00%
🔵 com/starrocks/load/loadv2/LoadManager.java 0 6 00.00%
🔵 com/starrocks/mysql/privilege/Auth.java 0 6 00.00%
🔵 com/starrocks/transaction/GlobalTransactionMgr.java 0 12 00.00%
🔵 com/starrocks/backup/BackupHandler.java 0 7 00.00%
🔵 com/starrocks/catalog/ColocateTableIndex.java 0 65 00.00%
🔵 com/starrocks/common/util/SmallFileMgr.java 0 6 00.00%
🔵 com/starrocks/catalog/ResourceMgr.java 0 2 00.00%
🔵 com/starrocks/load/Load.java 0 29 00.00%
🔵 com/starrocks/connector/ConnectorMetadata.java 0 1 00.00%
🔵 com/starrocks/statistic/AnalyzeManager.java 0 8 00.00%
🔵 com/starrocks/qe/VariableMgr.java 0 6 00.00%
🔵 com/starrocks/load/DeleteHandler.java 0 2 00.00%
🔵 com/starrocks/plugin/PluginMgr.java 0 6 00.00%
🔵 com/starrocks/catalog/CatalogRecycleBin.java 1 13 07.69%
🔵 com/starrocks/catalog/WorkGroupMgr.java 1 9 11.11%
🔵 com/starrocks/server/NodeMgr.java 12 40 30.00%
🔵 com/starrocks/server/LocalMetastore.java 27 63 42.86%

abc982627271 pushed a commit to abc982627271/starrocks that referenced this pull request Jun 22, 2022
Goals:
- Refactoring GlobalStateMgr (the original Fe catalog) is to prepare for building the connector framework.
- Reduce globalstatemgr redundant code, The original code is reduced from 8000 lines to 3000 lines.

Most of the code in globalstatemgr is transferred to three parts:
- NodeMgr: 
   It is mainly used to manage Fe/Be/Broker, heartbeat and Fe start BDB nodes.
- LocalMetastore: 
   It is mainly responsible for managing metadata related to internal catalog. It includes Cluster/Database/Table/Partition/Tablet/ Replica, and implements the API of `ConnectorMetadata`
- XXXMgr in GlobalStateMgr
   Some methods, such as loadxxx and savexxx, transfer them to the corresponding manager instance.  For example, transfer `loadExportJob()` and `saveExportJob()` in `GlobalStateMgr` to `ExportMgr`
jaogoy pushed a commit to jaogoy/starrocks that referenced this pull request Nov 15, 2023
…tarRocks#6315)

* Add doc for covar/corr etc funtion
Signed-off-by: before-Sunrise <unclejyj@gmail.com>
---------

Signed-off-by: before-Sunrise <unclejyj@gmail.com>
Co-authored-by: evelyn.zhaojie <everlyn.zhaojie@gmail.com>
(cherry picked from commit c3f1994)

Co-authored-by: before-Sunrise <71162020+before-Sunrise@users.noreply.github.com>
(cherry picked from commit e0d425c)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
jaogoy pushed a commit to jaogoy/starrocks that referenced this pull request Nov 15, 2023
jaogoy pushed a commit to jaogoy/starrocks that referenced this pull request Nov 15, 2023
* Add doc for covar/corr etc funtion
Signed-off-by: before-Sunrise <unclejyj@gmail.com>
---------

Signed-off-by: before-Sunrise <unclejyj@gmail.com>
Co-authored-by: evelyn.zhaojie <everlyn.zhaojie@gmail.com>
(cherry picked from commit c3f1994)

Co-authored-by: before-Sunrise <71162020+before-Sunrise@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants