Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

InfoSchema V2 #50959

Open
10 of 14 tasks
tiancaiamao opened this issue Feb 4, 2024 · 0 comments
Open
10 of 14 tasks

InfoSchema V2 #50959

tiancaiamao opened this issue Feb 4, 2024 · 0 comments
Labels
type/enhancement The issue or PR belongs to an enhancement.

Comments

@tiancaiamao
Copy link
Contributor

tiancaiamao commented Feb 4, 2024 •
edited
Loading

Enhancement

When we have millions of tables, the memory footprint for meta data is not negligible.
Some users use TiDB to provide services to their tenant, so there could be a huge amount of tables.
We need to develop a kind of caching mechanism to ensure that even in the extreme cases, the meta data do not cause TiDB OOM.

In the new design, the table.Table will not be persisted in memory, insted, they will be cached.

Development tasks
@tiancaiamao tiancaiamao added the type/enhancement The issue or PR belongs to an enhancement. label Feb 4, 2024
@tiancaiamao tiancaiamao mentioned this issue Feb 4, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Feb 4, 2024
@tiancaiamao
infoschema: remove some redundant methods from InfoSchema (#50916)
c6d7270 
ref #50959
This was referenced Feb 5, 2024
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Feb 7, 2024
@tiancaiamao
*: store db ID in model.TableInfo and move SchemaByTable method out f…
f5e0799 
…rom InfoSchema (#50917)

ref #50959
ti-chi-bot bot pushed a commit that referenced this issue Feb 7, 2024
@tiancaiamao
infoschema: split InfoSchemaMisc from InfoSchema interface (#51015)
f46bded 
ref #50959
@tiancaiamao tiancaiamao mentioned this issue Feb 7, 2024
Merged
13 tasks
@tiancaiamao tiancaiamao mentioned this issue Feb 22, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Feb 23, 2024
@tiancaiamao
infoschema,domain: introduce InfoSchemaV2, core data struct etc (#51058)
e06dc99 
ref #50959
@ywqzzy ywqzzy mentioned this issue Feb 28, 2024
Merged
13 tasks
@GMHDBJD GMHDBJD mentioned this issue Feb 28, 2024
Open
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Feb 28, 2024
@ywqzzy
infoschema: run some infoschema ut without domain (#51375)
aae2a36 
ref #50959
@ywqzzy ywqzzy mentioned this issue Feb 28, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Feb 29, 2024
@ywqzzy
infoschema/test: test apply diff (#51398)
353b75d 
ref #50959
@ywqzzy ywqzzy mentioned this issue Feb 29, 2024
Merged
13 tasks
@tiancaiamao tiancaiamao mentioned this issue Feb 29, 2024
Merged
13 tasks
@wjhuang2016 wjhuang2016 mentioned this issue Feb 29, 2024
Merged
13 tasks
@ywqzzy ywqzzy mentioned this issue Mar 1, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Mar 1, 2024
@ywqzzy
infoschema: refine common part for builder (#51411)
da7cd99 
ref #50959
ti-chi-bot bot pushed a commit that referenced this issue Mar 1, 2024
@tiancaiamao
infoschema: fix some InfoSchema V2 API for special schemas like INFOR…
bd17c5b 
…MATION_SCHEMA (#51434)

ref #50959
This was referenced Mar 4, 2024
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Mar 4, 2024
@ywqzzy
infoschema: more tests for applyDiff (#51463)
d440d70 
ref #50959
@ywqzzy ywqzzy mentioned this issue Mar 5, 2024
Merged
13 tasks
@tiancaiamao tiancaiamao mentioned this issue Mar 5, 2024
Merged
13 tasks
@ywqzzy ywqzzy mentioned this issue Mar 5, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Mar 6, 2024
@ywqzzy
infoschemaV2: tiny refine builder (#51522)
b2c3e67 
ref #50959
ti-chi-bot bot pushed a commit that referenced this issue Mar 6, 2024
@ywqzzy
infoschemaV2: support create&drop&update table/schema for v2 (#51424)
2830714 
ref #50959
ti-chi-bot bot pushed a commit that referenced this issue Mar 6, 2024
@ywqzzy
infoschemaV2: add ut for v2 and some refine (#51507)
46eafe4 
ref #50959
ti-chi-bot bot pushed a commit that referenced this issue Jul 23, 2024
@tiancaiamao
*: remove infoschema.SchemaTables() API, replace it with SchemaTableI…
a18b3c5 
…nfos() (#54664)

ref #50959
ti-chi-bot bot pushed a commit that referenced this issue Jul 24, 2024
@wjhuang2016
*: add table lock, foreign key to special attributes (#54672)
1cee700 
ref #50959
@tiancaiamao tiancaiamao mentioned this issue Jul 24, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Jul 29, 2024
@tiancaiamao
meta,infoschema: fix a bug that after HasTemporaryTable() wrong infos…
247b06e 
…chema v2 full load (#54879)

ref #50959
This was referenced Jul 31, 2024
Merged
Merged
hawkingrei pushed a commit to hawkingrei/tidb that referenced this issue Aug 1, 2024
@tiancaiamao @hawkingrei
meta,infoschema: fix a bug that after HasTemporaryTable() wrong infos…
37a9260 
…chema v2 full load (pingcap#54879)

ref pingcap#50959
@tiancaiamao tiancaiamao mentioned this issue Aug 1, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Aug 6, 2024
@tiancaiamao
infoschema: rewrite SchemaSimpleTableInfos for infoschema v2 to avoid…
1fb4019 
… network (#55088)

ref #50959
@tiancaiamao tiancaiamao mentioned this issue Aug 6, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Aug 6, 2024
@tiancaiamao
infoschema,executor: change TableByID to skip refill infoschema v2 ca…
3b19de6 
…che (#55101)

ref #50959
ti-chi-bot bot pushed a commit that referenced this issue Aug 6, 2024
@tiancaiamao
infoschema: support cached table for infoschema v2 (#55139)
977a5bf 
ref #50959
@tiancaiamao tiancaiamao mentioned this issue Aug 6, 2024
Merged
13 tasks
@lance6716 lance6716 mentioned this issue Aug 9, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Aug 9, 2024
@lance6716
*: add ctx parameter to infoschema TableByID (#55329)
876268f 
ref #50959
@tiancaiamao tiancaiamao mentioned this issue Aug 12, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Aug 12, 2024
@tiancaiamao
infoschema: update two tests to make them robust and a tiny fix (#55362)
4baaa18 
ref #50959, close #55132
ti-chi-bot bot pushed a commit that referenced this issue Aug 13, 2024
@tiancaiamao
*: update v2 infoschema reader support for tables Indexes, Views, Key…
bc6a18d 
…ColumnUsage, TableConstraints (#55241)

ref #50959
ti-chi-bot bot pushed a commit that referenced this issue Aug 14, 2024
@tiancaiamao
*: enable infoschema v2 by default (#55229)
70381ea 
ref #50959
@tiancaiamao tiancaiamao mentioned this issue Aug 19, 2024
Merged
13 tasks
@joechenrh joechenrh mentioned this issue Aug 20, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Aug 20, 2024
@joechenrh
domain: change groupSize in splitForConcurrentFetch (#55518)
85235b3 
ref #50959
ti-chi-bot bot pushed a commit that referenced this issue Aug 20, 2024
@tiancaiamao
infoschema: add WithRefillOption for TableByName and TableByID (#55511)
49828bd 
ref #50959
@tiancaiamao tiancaiamao mentioned this issue Aug 21, 2024
Merged
13 tasks
This was referenced Aug 22, 2024
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Aug 28, 2024
@fzzf678
infoschema_v2: add ut test Data struct fields' correctness (#55709)
74404a4 
ref #50959
@fzzf678 fzzf678 mentioned this issue Sep 2, 2024
Merged
13 tasks
@tiancaiamao tiancaiamao mentioned this issue Sep 25, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 26, 2024
@tiancaiamao
sessionctx,infoschema: change @@tidb_schema_cache_size lower bound to…
d788e3c 
… 64MB (#56316)

ref #50959
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tiancaiamao