Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TiFlash directly read/write S3 #6827

Closed
JinheLin opened this issue Feb 15, 2023 · 1 comment · Fixed by #6933, #6967 or #7086
Closed

TiFlash directly read/write S3 #6827

JinheLin opened this issue Feb 15, 2023 · 1 comment · Fixed by #6933, #6967 or #7086
Assignees
@JinheLin
Labels
component/storage type/feature-request Categorizes issue or PR as related to a new feature.

Comments

@JinheLin
Copy link
Contributor

JinheLin commented Feb 15, 2023 •
edited by JaySon-Huang
Loading

New configuration introduced by #6828:

[storage.s3]
endpoint = "127.0.0.1:8080"
bucket = "s3_bucket"

and these two config are read from env

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
@JinheLin JinheLin added type/feature-request Categorizes issue or PR as related to a new feature. component/storage labels Feb 15, 2023
@JinheLin JinheLin self-assigned this Feb 15, 2023
@JinheLin JinheLin mentioned this issue Feb 15, 2023
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Feb 17, 2023
@JinheLin
Add AWS SDK (#6828)
7c819be 
ref #6827
@JinheLin JinheLin mentioned this issue Feb 19, 2023
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Feb 20, 2023
@JinheLin
Update aws-c-common: v0.8.7 -> v0.8.11 (#6847)
f21e06a 
ref #6827
JinheLin added a commit to JinheLin/tiflash-1 that referenced this issue Feb 20, 2023
@JinheLin
Add AWS SDK (pingcap#6828)
80a3dd3 
ref pingcap#6827
This was referenced Feb 20, 2023
Merged
Merged
Merged
ti-chi-bot pushed a commit that referenced this issue Feb 23, 2023
@JaySon-Huang
Add S3 key utils (#6878)
4b83471 
ref #6827
JaySon-Huang added a commit to JaySon-Huang/tiflash that referenced this issue Feb 23, 2023
@JaySon-Huang
Add S3 key utils (pingcap#6878)
8098958 
ref pingcap#6827
@JaySon-Huang
Copy link
Contributor

ref #6882

@JinheLin JinheLin mentioned this issue Feb 24, 2023
Merged
12 tasks
@JaySon-Huang JaySon-Huang mentioned this issue Feb 25, 2023
Merged
17 tasks
ti-chi-bot pushed a commit that referenced this issue Feb 26, 2023
@JinheLin
Add S3 config and utils. (#6892)
6a56277 
ref #6827
@breezewish breezewish mentioned this issue Feb 26, 2023
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Feb 27, 2023
@JaySon-Huang
elect one tiflash node as gc owner (#6863)
a5e2362 
ref #6827
@flowbehappy flowbehappy changed the title TiFlash support S3 TiFlash directly read/write S3 Feb 28, 2023
@JinheLin JinheLin mentioned this issue Feb 28, 2023
Merged
16 tasks
ti-chi-bot pushed a commit that referenced this issue Feb 28, 2023
@breezewish
Add checkpoint utilities (#6900)
9ea9cf7 
ref #6827
ti-chi-bot pushed a commit that referenced this issue Feb 28, 2023
@JaySon-Huang
S3 Lock service (#6874)
dc28555 
ref #6827
@breezewish breezewish mentioned this issue Mar 2, 2023
Merged
12 tasks
@JinheLin JinheLin mentioned this issue Mar 2, 2023
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Mar 2, 2023
@JinheLin
Storage: DMFile supports read/write from S3. (#6913)
0b863a4 
ref #6827
@JaySon-Huang JaySon-Huang mentioned this issue Mar 2, 2023
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Mar 2, 2023
@JinheLin
Support read S3 authentication information from configuration. (#6933)
6a8507c 
close #6827
@JinheLin JinheLin reopened this Mar 2, 2023
This was referenced May 18, 2023
Closed
Merged
ti-chi-bot bot pushed a commit that referenced this issue May 18, 2023
@breezewish
storage: Add AddExtraTableIDColumn action and stream (#7493)
a1cbb63 
ref #6827
@breezewish breezewish mentioned this issue May 18, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue May 19, 2023
@breezewish
storage: Code refactor (#7504)
c1d2b83 
ref #6827
This was referenced May 23, 2023
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue May 25, 2023
@breezewish
storage: Introduce AddExtraTableIDColumnTransformOp (#7545)
2513bd4 
ref #6827
@breezewish breezewish mentioned this issue May 26, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Jun 1, 2023
@breezewish
storage: Refactor disaggregated read flow (#7530)
edf82d2 
ref #6827, close #7576
@ti-chi-bot ti-chi-bot mentioned this issue Jun 1, 2023
Closed
14 tasks
@lidezhu lidezhu mentioned this issue Jun 8, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Jun 9, 2023
@lidezhu
avoid unnecessary fsync in ps (#7616)
5b90c7e 
ref #6827
@lidezhu lidezhu mentioned this issue Jun 16, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Jun 28, 2023
@breezewish
storage: Support pipeline model for disaggregated S3 read (#7552)
2402b48 
ref #6827
@lidezhu lidezhu mentioned this issue Jun 29, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Jun 30, 2023
@lidezhu
improve ps gc speed by dumping snapshot from memory directly (#7668)
834389a 
ref #6827
This was referenced Jul 4, 2023
Merged
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Jul 4, 2023
@ti-chi-bot
avoid unnecessary fsync in ps (#7616) (#7733)
de01024 
ref #6827, close #7736
ti-chi-bot bot pushed a commit that referenced this issue Jul 4, 2023
@ti-chi-bot
Decrease ps write latency (#7154) (#7737)
5b97981 
ref #6827, close #7735
ti-chi-bot bot pushed a commit that referenced this issue Jul 4, 2023
@ti-chi-bot
avoid unnecessary fsync in ps (#7616) (#7734)
6e18968 
ref #6827, close #7736
@lidezhu lidezhu mentioned this issue Jul 6, 2023
Merged
12 tasks
lidezhu pushed a commit to lidezhu/tics that referenced this issue Jul 10, 2023
@ti-chi-bot @lidezhu
avoid unnecessary fsync in ps (pingcap#7616) (pingcap#7734)
66487a8 
ref pingcap#6827, close pingcap#7736
ti-chi-bot bot pushed a commit that referenced this issue Jul 10, 2023
@lidezhu
add config to disable upload page data when do checkpoint (#7753)
641a36b 
ref #6827
ti-chi-bot bot pushed a commit that referenced this issue Jul 17, 2023
@lidezhu
store raft data in dedicated blob file (#7719)
5ca7672 
ref #6827
@lidezhu lidezhu mentioned this issue Jul 19, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Jul 20, 2023
@lidezhu
support fetch raft log num of each region for page-ctl (#7551)
9711f91 
ref #6827
@JinheLin JinheLin closed this as completed Aug 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JinheLin JinheLin

Labels
component/storage type/feature-request Categorizes issue or PR as related to a new feature.
Projects
None yet
Milestone
No milestone
2 participants
@JaySon-Huang @JinheLin