Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backend]Cache - Cache logic with db interaction #3266

Merged
merged 69 commits into from
Mar 20, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
6994984
Initial execution cache
rui5i Feb 10, 2020
19a61e7
Add initial server logic
rui5i Feb 10, 2020
0448a12
Add const
rui5i Feb 10, 2020
d2fad48
Change folder name
rui5i Feb 10, 2020
c561ca4
Change execution key name
rui5i Feb 11, 2020
55ee8a7
Fix unit test
rui5i Feb 11, 2020
0f83144
Add Dockerfile and OWNERS file
rui5i Feb 14, 2020
1ecc855
fix go.sum
rui5i Feb 15, 2020
1b5b681
Add local deployment scripts
rui5i Feb 18, 2020
83126c7
Merge branch 'execution_cache' of https://github.com/rui5i/pipelines …
rui5i Feb 18, 2020
ae472d7
refactor src code
rui5i Feb 19, 2020
8c0cac8
Add standalone deployment scripts and yamls
rui5i Feb 20, 2020
cb075b2
Minor fix
rui5i Feb 20, 2020
4b0831a
Add execution cache image build in test folder
rui5i Feb 20, 2020
e1bedf7
Merge branch 'master' into execution_cache
rui5i Feb 20, 2020
ed1aa0e
fix test cloudbuild
rui5i Feb 20, 2020
3b1323c
Fix cloudbuild
rui5i Feb 20, 2020
da3eb42
Add execution cache deployer image to test folder
rui5i Feb 20, 2020
37521ec
Add copyright
rui5i Feb 20, 2020
5efe589
Fix deployer build
rui5i Feb 20, 2020
fa12c40
Add license for execution cache and cloudbuild for execution cache
rui5i Feb 21, 2020
3c302af
Refactor license intermediate data
rui5i Feb 21, 2020
17f714e
Fix execution cache image manifest
rui5i Feb 21, 2020
79e4711
Typo fix for cache and cache deployer images
rui5i Feb 21, 2020
a16335c
Add arguments in ca generation scripts and change deployer base image…
rui5i Feb 22, 2020
bb796c1
minor fix
rui5i Feb 22, 2020
db797b3
fix arg
rui5i Feb 22, 2020
57523c0
Mirror source code with MPL in execution_cache image
rui5i Feb 24, 2020
893e8b1
Minor fix
rui5i Feb 24, 2020
851ce25
minor refactor on error handling
rui5i Feb 24, 2020
1720ee6
Refactor cache source code, Docker image and manifest
rui5i Feb 25, 2020
f482c9f
Fix variable names
rui5i Feb 26, 2020
6523570
Add images in .release.cloudbuild.yaml
rui5i Feb 26, 2020
5c5298e
resolve merge conflict
rui5i Feb 27, 2020
914d9fd
Change execution_cache to generic name
rui5i Feb 27, 2020
4e7e3a9
revice readme
rui5i Feb 27, 2020
ad560f4
Move deployer job out of upgrade script
rui5i Feb 28, 2020
ba26e48
fix tests
rui5i Feb 28, 2020
065af7a
fix tests
rui5i Feb 28, 2020
a8aa15c
Seperate cache service and cache deployer job
rui5i Feb 28, 2020
e7aefad
mysql set up
rui5i Feb 28, 2020
d3e2755
wip
rui5i Mar 2, 2020
e986ca1
WIP
rui5i Mar 2, 2020
2df1c48
WIP
rui5i Mar 3, 2020
16b3e6d
work mysql connection
rui5i Mar 5, 2020
243307a
initial cache logic
rui5i Mar 5, 2020
da75b68
watcher
rui5i Mar 6, 2020
d79e909
WIP pod watching with mysql
rui5i Mar 9, 2020
980f089
worked crud
rui5i Mar 12, 2020
c0cf37e
Add sql unit test
rui5i Mar 13, 2020
616e968
Merge upstream master
rui5i Mar 13, 2020
25f0bcd
fix manifest
rui5i Mar 13, 2020
ca41d77
Add copyright
rui5i Mar 13, 2020
2b39036
Add watcher check and update cache key generation logic
rui5i Mar 13, 2020
28c77c0
test replace container images
rui5i Mar 16, 2020
8f987d9
work cache service
rui5i Mar 16, 2020
7e19564
Add configmap for cache service
rui5i Mar 16, 2020
f82b781
refactor
rui5i Mar 17, 2020
db9bc4e
fix manifest
rui5i Mar 18, 2020
4c49458
Add unit tests
rui5i Mar 18, 2020
c4ad659
Remove delete table
rui5i Mar 18, 2020
26c790d
Fix sql dialect
rui5i Mar 18, 2020
f702714
Add cached step log
rui5i Mar 18, 2020
14d05c0
Add metadata execution id
rui5i Mar 18, 2020
db1073b
minor fix
rui5i Mar 19, 2020
154d26e
revert go.mod and go.sum
rui5i Mar 19, 2020
901a7d2
revert go.sum and go.mod
rui5i Mar 19, 2020
c2156e6
revert go.sum and go.mod
rui5i Mar 19, 2020
075d255
revert go.mod and go.sum
rui5i Mar 19, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add execution cache image build in test folder
  • Loading branch information
rui5i committed Feb 20, 2020
commit 4b0831ab1b013b97f3e0f5454268bae452e206a3
2 changes: 2 additions & 0 deletions manifests/kustomize/base/kustomization.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,3 +28,5 @@ images:
newTag: 0.2.1
- name: gcr.io/ml-pipeline/metadata-writer
newTag: 0.2.1
- name: gcr.io/ml-pipeline-test/execution-cache
newTag: 0.2.1
1 change: 1 addition & 0 deletions test/build-images.sh
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ if
echo "$BUILT_IMAGES" | grep viewer-crd-controller && \
echo "$BUILT_IMAGES" | grep inverse-proxy-agent && \
echo "$BUILT_IMAGES" | grep metadata-writer && \
echo "$BUILT_IMAGES" | grep execution-cache && \
echo "$BUILT_IMAGES" | grep visualization-server;
then
echo "docker images for api-server, frontend, scheduledworkflow, \
Expand Down
5 changes: 5 additions & 0 deletions test/cloudbuild/batch_build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@ steps:
name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', '$_GCR_BASE/metadata-writer', '-f', 'backend/metadata_writer/Dockerfile', '.']
waitFor: ["-"]
- id: 'buildExecutionCache'
name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', '$_GCR_BASE/execution-cache', '-f', 'backend/execution_cache/Dockerfile', '.']
waitFor: ["-"]
options:
machineType: N1_HIGHCPU_8 # use a fast machine to build because there a lot of work
images:
Expand All @@ -43,4 +47,5 @@ images:
- "$_GCR_BASE/visualization-server"
- "$_GCR_BASE/inverse-proxy-agent"
- "$_GCR_BASE/metadata-writer"
- "$_GCR_BASE/execution-cache"
timeout: 1800s # 30min
1 change: 1 addition & 0 deletions test/deploy-pipeline-lite.sh
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ kustomize edit set image gcr.io/ml-pipeline/viewer-crd-controller=${GCR_IMAGE_BA
kustomize edit set image gcr.io/ml-pipeline/visualization-server=${GCR_IMAGE_BASE_DIR}/visualization-server:${GCR_IMAGE_TAG}
kustomize edit set image gcr.io/ml-pipeline/inverse-proxy-agent=${GCR_IMAGE_BASE_DIR}/inverse-proxy-agent:${GCR_IMAGE_TAG}
kustomize edit set image gcr.io/ml-pipeline/metadata-writer=${GCR_IMAGE_BASE_DIR}/metadata-writer:${GCR_IMAGE_TAG}
kustomize edit set image gcr.io/ml-pipeline-test/execution-cache=${GCR_IMAGE_BASE_DIR}/execution-cache:${GCR_IMAGE_TAG}
cat kustomization.yaml

kustomize build . | kubectl apply -f -
Expand Down