Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: new job to backup pins to s3 #1844

Open
wants to merge 69 commits into
base: main
Choose a base branch
from
Open

feat: new job to backup pins to s3 #1844

Changes from 1 commit
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
ca658d5
feat: added rough draft of the cron job to copy pins into s3
joshghent Aug 22, 2022
cd7a420
chore: add aws-sdk
joshghent Aug 22, 2022
79219bb
feat: add cron
joshghent Aug 22, 2022
2cd96c7
feat: added downloading and uploading of file correctly
joshghent Aug 22, 2022
48cb64d
chore: added required dependencies
joshghent Aug 23, 2022
55367e7
feat: add new backup_urls column to the psa_pin_requests table
joshghent Aug 23, 2022
fdb92d3
feat: finished job to backup files to s3
joshghent Aug 23, 2022
c910b65
feat: added new test suite for pins backup
joshghent Aug 23, 2022
bff06de
feat: add github action to run cron job
joshghent Sep 2, 2022
b34dd12
feat: added cron pins backup script
joshghent Sep 2, 2022
14cbd8c
chore: removed todo
joshghent Sep 2, 2022
ef2600e
feat: added test mocks
joshghent Sep 2, 2022
9e6f035
chore: add mocks for export car
joshghent Sep 6, 2022
ec58dd2
chore: update package-lock, fixed fetch reference
joshghent Sep 6, 2022
0f0cbe5
feat: swapped to a class tructure
joshghent Sep 6, 2022
d009be1
chore: update the job to create a new backup class instance
joshghent Sep 9, 2022
e63a4a1
feat: added more logging, updated query
joshghent Sep 9, 2022
e4e22a0
fix: updated test mocks
joshghent Sep 9, 2022
381ba65
fix: updated to make sql valid
joshghent Sep 9, 2022
9e01bcb
Fix issues with updating rows
flea89 Sep 9, 2022
7271673
chore: remove redundant argument
joshghent Sep 11, 2022
1010da8
chore: update variable names
joshghent Sep 11, 2022
5c0cc8f
chore: update to use npm for the cron
joshghent Sep 11, 2022
4b34fdb
chore: updated type definition to accept a pool not client for pg
joshghent Sep 11, 2022
4b59c7b
chore: remove commented code
joshghent Sep 11, 2022
1826006
feat: added way to not re-upload files we already have
joshghent Sep 11, 2022
e01dda0
chore: updated package lock
joshghent Sep 11, 2022
3c529e5
Make linter happier
flea89 Sep 12, 2022
d417922
feat: `w3 open <cid>` to open cid on w3s.link in browser (#1892)
olizilla Sep 14, 2022
7815d72
feat: put write to cluster behind a flag (#1785)
olizilla Sep 19, 2022
396222f
chore: updated package lock
joshghent Sep 11, 2022
e3a13da
chore: update package lock
joshghent Sep 20, 2022
6fad50f
feat: pass through the query limit on workflow dispatch
joshghent Sep 20, 2022
5c7bca4
feat: updated the backup urls column to default to an empty arr
joshghent Sep 20, 2022
f07c61b
chore: added dagula
joshghent Sep 23, 2022
36e7e08
feat: added default to empty array
joshghent Sep 23, 2022
2d3d56a
feat: added default on the init data
joshghent Sep 23, 2022
86b3de6
feat: updated query to return correct data
joshghent Sep 23, 2022
7ceeaf9
chore: updated return types
joshghent Sep 28, 2022
7368e13
chore: update package lock
joshghent Sep 28, 2022
e127dc0
chore: remove todo comment
joshghent Sep 28, 2022
bf8aa7a
Add fetch to cron pin-backup
flea89 Sep 29, 2022
4abd92f
Update types
flea89 Oct 10, 2022
874830c
Add Minio to cron
flea89 Oct 12, 2022
38b96a4
Add getS3 to cron utils
flea89 Oct 12, 2022
6aa93ef
update tests to be more e2endy
flea89 Oct 12, 2022
15fc4fd
Comment out size limiting
flea89 Oct 12, 2022
9d2fe99
more type fixes
flea89 Oct 12, 2022
ae3b044
Use s3 v3 and updats to backup updates
flea89 Oct 12, 2022
5bc3522
tidy tests up
flea89 Oct 21, 2022
1777df2
Set concurrency to avoid multiple jobs running together
flea89 Oct 21, 2022
dc431fc
Use ipfs node id rather than cluster one
flea89 Oct 21, 2022
f621c31
Create a car file for upload and test it holds the expected content
flea89 Oct 24, 2022
a266c32
Add abort controller
flea89 Oct 24, 2022
b7aa9db
Merge branch 'main' into feat/794-copy-pins-to-eips
flea89 Oct 24, 2022
c9e4dbb
Align api package with main
flea89 Oct 24, 2022
6e1f121
Improve logging and more tests
flea89 Oct 24, 2022
d573121
Improve testing and error handling
flea89 Oct 24, 2022
fd02b6a
Remove only from test
flea89 Oct 24, 2022
395dff7
Update logging
flea89 Oct 25, 2022
c58990b
Update node modules
flea89 Oct 25, 2022
52984d6
Update setup node action
flea89 Oct 25, 2022
694788e
Fix dependecies
flea89 Oct 25, 2022
9dec63f
Increase cron timeout
flea89 Oct 25, 2022
61990b2
Improve logging and remove stale code
flea89 Oct 31, 2022
749e20d
Cache instances of Dagula and use 1 libp2p
flea89 Oct 31, 2022
be24923
Better instanciation of resources
flea89 Oct 31, 2022
7a48df2
Default to quiet logging, but allow for more verbose when running man…
flea89 Oct 31, 2022
8eae8f6
Update cron help text
flea89 Oct 31, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
chore: added required dependencies
  • Loading branch information
@joshghent @flea89
joshghent authored and flea89 committed Oct 12, 2022
commit 48cb64df8b9522ef8832c9c0855dbd865a1864f0
5 changes: 5 additions & 0 deletions packages/cron/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,19 +19,24 @@
"author": "Alan Shaw",
"license": "(Apache-2.0 OR MIT)",
"dependencies": {
"@aws-sdk/lib-storage": "^3.154.0",
"@nftstorage/ipfs-cluster": "^5.0.1",
"@web-std/fetch": "^2.0.1",
"@web3-storage/db": "^4.0.0",
"aws-sdk": "^2.1196.0",
"debug": "^4.3.1",
"dotenv": "^9.0.2",
"format-number": "^3.0.0",
"it-batch": "^1.0.9",
"it-pipe": "^2.0.4",
"limiter": "2.0.1",
"mime-types": "^2.1.35",
"multiformats": "^9.6.2",
"node-fetch": "^2.6.1",
"p-retry": "^4.6.1",
"p-settle": "^5.0.0",
"pg": "^8.7.1",
"stream": "^0.0.2",
"throttled-queue": "^2.1.4"
},
"devDependencies": {
Expand Down