Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix mem usage #1337

Merged
merged 5 commits into from
Dec 2, 2023
Merged

Fix mem usage #1337

merged 5 commits into from
Dec 2, 2023

Conversation

Hitenjain14
Copy link
Member

Changes

Fixes

Tests

Tasks to complete before merging PR:

  • Ensure system tests are passing. If not Run them manually to check for any regressions 📋
  • Do any new system tests need added to test this change? do any existing system tests need updated? If so create a PR at 0chain/system_test
  • Merge your system tests PR to master AFTER merging this PR

Associated PRs (Link as appropriate):

  • 0chain:
  • gosdk:
  • system_test:
  • zboxcli:
  • zwalletcli:
  • Other: ...

@codecov-commenter
Copy link

codecov-commenter commented Dec 1, 2023 •
edited
Loading

Codecov Report

Attention: 7 lines in your changes are missing coverage. Please review.

Comparison is base (c72a891) 23.71% compared to head (c6914f5) 23.71%.

Files Patch % Lines
.../go/0chain.net/blobbercore/handler/client_quota.go 0.00% 4 Missing ⚠️
code/go/0chain.net/core/encryption/hash.go 0.00% 2 Missing ⚠️
code/go/0chain.net/validatorcore/storage/models.go 0.00% 1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files 
@@               Coverage Diff               @@
##           sprint-1.11    #1337      +/-   ##
===============================================
- Coverage        23.71%   23.71%   -0.01%     
===============================================
  Files               87       87              
  Lines            10058    10059       +1     
===============================================
  Hits              2385     2385              
- Misses            7378     7379       +1     
  Partials           295      295
Flag Coverage Δ
Unit-Tests 23.71% <22.22%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Hitenjain14 Hitenjain14 mentioned this pull request Dec 1, 2023
Merged
3 tasks
@Hitenjain14 Hitenjain14 marked this pull request as ready for review December 1, 2023 22:53
dabasov
dabasov approved these changes Dec 1, 2023
View reviewed changes
Copy link
Member

@dabasov dabasov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@dabasov dabasov merged commit 6d85230 into sprint-1.11 Dec 2, 2023
5 of 7 checks passed
@dabasov dabasov deleted the fix/mem-usage branch December 2, 2023 12:26
@shahnawaz-creator shahnawaz-creator restored the fix/mem-usage branch December 2, 2023 18:24
@shahnawaz-creator shahnawaz-creator deleted the fix/mem-usage branch December 2, 2023 18:53
dabasov added a commit that referenced this pull request Dec 19, 2023
@dabasov @Jayashsatolia403
@Hitenjain14 @Kishan-Dhakan @lpoli @devyetii @AmrAshraf @shahnawaz-creator @tapishsinha-rs @din-mukhammed @boddumanohar @YarikRevich @Sanchit011
Sprint 1.11 (#1279)
fdde95b 
* Debug (#1232)

* Revert "Debug (#1232)" (#1240)

This reverts commit 49a5c9e.

* Fix phase locking (#1230) (#1241)

* add logs for 2 phase lock

* add update log

* add root in log

* add update log

* use update lock

* add log in update repo

* use save

* use exec

* log rows affected

* use no key update

* use repo update

* cleanup

* add defer

* add defer

* increase timeout by 5 mins

* Case 3 and case 4 (#1198)

* Send WM commit status to conductor

* Rename file

* Modify log type

* Add state for stopping blobber from committing WM

* Send file meta root to rpc server

* Export function to use it in conductor client

* Modify field type

* Add code to get file meta root

* Modify file meta root retrieval

* Add field

* Apply rename for conductor test based on conductor state

* Add comment

* Log error

---------

Co-authored-by: Ebrahim Gomaa <ebrahim.gomaa.hg@gmail.com>

* Feature/ Challenge based on rounds (#1226)

* Fix

* Updated gosdk

* Fix

* Fix

* Fix

* Logging

* Debug

* Fix

* Fix

* Debug

* Debug

* Debug

* Fix unit tests

* Logigng

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Fix

* Fix

* Updated gosdk

* Fix

* Fix

* log config too

* Fix phase locking (#1230)

* add logs for 2 phase lock

* add update log

* add root in log

* add update log

* use update lock

* add log in update repo

* use save

* use exec

* log rows affected

* use no key update

* use repo update

* cleanup

* add defer

* add defer

* Fix

* Added logging

* Debug

* Debug

* Added reference to logs

* Fix

* Fix

* Updated gosdk

* Use slice instead of ll

* Validator last 5 transactions

* Fix

* Fix

* Updated gosdk

* Resolved comments

* Updated gosdk

---------

Co-authored-by: Hitenjain14 <57557631+Hitenjain14@users.noreply.github.com>

* Increase limit on number of open challenges per fetching (#1249)

* Fix

* Updated gosdk

* Fix

* Fix

* Fix

* Logging

* Debug

* Fix

* Fix

* Debug

* Debug

* Debug

* Fix unit tests

* Logigng

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Fix

* Fix

* Updated gosdk

* Fix

* Fix

* log config too

* Fix phase locking (#1230)

* add logs for 2 phase lock

* add update log

* add root in log

* add update log

* use update lock

* add log in update repo

* use save

* use exec

* log rows affected

* use no key update

* use repo update

* cleanup

* add defer

* add defer

* Fix

* Added logging

* Debug

* Debug

* Added reference to logs

* Fix

* Fix

* Updated gosdk

* Use slice instead of ll

* Validator last 5 transactions

* Fix

* Fix

* Updated gosdk

* Resolved comments

* Updated gosdk

* Fix

* Increase time limit

* Increase time limit

* Fix

* Fix

* Fix

* Fix

* Debug

* Debug

* Debug

* Fix

* Fix

* Fix

* Cleanup logging

---------

Co-authored-by: Hitenjain14 <57557631+Hitenjain14@users.noreply.github.com>

* Refactor commit (#1239)

* add szwg to commit

* lock for query

* aggregate queries

* check len

* rmv debug

* only update file ref

* move update fields to ref

* fix numBlock download stats

* rmv log

* rmv commented collector code

* Error on renaming dir (#1250)

* throw err when renaming dir

* fix unit test

* Fix replace blobber (#1251)

* Async processing (#1225)

* fix pre download

* rmv hash

* add async processing

* use range

* fix unit test

* fix delete change

* add logs

* fix unit test

* fix delete cmd

* save file ref

* add conn timing logs

* set default to 32MB

* fix timing log

* add inner lock

* fix test

* add ctx to cancel go routine

* parallel write to file

* fix connection

* revert storage changes

* empty commit

* Fix phase locking (#1230)

* add logs for 2 phase lock

* add update log

* add root in log

* add update log

* use update lock

* add log in update repo

* use save

* use exec

* log rows affected

* use no key update

* use repo update

* cleanup

* add defer

* add defer

* rmv allocObj from connObj

* Download block limit (#1254)

* add limit as 500 to numBlocks

* add daily and req limit

* fix test

* fix test

* Fix download stats (#1256)

* fix download stats

* fix typo

* Fix empty alloc cleanup (#1252)

* Fix replace blobber

* Fix empty alloc cleanup

* Debug Rounds fetch (#1255)

* Fix

* Updated gosdk

* Fix

* Fix

* Fix

* Logging

* Debug

* Fix

* Fix

* Debug

* Debug

* Debug

* Fix unit tests

* Logigng

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Updated gosdk

* Fix

* Fix

* Updated gosdk

* Fix

* Fix

* log config too

* Fix phase locking (#1230)

* add logs for 2 phase lock

* add update log

* add root in log

* add update log

* use update lock

* add log in update repo

* use save

* use exec

* log rows affected

* use no key update

* use repo update

* cleanup

* add defer

* add defer

* Fix

* Added logging

* Debug

* Debug

* Added reference to logs

* Fix

* Fix

* Updated gosdk

* Use slice instead of ll

* Validator last 5 transactions

* Fix

* Fix

* Updated gosdk

* Resolved comments

* Updated gosdk

* Fix

* Increase time limit

* Increase time limit

* Fix

* Fix

* Fix

* Fix

* Debug

* Debug

* Debug

* Fix

* Fix

* Fix

* Cleanup logging

* Fix

* Debug

* Fix

---------

Co-authored-by: Hitenjain14 <57557631+Hitenjain14@users.noreply.github.com>

* remove prefix (#1257)

* remove .gpg file if already exist

* introduce flock to apt commands

* check on docker context existance and install git

* change buildx to action

* refine buildx build name based on runner name

* create docker context based on runner name

* increase blobber and validator job timeout

* Refine docker buildx creation and usage

* Create Unique buildx per job per runner

* remove qemu and upgrade go action version to v4

* changed runners labels build-&-publish-docker-image.yml

* added server cleanup along with blobber build

* ADD disk cleanup for conductor test

* fix cleanup script

* fix cleanup script

* added sudo to perform cleanup

* bypassing cleanup failure

* bypassing cleanup failure for validators

* Fix Save change (#1263)

* add and update change

* fix empty allocation id in change

* add rename dir

* Fix move change when srcPath is same as destPath (#1270)

* fix move change when dest is same as src

* add fileID for new ref

* fix unit test

* Fix challenge timing issue (#1277)

* Fix challenge timing issue

* Fix challenge timing issue

* Fix challenge timing issue

* bypass cleanup if it is already running (#1280)

* Revert "Fix challenge timing issue (#1277)" (#1283)

* Limit Max File Size on blobber (#1274)

* Added update in config

* 5TB config

* Reverted max_file_size config on blobber and fetching from 0chain

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix unit tests

---------

Co-authored-by: dabasov <yuderbasov@gmail.com>

* add l2 caching for alloc (#1258)

* add l2 caching for alloc

* rmv return from commit repo

* use update

* fix update

* add log

* rmv update object

* add log for allocation

* move lock to middleware

* fix save alloc update

* empty commit

* fix blobber size update

* fix unit test

* cleanup

* rmv commit in initMap

* fix unit test

* fix renamefile mock db

* add commit method to enhancedDB

---------

Co-authored-by: Yury <yuderbasov@gmail.com>

* add list query param (#1273)

* Optimize GetRefs, correct and add indexes on ref table (#1284)

* Correct index on ref table, add path index for get refs optimization

* Fix gorm tags on ref table

* Fix typo

* fix goose migration tags

* add pg_trgm extension

* sharder keep list (#1266)

* changed get round

* updated version

* Fix/finalize (#1285)

* Debug finalize

* Debug finalize

* Debug finalize

* Debug finalize

* Debug finalize

* Fix finalize

* Debug

* Fix

* FIX

* Fixed

* Fixed

* fix list file (#1288)

* Cleanup worker for challenge tables (#1278)

* add cleanup worker for challenge tables

* move cleanupGap to config

---------

Co-authored-by: Yury <yuderbasov@gmail.com>

* add err check and increase wait time (#1289)

* Client stats and blacklist (#1286)

* add client stats and blacklist

* add client stats check

* fix goose migration

* fix migration

* fix typo

---------

Co-authored-by: Yury <yuderbasov@gmail.com>

* Fix nested rename dir (#1295)

* add log for rootRef

* add objTreePath

* fix rename nested dir

* use save (#1297)

* fix retry redeem wm check (#1299)

* Use rename in place of copy buffer (#1298)

* use rename than copy buffer

* fix storage tests

* fix update latest wm (#1301)

* Fix blobber stats (#1305)

* fix blobber stats

* add stats worker

* rmv listAlloc

* build stats on start

* Feat/remove native sc calls (#1306)

* removed native MakeSCRestAPICall

* removed native MakeSCRestAPICall

---------

Co-authored-by: Jayash Satolia <73050737+Jayashsatolia403@users.noreply.github.com>

* WM cleanup (#1296)

* add cleanup worker for wm

* add hdd tablespace

* add log for hdd path

* update default hdd path

* update owner

* add init db script

* we are already creating with tablespace in initdb.sh

* update initdb script.sh

* update sql for write_markers_archive table

---------

Co-authored-by: Manohar Reddy <b.manu199@gmail.com>
Co-authored-by: Yury <yuderbasov@gmail.com>

* Fix/max file size updateWorker and added config for storagesc update interval (#1308)

* Added worker for max file size

* Added config for update worker

* Debug

* Added logs for update worker

* Added fix for worker timings =

* Debug

* Debug

* Debug

* Debug

* Debug

* Removed logging

* Removed logging

* Refactor =

* Refactor

* fix tablespace permission issue in conductor test b0docker-compose.yml

* Fix/cond tablespace (#1309)

* updated docker-compose wrt conductor test tablspace

* updated docker-compose wrt conductor test tablspace

* updated docker-compose wrt conductor test tablspace

* updated blobber env

* update gosdk version to latest of sprint-1.11 (#1310)

* fix blobber stats (#1314)

* race condition fix (#1316)

* Node recovery conductor tests (#1259)

* notify on validator tickert generatd

* fix race condition

* Added changes for fail upload commit (#1318)

* Feat/download verify ct (#1317)

* notify on validator tickert generatd

* fix race condition

* miss up download for CT

* debug logs

* Improve download performance (#1315)

* improve download file

* write data

* rmv content length

* fix unit test

* add new db snapshot migration (#1312)

Co-authored-by: Yury <yuderbasov@gmail.com>

* fix commit error (#1320)

* update gosdk (#1319)

Co-authored-by: Yaroslav Svitlytskyi <53532703+YarikRevich@users.noreply.github.com>

* fix where statement (#1321)

* Fix/loop break (#1322)

* updated gosdk

* add mutex

* updated gosdk

* merged changes

* merged changes

---------

Co-authored-by: Hitenjain14 <hitenjain146@gmail.com>

* Fix blobber stats panic (#1324)

* Fix blobber stats panic

* Fix

* Fix

* fix break in switch (#1326)

* change hasher to blake3 (#1325)

* change hasher to blake3

* add blake hash

* fix lint

* fix lint for hash write

* update gosdk

* Fix challenge worker (#1327)

* revert changes

* fix challenge worker

* fix lint

* add goto for accept case

---------

Co-authored-by: Yury <yuderbasov@gmail.com>

* Config watcher (#1329)

* add watch config

* add on config change

* Remove min lock demand from blobber (#1332)

* fix alloc lock (#1334)

* Feature: implement fork per run strategy (#1335)

* feature: added fork creation

* feature: improved Tenderly fork creation flow

* Merge pull request #1336

* rmv wm lock table

* fix unit tests

* rmv write_locks table

* rmv gorm tag

* empty commit

* Fix mem usage (#1337)

* updated gosdk to blobber

* change hasher to sha2 256

* fix unit test

* changed runner for lint tests.yml

---------

Co-authored-by: shahnawaz-creator <msac8225@gmail.com>
Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com>

* fix rb for failed marker (#1342)

* Fix lwm (#1338)

* add wm log

* fix log

* fix update allocation

* fix log

* lru of value

* fix alloc update

* Fix auth ticket (#1341)

* fix read auth ticket

* fix unit test

* convert to base64

* Fix expiration column name (#1343)

* Fix expiration column name

* Fix size param

* merged changes

* Revert "merged changes"

This reverts commit 39f717a.

* feature: added custom block number during Tenderly fork creation (#1345)

* fix update alloc (#1347)

* extended field for mimetype

* add aws secrets for blobbers (#1349)

* add aws secrets for blobbers

* add validator secrets

* typo

* update logs

---------

Co-authored-by: sanchit <sanchit4242.com>

* Fix root hash (#1346)

* add root logs

* check alloc root

* fix lint

* rmv wm cleanup

* fix lint

* fix encrypted key point (#1350)

* fix encrypted key point

* check root ref precommit

* update min-submit to 20% (#1348)

* update gosdk to v1.11.0 (#1351)

* fix wm retries (#1352)

---------

Co-authored-by: Jayash Satolia <73050737+Jayashsatolia403@users.noreply.github.com>
Co-authored-by: Hitenjain14 <57557631+Hitenjain14@users.noreply.github.com>
Co-authored-by: Kishan Dhakan <kishandhakan17@gmail.com>
Co-authored-by: Kishan Dhakan <42718091+Kishan-Dhakan@users.noreply.github.com>
Co-authored-by: Laxmi Prasad Oli <olilaxmiprasad@gmail.com>
Co-authored-by: Ebrahim Gomaa <ebrahim.gomaa.hg@gmail.com>
Co-authored-by: Amr Amin <amr_ashraf_mohamed_amin@yahoo.com>
Co-authored-by: shahnawaz-creator <117025384+shahnawaz-creator@users.noreply.github.com>
Co-authored-by: shahnawaz-creator <msac8225@gmail.com>
Co-authored-by: tapishsinha-rs <tapish.sinha@remotestate.com>
Co-authored-by: Dinmukhammed Kambarov <52813950+din-mukhammed@users.noreply.github.com>
Co-authored-by: Manohar Reddy <b.manu199@gmail.com>
Co-authored-by: Yaroslav Svitlytskyi <53532703+YarikRevich@users.noreply.github.com>
Co-authored-by: Hitenjain14 <hitenjain146@gmail.com>
Co-authored-by: Sanchit Sharma <47826073+Sanchit011@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dabasov dabasov dabasov approved these changes

@peterlimg peterlimg Awaiting requested review from peterlimg peterlimg is a code owner

@guruhubb guruhubb Awaiting requested review from guruhubb guruhubb is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Hitenjain14 @codecov-commenter @dabasov @shahnawaz-creator