fix: fixing graphql schema update when the data is restored + skipping /probe/graphql from audit #7925

aman-bansal · 2021-06-24T21:58:58Z

The issue is graphql layer relies upon the badger subscribe events to update the inmemory schema. But with new restore changes, where stream is used to backups, there were no subscribed updated events for the graphql layer.

Hence after restores, gql schema was not getting updated. This PR refreshed the gql schema if there is a successful restore operation. One thing is dgraph stores schema in memory. This PR only makes the current node to refresh the inmemory state of schema. Now restore operation runs on every node. Therefore in the end each node will refresh its schema and thus returns to the stable state.

Skipping /probe/graphql from audit as this is used by cloud to check pods health status

This change is

NamanJain8

Got some comments.

graphql/admin/admin.go

worker/graphql_schema.go

NamanJain8

Would be good to add a test. But should be enough for now if manually tested already.

…nt for kube

…g /probe/graphql from audit (#7925) * fix: fixing grapgql schema update when the data is restored * making audit to skip /probe/graphql endpoint as this is health endpoint for kube (cherry picked from commit b85036e)

…red + skippin… (#7970) * fix: fixing graphql schema update when the data is restored + skipping /probe/graphql from audit (#7925) * fix: fixing grapgql schema update when the data is restored * making audit to skip /probe/graphql endpoint as this is health endpoint for kube (cherry picked from commit b85036e) Co-authored-by: aman bansal <bansalaman2905@gmail.com>

…g /probe/graphql from audit (#7925) * fix: fixing grapgql schema update when the data is restored * making audit to skip /probe/graphql endpoint as this is health endpoint for kube (cherry picked from commit b85036e)

…g /probe/graphql from audit (#7925) * fix: fixing grapgql schema update when the data is restored * making audit to skip /probe/graphql endpoint as this is health endpoint for kube

This commit is a major rewrite of online restore code. It used to use KVLoader in badger. Now it instead uses StreamWriter that is much faster for writes in the case of restore. following commits are cherry-picked (in reverse order): * fix(backup): Free the UidPack after use (#7786) * fix(export-backup): Fix double free in export backup (#7780) (#7783) * fix(lsbackup): Fix profiler in lsBackup (#7729) * Bring back "perf(Backup): Improve backup performance (#7601)" * Opt(Backup): Make backups faster (#7680) * Fix s3 backup copy (#7669) * [BREAKING] Opt(Restore): Optimize Restore's new map-reduce based design (#7666) * Perf(restore): Implement map-reduce based restore (#7664) * feat(backup): Merge backup refactoring * Revert "perf(Backup): Improve backup performance (#7601)" * fix(ee): GetKeys should return an error (#7713) (#7797) * fix(restore): Bump uid and namespace after restore (#7790) (#7800) * fix: fixing graphql schema update when the data is restored + skipping /probe/graphql from audit (#7925) * Don't ban namespace in export_backup

This commit is a major rewrite of backup and online restore code. It used to use KVLoader in badger. Now it instead uses StreamWriter that is much faster for writes. cherry-pick PR #7753 following commits are cherry-picked (in reverse order): * Don't ban namespace in export_backup * fix: fixing graphql schema update when the data is restored + skipping /probe/graphql from audit (#7925) * fix(restore): Bump uid and namespace after restore (#7790) (#7800) * fix(ee): GetKeys should return an error (#7713) (#7797) * fix(backup): Free the UidPack after use (#7786) * fix(export-backup): Fix double free in export backup (#7780) (#7783) * fix(lsbackup): Fix profiler in lsBackup (#7729) * Bring back "perf(Backup): Improve backup performance (#7601)" * Opt(Backup): Make backups faster (#7680) * Fix s3 backup copy (#7669) * [BREAKING] Opt(Restore): Optimize Restore's new map-reduce based design (#7666) * Perf(restore): Implement map-reduce based restore (#7664) * feat(backup): Merge backup refactoring * Revert "perf(Backup): Improve backup performance (#7601)"

This commit is a major rewrite of backup and online restore code. It used to use KVLoader in badger. Now it instead uses StreamWriter that is much faster for writes. cherry-pick PR #7753 following commits are cherry-picked (in reverse order): * reset the kv.StreamId before sending to stream writer (#7833) (#7837) * Don't ban namespace in export_backup * fix: fixing graphql schema update when the data is restored + skipping /probe/graphql from audit (#7925) * fix(restore): Bump uid and namespace after restore (#7790) (#7800) * fix(ee): GetKeys should return an error (#7713) (#7797) * fix(backup): Free the UidPack after use (#7786) * fix(export-backup): Fix double free in export backup (#7780) (#7783) * fix(lsbackup): Fix profiler in lsBackup (#7729) * Bring back "perf(Backup): Improve backup performance (#7601)" * Opt(Backup): Make backups faster (#7680) * Fix s3 backup copy (#7669) * [BREAKING] Opt(Restore): Optimize Restore's new map-reduce based design (#7666) * Perf(restore): Implement map-reduce based restore (#7664) * feat(backup): Merge backup refactoring * Revert "perf(Backup): Improve backup performance (#7601)"

This commit is a major rewrite of backup and online restore code. It used to use KVLoader in badger. Now it instead uses StreamWriter that is much faster for writes. cherry-pick PR #7753 following commits are cherry-picked (in reverse order): * fix(backup): Fix full backup request (#7932) (#7933) * fix: fixing graphql schema update when the data is restored + * fix(restore): return nil if there is error (#7899) skipping /probe/graphql from audit (#7925) * Don't ban namespace in export_backup * reset the kv.StreamId before sending to stream writer (#7833) (#7837) * fix(restore): Bump uid and namespace after restore (#7790) (#7800) * fix(ee): GetKeys should return an error (#7713) (#7797) * fix(backup): Free the UidPack after use (#7786) * fix(export-backup): Fix double free in export backup (#7780) (#7783) * fix(lsbackup): Fix profiler in lsBackup (#7729) * Bring back "perf(Backup): Improve backup performance (#7601)" * Opt(Backup): Make backups faster (#7680) * Fix s3 backup copy (#7669) * [BREAKING] Opt(Restore): Optimize Restore's new map-reduce based design (#7666) * Perf(restore): Implement map-reduce based restore (#7664) * feat(backup): Merge backup refactoring * Revert "perf(Backup): Improve backup performance (#7601)"

This commit is a major rewrite of backup and online restore code. It used to use KVLoader in badger. Now it instead uses StreamWriter that is much faster for writes. cherry-pick PR #7753 following commits are cherry-picked (in reverse order): * opt(restore): Sort the buffer before spinning the writeToDisk goroutine (#7984) (#7996) * fix(backup): Fix full backup request (#7932) (#7933) * fix: fixing graphql schema update when the data is restored + * fix(restore): return nil if there is error (#7899) skipping /probe/graphql from audit (#7925) * Don't ban namespace in export_backup * reset the kv.StreamId before sending to stream writer (#7833) (#7837) * fix(restore): Bump uid and namespace after restore (#7790) (#7800) * fix(ee): GetKeys should return an error (#7713) (#7797) * fix(backup): Free the UidPack after use (#7786) * fix(export-backup): Fix double free in export backup (#7780) (#7783) * fix(lsbackup): Fix profiler in lsBackup (#7729) * Bring back "perf(Backup): Improve backup performance (#7601)" * Opt(Backup): Make backups faster (#7680) * Fix s3 backup copy (#7669) * [BREAKING] Opt(Restore): Optimize Restore's new map-reduce based design (#7666) * Perf(restore): Implement map-reduce based restore (#7664) * feat(backup): Merge backup refactoring * Revert "perf(Backup): Improve backup performance (#7601)"

This commit is a major rewrite of backup and online restore code. It used to use KVLoader in badger. Now it instead uses StreamWriter that is much faster for writes. cherry-pick PR #7753 following commits are cherry-picked (in reverse order): * opt(restore): Sort the buffer before spinning the writeToDisk goroutine (#7984) (#7996) * fix(backup): Fix full backup request (#7932) (#7933) * fix: fixing graphql schema update when the data is restored + skipping /probe/graphql from audit (#7925) * fix(restore): return nil if there is error (#7899) * Don't ban namespace in export_backup * reset the kv.StreamId before sending to stream writer (#7833) (#7837) * fix(restore): Bump uid and namespace after restore (#7790) (#7800) * fix(ee): GetKeys should return an error (#7713) (#7797) * fix(backup): Free the UidPack after use (#7786) * fix(export-backup): Fix double free in export backup (#7780) (#7783) * fix(lsbackup): Fix profiler in lsBackup (#7729) * Bring back "perf(Backup): Improve backup performance (#7601)" * Opt(Backup): Make backups faster (#7680) * Fix s3 backup copy (#7669) * [BREAKING] Opt(Restore): Optimize Restore's new map-reduce based design (#7666) * Perf(restore): Implement map-reduce based restore (#7664) * feat(backup): Merge backup refactoring * Revert "perf(Backup): Improve backup performance (#7601)"

This commit is a major rewrite of backup and online restore code. It used to use KVLoader in badger. Now it instead uses StreamWriter that is much faster for writes. cherry-pick PR #7753 following commits are cherry-picked (in reverse order): * opt(restore): Sort the buffer before spinning the writeToDisk goroutine (#7984) (#7996) * fix(backup): Fix full backup request (#7932) (#7933) * fix: fixing graphql schema update when the data is restored + * fix(restore): return nil if there is error (#7899) skipping /probe/graphql from audit (#7925) * Don't ban namespace in export_backup * reset the kv.StreamId before sending to stream writer (#7833) (#7837) * fix(restore): Bump uid and namespace after restore (#7790) (#7800) * fix(ee): GetKeys should return an error (#7713) (#7797) * fix(backup): Free the UidPack after use (#7786) * fix(export-backup): Fix double free in export backup (#7780) (#7783) * fix(lsbackup): Fix profiler in lsBackup (#7729) * Bring back "perf(Backup): Improve backup performance (#7601)" * Opt(Backup): Make backups faster (#7680) * Fix s3 backup copy (#7669) * [BREAKING] Opt(Restore): Optimize Restore's new map-reduce based design (#7666) * Perf(restore): Implement map-reduce based restore (#7664) * feat(backup): Merge backup refactoring * Revert "perf(Backup): Improve backup performance (#7601)"

This commit is a major rewrite of backup and online restore code. It used to use KVLoader in badger. Now it instead uses StreamWriter that is much faster for writes. cherry-pick PR #7753 following commits are cherry-picked (in reverse order): * opt(restore): Sort the buffer before spinning the writeToDisk goroutine (#7984) (#7996) * fix(backup): Fix full backup request (#7932) (#7933) * fix: fixing graphql schema update when the data is restored + skipping /probe/graphql from audit (#7925) * fix(restore): return nil if there is error (#7899) * Don't ban namespace in export_backup * reset the kv.StreamId before sending to stream writer (#7833) (#7837) * fix(restore): Bump uid and namespace after restore (#7790) (#7800) * fix(ee): GetKeys should return an error (#7713) (#7797) * fix(backup): Free the UidPack after use (#7786) * fix(export-backup): Fix double free in export backup (#7780) (#7783) * fix(lsbackup): Fix profiler in lsBackup (#7729) * Bring back "perf(Backup): Improve backup performance (#7601)" * Opt(Backup): Make backups faster (#7680) * Fix s3 backup copy (#7669) * [BREAKING] Opt(Restore): Optimize Restore's new map-reduce based design (#7666) * Perf(restore): Implement map-reduce based restore (#7664) * feat(backup): Merge backup refactoring * Revert "perf(Backup): Improve backup performance (#7601)"

aman-bansal added 2 commits June 25, 2021 02:20

fix: fixing grapgql schema update when the data is restored

e88656f

making to run on all nodes

4de0986

aman-bansal requested a review from manishrjain as a code owner June 24, 2021 21:58

github-actions bot added the area/graphql Issues related to GraphQL support on Dgraph. label Jun 24, 2021

fmt run fix

9d6cbe4

NamanJain8 reviewed Jun 25, 2021

View reviewed changes

graphql/admin/admin.go Outdated Show resolved Hide resolved

worker/graphql_schema.go Outdated Show resolved Hide resolved

worker/graphql_schema.go Outdated Show resolved Hide resolved

worker/graphql_schema.go Show resolved Hide resolved

NamanJain8 approved these changes Jun 25, 2021

View reviewed changes

aman-bansal added 3 commits June 28, 2021 13:30

review comments fix

f5398d4

adding logs and renaming few bits

78d41b2

making audit to skip /probe/graphql endpoint as this is health endpoi…

228bd72

…nt for kube

aman-bansal changed the title ~~fix: fixing graphql schema update when the data is restored~~ fix: fixing graphql schema update when the data is restored + skipping /probe/graphql from audit Jun 28, 2021

aman-bansal merged commit b85036e into release/v21.03-slash Jun 28, 2021

aman-bansal deleted the aman/data_restore_fix branch June 28, 2021 08:51

ahsanbarkati mentioned this pull request Aug 2, 2021

cherry-pick: fix: fixing graphql schema update when the data is restored + skippin… #7970

Merged

mangalaman93 mentioned this pull request Jan 4, 2023

fix(probe): do not contend for lock in lazy load (#8037) (#8041) #8566

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: fixing graphql schema update when the data is restored + skipping /probe/graphql from audit #7925

fix: fixing graphql schema update when the data is restored + skipping /probe/graphql from audit #7925

aman-bansal commented Jun 24, 2021 •

edited

Loading

NamanJain8 left a comment

NamanJain8 left a comment

fix: fixing graphql schema update when the data is restored + skipping /probe/graphql from audit #7925

fix: fixing graphql schema update when the data is restored + skipping /probe/graphql from audit #7925

Conversation

aman-bansal commented Jun 24, 2021 • edited Loading

NamanJain8 left a comment

Choose a reason for hiding this comment

NamanJain8 left a comment

Choose a reason for hiding this comment

aman-bansal commented Jun 24, 2021 •

edited

Loading