Skip to content

Store last-tailed oplog timestamp so the tailer can resume properly #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Parent: it's no need to wait because all agents are not started at the same t…
merged 5 commits into from
Jul 10, 2018
Merged

Store last-tailed oplog timestamp so the tailer can resume properly #2

merged 5 commits into from
Jul 10, 2018

Conversation

@timvaillancourt
Copy link
Contributor

@timvaillancourt timvaillancourt commented Jul 9, 2018 •
edited
Loading

This PR stores the last-tailed oplog entry so that the tailer can resume from the last position in .tailQuery() properly.

.lastOplogEntry (that stored a whole oplog doc in my POC) is renamed to .lastOplogTimestamp. Storing the bson.MongoTimestamp instead of the full last oplog document (mdbstructs.Oplog) reduces memory from 16MB max (max size of an oplog doc) to a consistent 8 bytes (size of a bson.MongoTimestamp).

.lastOplogTimestamp is a pointer to bson.MongoTimestamp so that it can be checked with "if != nil".

A new struct OplogTimestampOnly was added to reduce umarshalling and GC overhead when we only need to unmarshal the "ts"/timestamp. I moved .getOplogTailTimestamp() to use this also, as it only needs the "ts" field.

@timvaillancourt timvaillancourt changed the title Store last-tailed oplog entry to the tailer can resume properly Store last-tailed oplog entry so the tailer can resume properly Jul 9, 2018
@timvaillancourt timvaillancourt changed the title Store last-tailed oplog entry so the tailer can resume properly Store last-tailed oplog timestamp so the tailer can resume properly Jul 9, 2018
@percona-csalguero percona-csalguero merged commit fb0abb7 into master Jul 10, 2018
@timvaillancourt timvaillancourt deleted the store_last_oplog_entry branch July 11, 2018 17:07
timvaillancourt added a commit that referenced this pull request Jul 18, 2018
@timvaillancourt
Shorten the locking in tailQuery() #2
fb6defa
timvaillancourt added a commit that referenced this pull request Jul 18, 2018
@timvaillancourt
Shorten the locking in tailQuery() #2
0d34f5b
timvaillancourt added a commit that referenced this pull request Jul 18, 2018
@timvaillancourt
Shorten oplog tail lock v1 (#24)
d36a97f 
* Shorten the locking in tailQuery()

* Shorten the locking in tailQuery() #2
timvaillancourt added a commit that referenced this pull request Aug 2, 2018
@timvaillancourt
Improve code comments #2
9e0c5f1
timvaillancourt added a commit that referenced this pull request Aug 2, 2018
@timvaillancourt
Add configsvr+mongos to test #2
1409fef
timvaillancourt added a commit that referenced this pull request Aug 3, 2018
@timvaillancourt
Update readme #2
e91fe70
timvaillancourt added a commit that referenced this pull request Aug 6, 2018
@timvaillancourt
Test isXXXX funcs with db conn #2
31ea0c9
timvaillancourt added a commit that referenced this pull request Aug 6, 2018
@timvaillancourt
Fix broken tests from adding internal/cluster (#38)
bdb2ef0 
* add configsvr shard getter

* Test isXXXX funcs with db conn

* Test isXXXX funcs with db conn #2
timvaillancourt added a commit that referenced this pull request Aug 10, 2018
@timvaillancourt
Use testutils vars in test #2
581ca9d
timvaillancourt added a commit that referenced this pull request Aug 10, 2018
@timvaillancourt
Add internal cluster v5 (#41)
eca90ef 
* add structs for replsets

* Add more internal/cluster tests

* Fix tests

* Scoring WIP

* Splitting up more code

* More scorer tweaks

* more cleanup

* Add more tests

* Fix lag tests

* Cleanup code

* Add shortcut for primary

* Add more tests

* add more tests

* Throw error if compare member is nil

* fix test

* Add comments and improve scorer

* Fix scoring code

* fix scorer tests

* Check for no winner

* Rename structs + file

* Use testutils vars in test

* Use testutils vars in test #2

* Fixing tests
timvaillancourt added a commit that referenced this pull request Aug 11, 2018
@timvaillancourt
Add balancer funcs, decouple funcs that used mgo.Session #2
8ca9ae3
timvaillancourt added a commit that referenced this pull request Aug 11, 2018
@timvaillancourt
Internal cluster balancer funcs (#44)
6d0212d 
* Make isMaster funcs in internal/cluster public, add IsShardsvr()

* Add balancer funcs

* Add balancer funcs, decouple funcs that used mgo.Session

* Add balancer funcs, decouple funcs that used mgo.Session #2
timvaillancourt added a commit that referenced this pull request Aug 18, 2018
@timvaillancourt
Move ShardingState to struct-methods #2
18c7ecd
timvaillancourt added a commit that referenced this pull request Aug 18, 2018
@timvaillancourt
Move missing replset funcs to structmethods v1 (#56)
9de91a6 
* Cleanup tests

* Func rename

* Move map->slice

* Move ShardingState to struct-methods

* Move ShardingState to struct-methods #2
timvaillancourt added a commit that referenced this pull request Aug 19, 2018
@timvaillancourt
Add hotbackup support #2
08c3f53
timvaillancourt added a commit that referenced this pull request Aug 19, 2018
@timvaillancourt
internal/hotbackup v0 (#57)
f45f119 
* Add hotbackup support

* Add hotbackup support #2

* Add hotbackup support #3

* Add hotbackup support #4

* Test hotbackup

* More updates for hot backup

* Fix tests
timvaillancourt added a commit that referenced this pull request Aug 20, 2018
@timvaillancourt
var rename #2
a795366
timvaillancourt added a commit that referenced this pull request Aug 20, 2018
@timvaillancourt
Internal hotbackup v2 (#59)
cf16e55 
* Use strings.SplitN()

* Add error var

* var rename

* var rename #2
timvaillancourt added a commit that referenced this pull request Aug 21, 2018
@timvaillancourt
Check hot backup is supported, split up files #2
d06c214
timvaillancourt added a commit that referenced this pull request Aug 22, 2018
@timvaillancourt
Check ownership of restore+dbpath #2
47448f9
timvaillancourt added a commit that referenced this pull request Aug 22, 2018
@timvaillancourt
Hotbackup restore features, adding dep (#70)
872c5a6 
* Check hot backup is supported, split up files

* Check hot backup is supported, split up files #2

* cleanup testing

* Improve testing

* Add testdata for hotbackup restore

* Add testdata for hotbackup restore

* Add-back defers

* Add restore funcs

* Add restore funcs test

* Add restore tests

* add vendor dir for globalsign/mgo

* Remove root from docker-compose.yml

* Updates

* Update mgo

* remove useless check/removeall

* Check ownership of restore+dbpath

* Check ownership of restore+dbpath #2

* Update restore test

* Use dep

* dep updates

* Fix build automation

* Building readme
timvaillancourt added a commit that referenced this pull request Aug 23, 2018
@timvaillancourt
Use ioutil.TempDir() instead of static test paths #2
14feac2
timvaillancourt added a commit that referenced this pull request Aug 27, 2018
@timvaillancourt
Hotbackup restore v5 (#78)
53424ef 
* Allow test mongodb user to read 'config'

* Add interface

* Add locking, but comment-out because it is broken

* Use ioutil.TempDir() instead of static test paths

* Use ioutil.TempDir() instead of static test paths #2

* Tweaks for hotbackup startServer()
timvaillancourt added a commit that referenced this pull request Nov 2, 2018
@timvaillancourt
Update README.md for Docker images #2
b70bd5f
timvaillancourt added a commit that referenced this pull request Nov 5, 2018
@timvaillancourt
Fix test #2
5311e29
timvaillancourt added a commit that referenced this pull request Nov 5, 2018
@timvaillancourt
Get mongos routers (#116)
760056f 
* Add funcs to find mongos using configsvr

* Fix test

* Fix test

* Fix test #2

* Fix test #3

* Move struct to mdbstructs

* Missing import

* Fix import cycle between internal/testutils/grpc-daemon.go and internal/cluster
timvaillancourt added a commit that referenced this pull request Nov 12, 2018
@timvaillancourt
Add Dockerfile for agent/node #2
a3d24e2
timvaillancourt added a commit that referenced this pull request Nov 12, 2018
@timvaillancourt
Docker images v1 (#120)
6807169 
* Add Dockerfile for agent/node

* Add Dockerfile for agent/node #2

* Add Dockerfile for agent/node #3

* Update agent backup dir

* Update agent backup dir
timvaillancourt added a commit that referenced this pull request Dec 11, 2018
@timvaillancourt
Fix .gitignore #2
1b898ef
timvaillancourt added a commit that referenced this pull request Dec 14, 2018
@timvaillancourt
Update README.md #2
ad9e391
timvaillancourt added a commit that referenced this pull request Dec 20, 2018
@timvaillancourt
Goreleaser ldflags and arch fix #2
0ed976e
timvaillancourt added a commit that referenced this pull request Dec 20, 2018
@timvaillancourt
Goreleaser ldflags and arch fix2 (#150)
813b496 
* Fix --version output, skip i386

* Rename for new repo

* Goreleaser ldflags and arch fix #2
timvaillancourt added a commit that referenced this pull request Dec 26, 2018
@timvaillancourt
CLI --help/flags cleanup #2 (#154)
d669e90 
* Add kingpin short flags + more CLI flag cleanup

* Make 'make clean' cleanup upx tmp files

* Fix typo

* Cleanup 'make clean'

* 'server compression algorithm' -> 'server compression'
timvaillancourt added a commit that referenced this pull request Dec 29, 2018
@timvaillancourt
Fix authenticationRestrictions problem with testing #2
fb521b2
timvaillancourt added a commit that referenced this pull request Dec 29, 2018
@timvaillancourt
Fix authenticationRestrictions problem with testing #2 (#169)
72e7d0e
timvaillancourt added a commit that referenced this pull request Jan 10, 2019
@timvaillancourt
Add install notes #2
dc36888
timvaillancourt added a commit that referenced this pull request Jan 10, 2019
@timvaillancourt
PBM-154: Add install notes to README (#173)
3dbe762 
* Add install notes

* Add install notes #2

* Add install notes #3

* Fix ToC
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@percona-csalguero percona-csalguero percona-csalguero approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@timvaillancourt @percona-csalguero