Skip to content

Undo disarming VM check warning in vacuumlazy.c #184

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 156 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
156 commits
Select commit Hold shift + click to select a range
d49776a
[smgr_api] [community] smgr_api.patch
lubennikovaav May 16, 2021
497ae12
[contrib/zenith] contrib_zenith.patch
lubennikovaav May 16, 2021
0bea9e9
[walredo] zenith_wal_redo.patch
lubennikovaav May 16, 2021
eb463f0
lastWrittenPageLSN.patch
lubennikovaav May 16, 2021
db82549
Fix GetPage requests right after replaying CREATE DATABASE
hlinnaka May 21, 2021
712d53f
handle_eviction_of_non_wal_logged_pages.patch
lubennikovaav May 16, 2021
736487d
[walproposer] wal_proposer.patch
lubennikovaav May 16, 2021
1b994c3
persist_unlogged_tables.patch
lubennikovaav May 16, 2021
8d7a4ea
fix_pg_table_size.patch
lubennikovaav May 16, 2021
480a745
[walredo] fix_gin_redo.patch
lubennikovaav May 16, 2021
7f82995
[walredo] fix_brin_redo.patch
lubennikovaav May 16, 2021
a5e82cb
speculative_records_workaround.patch
lubennikovaav May 16, 2021
176878f
wallog_t_ctid.patch
lubennikovaav May 16, 2021
b67388f
vacuumlazy_debug_stub.patch
lubennikovaav May 16, 2021
497c6a1
[test] zenith_test_evict.patch
lubennikovaav May 16, 2021
0ad2d1c
fix_sequence_wallogging.patch
lubennikovaav May 16, 2021
bf5d710
Bring back change that got lost in refactoring. silence ReadBuffer_co…
lubennikovaav May 18, 2021
1c6a260
[contrib/zenith] [refer #225] if insert WAL position points at the en…
knizhnik Jun 7, 2021
45fad00
[walproposer] Create replication slot for walproposer to avoid loose …
knizhnik Jun 8, 2021
7589331
[walproposer] Skip absent WAL segment removed by pg_resetwal
lubennikovaav Aug 17, 2021
785854a
[walproposer] Fix breaking out of WalProposerPoll and WaitEventSetWai…
arssher Jun 11, 2021
960cb8c
[walproposer] Make it possible to start postgres without reading chec…
knizhnik Jun 17, 2021
4cdee32
[walproposer] Simplify WL_LATCH_SET testing in the walproposer
ololobus Jul 5, 2021
4595a2f
[walredo] Add basic support for Seccomp BPF mode
funbringer Jun 24, 2021
81735a8
[smgr_api] [contrib/zenith] 1. Do not call mdinit from smgrinit() bec…
knizhnik Jul 9, 2021
96b0212
[walproposer] [contrib/zenith] support zenith_tenant
LizardWizzard Jul 15, 2021
1da7f49
[walproposer] Remove graceful termination of COPY during walproposer …
arssher Aug 4, 2021
d84f93e
[walproposer] [contrib/zenith] [refer #395] Do no align sart replicat…
knizhnik Aug 9, 2021
f0aeb64
[test] Add contrib/zenith_test_utils with helpers for testing and deb…
lubennikovaav Aug 10, 2021
843ec86
[walproposer] Change condition for triggering recovery
knizhnik Aug 11, 2021
e1a2580
[contrib/zenith] Use authentication token passed as environment varia…
LizardWizzard Aug 4, 2021
f2579e8
[contrib/zenith] Fix race condition while WAL-logging page, leading t…
hlinnaka Aug 13, 2021
64b4cec
[walproposer] Rework walkeeper protocol to use libpq (#60)
sharnoff Aug 13, 2021
aad79ea
Build zenithdb/compute-node:latest in CI (zenithdb/console#125)
ololobus Aug 16, 2021
a35c8dd
zenith_regression_tests.patch
lubennikovaav Aug 23, 2021
cc6c1ca
Add test function to flush the shared buffer cache.
hlinnaka Aug 24, 2021
fe0f234
Basic safekeeper refactoring and bug fixing.
arssher Aug 17, 2021
2407a9c
Rename VCL to epochStartLsn and restart_lsn to truncate_lsn.
arssher Aug 27, 2021
5d2d491
[refer #27] Implement shared relsize cache to improve zenith performa…
knizhnik Aug 3, 2021
677b62a
Cleanup walproposer changes from #60
sharnoff Aug 31, 2021
e7df8a3
pass tenant id in connection string to safekeeper
LizardWizzard Aug 31, 2021
5dced7b
Ask pageserver only with LSN's aligned on record boundary.
kelvich Aug 31, 2021
2280729
[refer #506] Correctly initialize all fields of WAL page header for f…
knizhnik Sep 1, 2021
e6ce2f9
Add --sync-safekeepers starting standalone walproposer to sync safeke…
knizhnik Aug 19, 2021
7a37ace
Update Dockerfile
ololobus Aug 27, 2021
45bd48a
Fix compiler warnings in walproposer.c
ololobus Sep 10, 2021
44e6d6d
Always advance truncateLsn to commitLsn, keeping it on record boundary.
arssher Sep 9, 2021
22592b7
Minor logging editing.
arssher Sep 10, 2021
31f03e1
Fix walproposer starting streaming point.
arssher Sep 10, 2021
5d43dbc
Mark all recovery messages as received by the donor.
arssher Sep 14, 2021
cc14053
Don't FATAL in walproposer when EOF arrives in SS_IDLE state.
arssher Sep 14, 2021
c1f1b51
Optimize walproposer starting streaming point.
arssher Sep 15, 2021
698de74
Silence compiler warnings:
hlinnaka Sep 17, 2021
d240f2e
Remove unused functions for reading non-rel pages.
hlinnaka Sep 17, 2021
a122d30
Misc cleanup in the code that communicates with the page server.
hlinnaka Sep 17, 2021
3c2fe34
Improve the protocol between Postgres and page server.
hlinnaka Sep 17, 2021
f014b81
Run 'pgindent' on zenith-specific code in contrib/zenith and in walpr…
hlinnaka Sep 17, 2021
62b2d74
Fix a badly worded comment
hlinnaka Sep 20, 2021
56493b5
Simplify a2e929e by storing starting point in walkeeper itself.
arssher Sep 16, 2021
eb81d9e
Catch walkeeper ErrorResponse in PQgetCopyData
sharnoff Sep 22, 2021
1b4cb30
Use buffered I/O for reading commands from stdin.
hlinnaka Sep 24, 2021
e20c67f
Replace fread() with plain read() and a hand-written buffer.
hlinnaka Sep 30, 2021
7484d00
Store unlogged tables locally, and replace PD_WAL_LOGGED.
hlinnaka Oct 6, 2021
ddd3792
Fix queue cleanup in proposer (#93)
petuhovskiy Oct 14, 2021
c106d4f
Support read-only nodes
hlinnaka Oct 19, 2021
d40ea90
Remove a drop of syncSafekeepers complexity.
arssher Oct 21, 2021
df92cef
Fix compiler warning.
arssher Oct 21, 2021
c95c552
Implement backpressure for compute node to avoid WAL overflow
knizhnik Sep 18, 2021
353e66c
Initialize FSM/VM pages through buffer cache
ololobus Oct 19, 2021
f39a780
Turn off back pressure by default
ololobus Oct 21, 2021
7a3814b
ShutdownConnection instead of ResetConnection in more places.
arssher Oct 21, 2021
3158a3b
Handle partial writes to stdout in WAL redo process.
hlinnaka Nov 3, 2021
827f3f0
Handle keepalives while receiving WAL in recovery.
arssher Nov 8, 2021
4bd4f68
Fix truncateLsn update (#101)
petuhovskiy Nov 9, 2021
8168cd8
[walproposer] Get rid of SAB_Error after rebase
ololobus Nov 10, 2021
9324044
Add term history to safekeepers.
arssher Nov 5, 2021
0b538bc
Clarify the meaning of StandbyReply LSNs used for backpressure
lubennikovaav Nov 9, 2021
41fb2a4
Use max_replication_apply_lag instead of max_replication_write_lag.
lubennikovaav Nov 11, 2021
e52fd3b
Forward pageserver connection string to safekeeper
LizardWizzard Nov 25, 2021
990799d
Move backpressure throttling to ProcessInterrupts()
lubennikovaav Nov 26, 2021
5d5b1a2
Stop building docker images in this repo.
kelvich Dec 15, 2021
774187e
[walproposer] Async WAL append (#105)
petuhovskiy Dec 21, 2021
e9ac127
Fix walsender to work with zenith style standbyReply that sends non-z…
lubennikovaav Dec 22, 2021
cab155e
Do not copy the obsolete apply_conf binary into Docker image
ololobus Dec 25, 2021
1cddde5
Reorder walproposer code in a more natural order (#112)
petuhovskiy Dec 31, 2021
c2a29ee
Simplify walproposer code (#114)
petuhovskiy Jan 4, 2022
69b7529
Don't change propTerm after quorum is acquired (#107)
petuhovskiy Jan 4, 2022
8ebe570
walproposer renames (#116)
petuhovskiy Jan 5, 2022
d36e6aa
Report back-pressure trottling status of backend
knizhnik Dec 29, 2021
9877c5f
Add max_replication_write_lag
knizhnik Dec 29, 2021
7d5b2a6
Do not throttle wal sender
knizhnik Jan 10, 2022
6863e08
Don't track acks in walproposer (#119)
petuhovskiy Jan 13, 2022
ccfb72a
Silence excessively noisy logging from walproposer.
hlinnaka Jan 14, 2022
5671fba
Extend replication protocol with ZenithFeedback message.
lubennikovaav Dec 20, 2021
495e413
Allow to join empty safekeeper to existing cluster (#123)
petuhovskiy Jan 27, 2022
4708987
Use local relation cache for smgr_exists
knizhnik Dec 29, 2021
04af533
Reduce walproposer logging after ca5e7beaf.
arssher Feb 4, 2022
f2aad65
Implement cluster size quota for zenith compute node.
lubennikovaav Dec 20, 2021
98475d3
Revert "Use local relation cache for smgr_exists"
arssher Feb 9, 2022
ff50910
Change the unit of cluster size limit GUC to MB, and other fixes.
hlinnaka Feb 9, 2022
c1649a2
Improve error handling while connecting to page server.
hlinnaka Feb 9, 2022
ae5f3d5
Make getpage requests interruptible.
hlinnaka Feb 9, 2022
51ab60b
Fix memory leak of messages received from safekeepers.
hlinnaka Feb 10, 2022
0a0d25d
Initialize pgxactoff for walproposer
knizhnik Feb 11, 2022
cb9a942
Fix compilation warning after 8524105dc.
arssher Feb 14, 2022
afac0e8
Fix more compiler warnings.
arssher Feb 14, 2022
024b4c7
Remove dead code in handling ZenithFeedback part of an AppendResponse.
hlinnaka Feb 16, 2022
8ee7462
Prevent recursive call of XLogBeginInsert
knizhnik Dec 24, 2021
b29ad07
Update src/backend/access/gin/ginfast.c
knizhnik Feb 16, 2022
93fc1d8
Expose reading a relation page at a specific LSN (#131)
antons-antons Feb 18, 2022
c62b6b7
Add backpressure_lsns() function. Fix zenith feedback processing
lubennikovaav Feb 17, 2022
2e7775f
Fix zenith_test_utils linkage on macOS
kelvich Feb 23, 2022
2893db3
Add warning fr unrecgonized GUCs with zenith prefix
knizhnik Feb 21, 2022
2d2c207
Count WAL flushes in walreceiver (#139)
petuhovskiy Mar 8, 2022
e166fc8
Use local relation cache for smgr_exists
knizhnik Dec 29, 2021
9af3e75
Populate relsize cache when relation is created.
hlinnaka Mar 10, 2022
bb15da1
Fix pg_table_size() on a view
hlinnaka Mar 28, 2022
7b2c52b
Don't set commitLsn to truncateLsn.
arssher Mar 28, 2022
133ccb2
Raise fatal error on failed recovery (#147)
petuhovskiy Mar 31, 2022
e4678ba
Enable dumping corrupt WAL segments (#145)
antons-antons Apr 1, 2022
7f2e701
Don't hold walproposer WAL in memory (#141)
petuhovskiy Apr 4, 2022
08303a3
Add --sysid parameter to initdb
knizhnik Jan 27, 2022
2095bdf
Give up connection attempt to safekeeper after timeout.
arssher Apr 15, 2022
ab2bbb2
Avoid redundand memory allocation and sycnhronization in walredo (#144)
knizhnik Apr 23, 2022
9f8eb34
Fix missed include for InRecovery (#149)
knizhnik Apr 25, 2022
eafc1f4
Avoid "bad syscall 39" on assertion failure in WAL redo process.
hlinnaka Apr 26, 2022
4f588b2
Add error context, if replaying a WAL record fails in WAL redo process.
hlinnaka Apr 26, 2022
8cf8b4c
Add WARNING for debugging purposes.
hlinnaka Apr 26, 2022
c36cd66
Turn Assertion into elog(ERROR), to help with debugging.
hlinnaka Apr 26, 2022
ebbcdbb
Fix errors in WAL redo about relpersistence mismatch.
hlinnaka Apr 26, 2022
d2a4cb0
Don't log 'last written LSN ahead of flushed'.
arssher Apr 26, 2022
f4611aa
Perform inmem_smgr cleaup after processing each record (#154)
knizhnik Apr 27, 2022
63c0b04
Avoid extending relation in the WAL redo process.
hlinnaka May 3, 2022
ff333ae
Send timeline_start_lsn in Elected and receive it in VoteResponse mes…
arssher May 1, 2022
ff41b70
Verify basebackup LSN against consensus LSN in walproposer.
arssher May 2, 2022
f7ab7b5
Implement pg_database_size():
lubennikovaav Jul 23, 2021
88cd1c4
Shut down instance on basebackup LSN mismatch.
arssher May 6, 2022
46a03a3
Use compute-tools from the new neondatabase Docker Hub repo
ololobus May 9, 2022
f9fb5eb
zenith_test_utils extension: add neon_xlogflush()
yeputons May 10, 2022
6b3a105
Reduce noise in the logs from inmem_write()
hlinnaka May 13, 2022
17142a1
Use compute_ctl instead of zenith_ctl (#162)
ololobus May 18, 2022
4a6a1e0
Improve error messages on seccomp loading errors.
hlinnaka May 24, 2022
55644db
Rename contrib/zenith to contrib/neon. Rename custom GUCs:
lubennikovaav May 26, 2022
25e4321
Rename 'wal_acceptors' GUC to 'safekeepers'
lubennikovaav Apr 19, 2022
ee26890
Fix basebackup LSN comparison in walproposer.
arssher May 31, 2022
493c8d2
Allow specifiyng the different compute-tools tag (#170)
ololobus May 31, 2022
a9714fe
Do not allocate shared memory for wal_redo process (#165)
knizhnik Jun 6, 2022
eace46e
Add check for NULL for malloc in InternalIpcMemoryCreate (#173)
knizhnik Jun 8, 2022
0019c01
Rename ZenithFeedback (#174)
SomeoneToIgnore Jun 10, 2022
b41fbe6
Remove callmemaybe bits from compute (#172)
SomeoneToIgnore Jun 10, 2022
4e3ab45
Panic on receiving higher term (#176)
petuhovskiy Jun 22, 2022
9c99008
Misc cleanup in libpagestore.c.
hlinnaka May 16, 2022
595ac69
Large last written lsn cache (#177)
knizhnik Jul 9, 2022
db67809
Add uuid-ossp to the supported extensions (#181)
MMeent Jul 12, 2022
7517d1c
Update last written LSN for gin/gist index metadata (#182)
knizhnik Jul 19, 2022
ae6d46a
Undo diasming VM check warning in vacuumlazy.c
knizhnik Jul 22, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Use buffered I/O for reading commands from stdin.
Whatever the bug mentioned in the FIXME comment was with buffered I/O,
it has been fixed now. This greatly reduces the amount of CPU time spent
in WAL redo.
  • Loading branch information
hlinnaka authored and MMeent committed Jul 7, 2022
commit 1b4cb30a32f562d6a84dc48e850aa01676bcea32
74 changes: 18 additions & 56 deletions src/backend/tcop/zenith_wal_redo.c
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ enter_seccomp_mode(void)
PG_SCMP_ALLOW(exit_group),
PG_SCMP_ALLOW(pselect6),
PG_SCMP_ALLOW(read),
PG_SCMP_ALLOW(fstat), /* needed by fread() */
PG_SCMP_ALLOW(select),
PG_SCMP_ALLOW(write),

Expand Down Expand Up @@ -433,84 +434,45 @@ pprint_tag(BufferTag *tag)
* EOF is returned if end-of-file input is seen; time to shut down.
* ----------------
*/

/*
* Wait until there is data in stdin. Prints a log message every 10 s whil
* waiting.
*/
static void
wait_with_timeout(void)
{
for (;;)
{
struct timeval timeout = {10, 0};
fd_set fds;
int ret;

FD_ZERO(&fds);
FD_SET(STDIN_FILENO, &fds);

ret = select(1, &fds, NULL, NULL, &timeout);
if (ret != 0)
break;
elog(DEBUG1, "still alive");
}
}

static int
ReadRedoCommand(StringInfo inBuf)
{
char c;
char hdr[1 + sizeof(int32)];
int qtype;
int32 len;
int nread;

/* FIXME: Use unbuffered I/O here, because the WAL redo process was getting
* stuck with buffered I/O. I'm not sure why, or whether the bug was somewhere
* in here or in the calling page server side.
*/
wait_with_timeout();
if (read(STDIN_FILENO, &c, 1) == 0)
return EOF;
qtype = c;

/*
* Like in the FE/BE protocol, all messages have a length word next
* after the type code; we can read the message contents independently of
* the type.
*/
if (read(STDIN_FILENO, &len, 4) != 4)
/* Read message type and message length */
if (fread(hdr, 1, sizeof(hdr), stdin) != sizeof(hdr))
{
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
errmsg("could not read message length")));
if (ferror(stdin) != 0)
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("could not read message header")));
return EOF;
}

qtype = hdr[0];
memcpy(&len, &hdr[1], sizeof(int32));
len = pg_ntoh32(len);

if (len < 4)
{
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
errmsg("invalid message length")));
return EOF;
}

len -= 4; /* discount length itself */

/* Read the message payload */
enlargeStringInfo(inBuf, len);
nread = 0;
while (nread < len) {
int n = read(STDIN_FILENO, inBuf->data + nread, len - nread);
if (n == -1)
if (fread(inBuf->data, 1, len, stdin) != len)
{
if (ferror(stdin) != 0)
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
errmsg("read error: %m")));
if (n == 0)
(errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("could not read message")));
else
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
errmsg("unexpected EOF")));
nread += n;
}
inBuf->len = len;
inBuf->data[len] = '\0';
Expand Down