Make it possible to configure backpressure delay

[knizhnik]

No description provided.

[lubennikovaav]

UPD: I tested it with 10MB max_replication_write_lag and test_bulk_insert still fails on my laptop

Is it debug or release build?

It is the debug build.

I am a bit surprised that bigger delay shows better result.

I can not reproduce this result. In may case increasing zenith_backpressure_delay to 100msec just increase time to 547 seconds.

I double checked and it must have been some interferring process on my laptop. Results are around 600seconds.

May be time of test_bulk_insert is increased after reducing max_wal_replication_lag. But it is expected (more aggressive backpressure may increase write time). But it seems to me that we are going to increase test timeout in any case, because test_pageserver_chaos test

It fails due to statement timeout (it is set to 120s in python tests).
I am ok with increasing it for this test, but I think we should define the acceptable speed.
Or we can simply split this huge insert into two transactions.

[knizhnik]

i really want to receive some feedback concerning propose algorithm.
Why do I think that it is better than Vegas for our needs?
Vegas algorithm adjusts TCP window based on roundtrip time.
We do not know roundtrip time and do not have any analog of TCP window.
Certainly we can somehow try to emulate them. But it requires more changes in walsender.

Also this two algorithms are addressing different problems: in case of TCP sending more packets than receiver can consume just cause loosing packages and resends. Which even more increase network load and escalate the problem. So it is necessary to react fast.

In our case large lag just means long wait time. It is certainly bad but not fatal until wait time exceeds timeout (one minute). So out backpressure mechanism can react more fluently, trying to keep lag in the specified boundaries.

Also performance of pageserver can greatly depends on activity of other tnants and background tasks (as compaction and checkpointing). It is not so good to completely block writ transaction just because at this moment page server is busy with other tasks. If we can estimate that in some reasonable period of time pageserver will be able to catch up, then there is no need to completely suspend writer. It ca b done by measuring moving average of WAL producing/consuming speed. Choosing right window size (for moving average) may be a challenge. Because with small window we will suffer from pageserver performance fluctuation caused by background activities. Too larger window doesn't allow us to react fast enough on changed workload.

[hlinnaka]

I haven't looked very closely at the algorithm you implemented in this PR. I'll do that next, but let me reply first:

i really want to receive some feedback concerning propose algorithm. Why do I think that it is better than Vegas for our needs? Vegas algorithm adjusts TCP window based on roundtrip time. We do not know roundtrip time and do not have any analog of TCP window. Certainly we can somehow try to emulate them. But it requires more changes in walsender.

We can measure the roundtrip time, same as with TCP implementations. That's pretty straightforward.

max_replication_write_lag is the analogue of TCP send window. In TCP, the send window is the max amount of data sent, but not yet acknowledged by the receiver as received. For us, max_replication_write_lag is the max amount of WAL generated, but not yet acknowledged by the page server as received. Same thing.

To put that the other way 'round, our current mechanism with the max_replication_write_lag setting is the same as running TCP with a constant send window size.

One difference is that we cannot directly stop sending WAL, when the window is full. We can only stop after the window has already been exceeded, and pause the backend so that it won't generate even more WAL exceeding the window even more. But I don't think that difference affects the overall algorithm and math much.

Also this two algorithms are addressing different problems: in case of TCP sending more packets than receiver can consume just cause loosing packages and resends. Which even more increase network load and escalate the problem. So it is necessary to react fast.

In our case large lag just means long wait time. It is certainly bad but not fatal until wait time exceeds timeout (one minute). So out backpressure mechanism can react more fluently, trying to keep lag in the specified boundaries.

Yeah, if the send window is too large, we won't experience packet loss. We will just see increased latency. That doesn't change the overall algorithm and math though. We might want to tune the algorithm so that it settles on a somewhat larger send window size than you'd want with TCP, but the same concepts and reasoning still applies.

In fact, in the networking world, "bufferbloat" has been a hot topic in recent years. That's an analogous thing: if there are nodes in the middle of your network path that do buffering, you will start to see increased roundtrip times long before you see packet loss. And people try to avoid that.

Also performance of pageserver can greatly depends on activity of other tnants and background tasks (as compaction and checkpointing). It is not so good to completely block writ transaction just because at this moment page server is busy with other tasks. If we can estimate that in some reasonable period of time pageserver will be able to catch up, then there is no need to completely suspend writer. It ca b done by measuring moving average of WAL producing/consuming speed. Choosing right window size (for moving average) may be a challenge. Because with small window we will suffer from pageserver performance fluctuation caused by background activities. Too larger window doesn't allow us to react fast enough on changed workload.

Yeah, I'm not sure what exactly the right parameters for the algorithm here are. Any algorithm we choose will have to strike a balance between reacting quickly to changes in how fast the pageserver can process the WAL, and read latency.

[hlinnaka]

Ok, some comments on the "predictable" algorithm:

ProcessInterrupts is not a predictable timer interrupt. It's very unpredictable when it gets called, and how frequently. That depends a lot on what query, and which part of query execution is currently running. As a result, the length of history you use to estimate the rx and tx speed will vary a lot. It could be 1 ms or it could be 100 s, and could swing between the extremes. It might work OK in practice, and I'm not sure what the optimal length of the history would be, but it sure seems weird.

Another weird thing is that the history is collected and the logic is executed separately in each backend, even though there is only one WAL stream. One backend might estimate that rx_speed is 100, while another backend might estimate that it's 1000, at the same instant. The second backend would feel free to generate WAL at speed 700, for example, while the first backend would feel that it needs to throttle. Assuming the real rx_speed is constant, both will converge on the correct value, but one might react much faster than the other.

The rx_speed this estimates is the rate at which the pageserver consumes WAL, not the rate it could consume it. For example, if postgres generates a slow stream of WAL at rate 1 kB / s, rx_speed will converge to 1 kB / s, even if the pageserver could process WAL at 100 MB / s. That leads to a slow start effect: if you then start a bulk operation that generates WAL at 100 MB / s, it will take a while for the estimated rx_speed to ramp up to 100 MB / s. Actually, is there anything to guarantee that it will ever ramp up? If the algorithm tries to maintain that tx_speed = rx_speed, and in the long run the average rx_speed cannot be higher than average tx_speed, and the rx_speed estimate is initially too small, how does it ever catch up to the real rx_speed? Or worse, if there's some error that causes rx_speed to be constantly estimated a little bit too low, the error will cumulate and rx_speed and tx_speed will converge to zero over time.

That said, who knows, this might work fine in practice, even if it's horrible in theory :-). We desperately need performance tests to test the behaviour in different scenarios (neondatabase/neon#1889).

[knizhnik]

ProcessInterrupts is not a predictable timer interrupt.

I agree with you. Yes, interrupts in Postgres are checked in many different places and it is hard to predicts exact interval between ProcessInterrupts invocations. But still we can estimate bounds of this interval. CHECK_FOR_INTERRUPTS can not be called too frequently, because otherwise it may affect performance. SO nobody inset it in some performance critical loop. From the other side, there should not be too long delay between CHECK_FOR_INTERRUPTS because in this case backend can not react on events fast enough. So we can assume that interval is in 1ms..1sec range.

For flow control it is enough to estimate average value of interval (and this is doe my my algorithm). By making delay in ProcessInterrupts we actually affect interval between next invocation of ProcessInterrupts - it can not be smaller than chosen delay.

So, yes - we are not able to predict precise value of interval, but we can estimate delay based on average interval value. I do not see big problem here.

Another weird thing is that the history is collected and the logic is executed separately in each backend

Yes, it is also true. But do you think that we should need to have some shared data structure and use some mutex to control access to it? It seems to be be overkill. Size of this structure is not expected to be so large and do not significantly increase memory consumption of backend. And data collected in this history (LSN) is actually global. So average speeds calculated by different backends should be mostly the same.

Also please notice that history is filled only if lag exceeds backpressure threshold. So we stattically allocate memory for this history but it is only accessed when backpressure is actually needed. Do you think that we should create this buffer dynamically on demand?

One more moment: right now backpressure is mostly needed during bulk load: when data is populated just by one backend.

The rx_speed this estimates is the rate at which the pageserver consumes WAL, not the rate it could consume it.

And once again, you are right and I realized it. But my assumption was that if pageserver is not working on maximal possible speed, then there should be no large lag between it and backend and backpressure will not be needed at all.
This algorithm start to work when pageservr can not catch up and lag exceeds specified threshold.

As far as performance of pageserver depends on so many factors, any algorithm we will use will not be precise.
We in any case have to use some probabilistic algorithm.

So I just want to once again describe the situation with backpressure (from my point of view):

1. Backpressure is absolutely needed. We never can guaranty that pageserver will be fast enough to accept WAL faster than backends can produce them.
2. 500MB max_replication_writ_lag is not acceptable. And reducing it to 100Mb also doesn't help. In both case we have ~1 minute delays for login and selects. With my second related PR (Cache last written LSN for last updated relations to reduce wait LSN time for queries to other relations #167) such large login delay can be prevented. But if application tries to access populated table, then it still faced with minute delays.
3. Reducing max_replication_write_lag to 10Mb (in this case delays are within few seconds and so can be assumed as acceptable) slowdown writer about 2-3 times. It is quite noticeable. This is why I think that we should try to provide some smarter backpressure algorithm than "stop and wait". Especially taken in account that such small max_replication_write_lag may cause deadlock or starvation because walsender will just not be notified while backend is already blocked.

This is why my proposl is:

1. Commit PR with last written LSN cache. It seems to be quite simple and safe, having not impact on performance and preventing most critical problem: long login delays.
2. Think more about flow control algorithm. I am not completely sure that algorithm I have proposed is the best one and agree with your critics. But it is in any case better (and safer) than what we have now.
3. Reducing max_replication_write_lag` to 10Mb with current algorithm (stop and wait) is not desirable. We can choose some "compromise" value (100Mb), although it is still nto enough to eliminate the problem. 10Mb lg threshold cn be used only with smarter backpressure algorithm.