Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When the slave sets maintenance, there is a problem with the status change of proxysql. #752

Open
peterpan104 opened this issue Jul 22, 2024 · 9 comments

Comments

@peterpan104
Copy link

peterpan104 commented Jul 22, 2024

This is the state when repman parameters, repman and proxysql are first start:

## repman parameter
proxy-servers-read-on-master = false
proxy-servers-read-on-master-no-slave = true

image
image

I set 172.24.44.182 as maintenance,
the status of repman and proxysql is as follows:

image
image

Then set 172.24.47.188 as maintenance,
the status of repman and proxysql is as follows:

image
image

Now set 172.24.47.188 as online,
the status of repman and proxysql is as follows:

image
image

then set 172.24.43.182 as online,
the status of repman and proxysql is as follows:

image
image

proxy-servers-read-on-master-no-slave seems to have no effect, but proxysql status is normal

Now I trigger the switchover,
the status of repman and proxysql is as follows:

image
image

I set 172.24.43.182 as maintenance,
the status of repman and proxysql is as follows:

image
image

Then set 172.24.47.188 maintenance,
the status of repman and proxysql is as follows:

image
image

You can see that 172.24.43.182 is online in the proxysql status!

Now I restore the status to after switchover:

image
image

I set 172.24.47.188 as maintenance,
the status of repman and proxysql is as follows:

image
image

Then set 172.24.43.183 as maintenance,
the status of repman and proxysql is as follows:

image
image

At this time, proxysql status is still normal.

Now set 172.24.47.188 as online,
the status of repman and proxysql is as follows:

image
image

You can see that 172.24.43.182 is online in the proxysql status!

I tried setting proxysql-bootstrap=true,
It just allows proxy-servers-read-on-master-no-slave to work,
However, the status of proxysql will still be the same as the above test. Can this problem be fixed?

@svaroqui
Copy link
Collaborator

svaroqui commented Jul 22, 2024

Hi peterpan104 We have to think on it a lot, the solution was never made to allow to put master in maintenance it is just made to avoid that situation to do that and to to preserve the writer traffic . A maintenance would be done on slave one by one , once all slaves have finish maintenance one is promote as master and the old master now a slave or a broken slave will receive the maintenance and get reintroduce as the master

@svaroqui
Copy link
Collaborator

svaroqui commented Jul 22, 2024

Sorry please correct I set 172.24.43.182 as maintenance on the first lines as it lead to miss understand of the issue , we need to reproduce as what you are trying to do make sens, still i would never stop all my slaves to put all read traffic on a single node

@svaroqui
Copy link
Collaborator

Is this scripting on your side or is it reproductible manually by using the same steps from the gui ?

@peterpan104
Copy link
Author

@svaroqui
Sorry, I described it wrong. The above maintenance is all for slave actions.

Under normal conditions, all slaves will not be maintenance.
I am simulating the processing after all slaves are failed.
The above tests don't use any monitor scripts. The operation can be reproduced through the gui.
I will find that it is because my script has set maintenance, But I found that proxysql's changes were not what I expected.

thanks for your reply.

@caffeinated92
Copy link
Collaborator

Do you enable proxysql bootstrap?

@peterpan104
Copy link
Author

HI @caffeinated92
I didn't enable proxysql bootstrap for the above test, but I also enabled this parameter test and the result was the same. The only difference is that when there is no readable machine, master will be used as readable.

@caffeinated92
Copy link
Collaborator

Added bug fix in v2.3.44 but it will need bootstrap to ON. Since we don't want the script to add master in reader list except with acknowledgement

@caffeinated92 caffeinated92 added this to the v2.3.44 milestone Aug 7, 2024
@peterpan104
Copy link
Author

Hi @caffeinated92
I'm using v2.3.44 and setting bootstrap to ON but the problem is not solved.
When I set maintenance on both slaves, one will always show online in proxysql.
Can this issue be opened and fixed?

@svaroqui
Copy link
Collaborator

Hi we are investigating and found some issues in the new logic that maintenance state could differ from proxy to proxy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants