can not join additional node to cluster

[daniva6]

Issue report

What version of MicroCeph are you using ?

18.2.4 reef (stable)

Use this section to describe the channel/revision which produces the unexpected behaviour.

I had to forcefully remove a node due to a hardware failure. Afterwards I wanted to join a new node with the previous node name and ip address, unfortunately it run in a timeout. Now I'm getting the following error:
Error: failed to record mon db entries: failed to record mon host: This "config" entry already exists
I then tried to add a new node with a new ip address and name, unfortunately same result.

What are the steps to reproduce this issue ?

1. install new node with 'sudo snap install microceph --channel reef/stable'
2. create token on a cluster node with 'sudo microceph cluster add '
3. join cluster on new node with 'sudo microceph cluster join '
4. Error: failed to record mon db entries: failed to record mon host: This "config" entry already exists

What happens (observed behaviour) ?

Error: failed to record mon db entries: failed to record mon host: This "config" entry already exists
…

What were you expecting to happen ?

new node should have joined the cluster
…

Relevant logs, error output, etc.

If it’s considerably long, please paste to https://gist.github.com/ and insert the link here.

Additional comments.

I've realized that in the ceph.conf entry 'mon host = ' only the ip of the new hosts are present, the ip's of the two currently operating nodes are missing
…

[daniva6]

following timeout error when trying to add a new node: 'Error: Post "http://control.socket/cluster/control": context deadline exceeded'

[nickwales]

Did you get anywhere with this issue? I have the same problem.

[daniva6]

Did you get anywhere with this issue? I have the same problem.

Hi NIck, unfortunately not. If someone knows how update the 'mon host =' statement with the correct ip's, that might solve the problem.

[UtkarshBhatthere]

The mon host = statement is populated using the mon.host.$hostname config entries from MicroCeph's internal DQlite table.
With a little bit of SQL magic you can remove and insert new entries in the table, using which MicroCeph will repopulate the conf file (in a few mins).
See config 3 below and the ceph.conf file.

$ sudo microceph cluster sql "select * from config"
+----+----------------------+------------------------------------------+
| id |         key          |                  value                   |
+----+----------------------+------------------------------------------+
| 1  | fsid                 | a307994e-03ed-4122-9ca3-3bb289af9665     |
| 2  | keyring.client.admin | AQAsCBpnPTElMRAA1CRvvsceRWWdm8f/SByOJw== |
| 3* | mon.host.workbook    | 192.168.29.152                           |
| 4  | public_network       | 192.168.29.152/24                        |
+----+----------------------+------------------------------------------+

$ pwd
/var/snap/microceph/current/conf
$ cat ceph.conf 
# # Generated by MicroCeph, DO NOT EDIT.
[global]
run dir = /var/snap/microceph/current/run
fsid = a307994e-03ed-4122-9ca3-3bb289af9665
mon host = 192.168.29.152
public_network = 192.168.29.152/24
auth allow insecure global id reclaim = false
ms bind ipv4 = true
ms bind ipv6 = false

[UtkarshBhatthere]

@daniva6 @nickwales can you please try the above method (read Hack) and see if that solves it for you ?

[UtkarshBhatthere]

@sabaini #446

[daniva6]

@UtkarshBhatthere
I applied the hack and was able to remove the non-existing nodes.
Unfortunately trying to add a new node results now in a timeout error.
The microceph commands seem to work (cluster list, status, disk list) but the ceph command hangs, and the node can not bring up the osds anymore.

Is there a possibility to extract data from the disks? Or to import an osd in another cluster?

[UtkarshBhatthere]

Hey @daniva6 can you please provide a bit more information 1. sudo microceph cluster sql "select * from config", 2. ceph mon dump, and 3. Hostnames and IP address for member nodes to compare what goes where.

[daniva6]

@UtkarshBhatthere I did set up a new ceph cluster and had to delete the old one for space reasons - this was a good test for my backups :)

[fzhan]

@UtkarshBhatthere same issue here, similar scenario to @daniva6

1. sudo microceph cluster sql "select * from config

+----+----------------------+------------------------------------------+
| id |         key          |                  value                   |
+----+----------------------+------------------------------------------+
| 1  | fsid                 | xxx     |
| 2  | keyring.client.admin | xxx |
| 3  | public_network       | 192.168.x.x/24                         |                                            +----+----------------------+------------------------------------------+

2. ceph mon dump

epoch 15
fsid xxx
last_changed 2024-09-04T09:52:24.830192+1000
created 2024-01-24T21:59:54.363795+1100
min_mon_release 18 (reef)
election_strategy: 1
0: [v2:192.168.x.a:3300/0,v1:192.168.x.a:6789/0] mon.1.x
1: [v2:192.168.x.b:3300/0,v1:192.168.x.b:6789/0] mon.2.x
2: [v2:192.168.x.c:3300/0,v1:192.168.x.c:6789/0] mon.3.x
dumped monmap epoch 15

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CEPH-1172.

This message was autogenerated