microceph.ceph vs ceph native commands : needed clarification

[necarnot]

Issue report

Version of MicroCeph : 18.2.4

What are the steps to reproduce this issue ?

1. Brand new Ubuntu 22.04 VM, fresh install of microceph 18.2.4
2. Following this single-node install tutorial : https://canonical-microceph.readthedocs-hosted.com/en/reef-stable/tutorial/single-node/
3. Added three loop files as disks
4. Kept following the next tutorial : https://canonical-microceph.readthedocs-hosted.com/en/reef-stable/tutorial/mount-block-device/
5. Was happy, typing native ceph command ("ceph osd pool create block_pool", "rbd create bd_foo --size 1024 --image-feature layering -p block_pool", "rbd status -p block_pool", etc...)
6. One week later (thank you COVID), I found out that I can not use native ceph commands anymore, and I have to use microceph.xyz commands ("microceph.ceph", "microceph.rbd", "microceph.rados"...)
7. Trying to use native commands leads to a failure.

What happens (observed behaviour) ?

# ceph status
Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')

My workaround is to provide the conf file path and the keyring :
# ceph --conf /var/snap/microceph/current/conf/ceph.conf -k /var/snap/microceph/current/conf/ceph.keyring status

What were you expecting to happen ?

I was expecting the first behaviour not to change, and be able to use the native commands.

The bash history is clearly showing I was using the native commands seamlessly, and I can't get what could have changed since then.

[necarnot]

See #408

[UtkarshBhatthere]

If you want to use native ceph commands, please do not install the ceph-common package as microceph automatically creates aliases for the ceph commands. (microceph.ceph -> ceph, microceph.rbd -> rbd).

On my laptop where the ceph-common package is NOT installed.

utkarsh@workbook:~$ sudo snap install microceph 
microceph (reef/stable) 18.2.4+snapc9f2b08f92 from Canonical** installed
utkarsh@workbook:~$ sudo microceph cluster bootstrap 
utkarsh@workbook:~$ sudo ceph -s
  cluster:
    id:     a307994e-03ed-4122-9ca3-3bb289af9665
    health: HEALTH_OK
 
  services:
    mon: 1 daemons, quorum workbook (age 7s)
    mgr: workbook(active, since 1.4769s)
    osd: 0 osds: 0 up, 0 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:     

[UtkarshBhatthere]

There is however a catch, you cannot perform rbd map or other such commands because microceph is a confined snap (it does not have raw access to the host kernel). For such operations, (which are mostly performed on the client where microceph snap is anyways not present) you would have to use the ceph-common package.

[UtkarshBhatthere]

marking the issue as closed, @necarnot please feel free to reopen this if this is not resolved.

[necarnot]

If you want to use native ceph commands, please do not install the ceph-common package as microceph automatically creates aliases for the ceph commands. (microceph.ceph -> ceph, microceph.rbd -> rbd).

Hi @UtkarshBhatthere ,
your answer totally makes sense (thank you), as I could not understand how native commands (ie "ceph -s") were working, then stopped working. This must have changed after having installed ceph-common on the same host.
To make things really clear, do I have to understand that microceph installation creates the relevant native commands, but then, the ceph-common installation is breaking them?

So far, this is exactly what I have witnessed.

Not a big issue, but if proved, that would be worth mentioning in the documentation.