Introduction

This article focuses on CephX Certification, an important system in Ceph.The naming format of CephX is briefly introduced.The role of CephX in the process from cluster startup to user connection cluster is also described.Finally, the experiment explains how to fully recover all the secret keys of a cluster without losing them, and some considerations for using CephX in a real production environment.

What is CephX?

CephX understands simply that it is the username/password of the whole Ceph system. This user does not refer to only the client that we typed ceph-s on the terminal. In this authentication system, there is also a special user group, MON/OSD/MDS. In other words, Monitor, OSD and MDS also need a pair of account passwords to log on to the Ceph system.

Naming rules for CephX

User names/passwords follow certain naming rules:
User name:
User names generally follow the naming rules of <TYPE. ID>, where there are three types of TYPE: mon,osd,client.IDs vary according to different types of users:
mon:ID is empty.
OSD:ID is the ID of the OSD.
The client:ID is the name of the client, such as admin,cinder,nova.
Password:
The password is usually a 40-character string, such as: AQBh1XlZAAAAABAAcVaBh1p8w4Q3oaGoPW0R8w==

Default user

To interact with a Ceph cluster, we usually need to know at least four pieces of information, which is essential:
1. fsid of cluster.
2. The IP address of the onitor of the cluster must be connected to the MON before the cluster information can be obtained.
3. A user name for login.
4. Password corresponding to the logged-in user.
In fact, many students will find that ceph-s can be executed to get the state of the cluster without specifying these parameters when we interact with the CEPH cluster on a daily basis.In fact, we have used several default parameters provided by Ceph, and the full name of ceph-s with default parameters is:
ceph -s --conf /etc/ceph/ceph.conf --name client.admin --keyring /etc/ceph/ceph.client.admin.keyring
As you can see from the instructions above, the default user Ceph uses is client.admin, and the user's key file is usually saved in the path / etc/ceph/ceph.client.admin.keyring.If here, we delete the key file from the / etc/ceph directory and execute ceph-s again, you will get the most common error below:

2017-07-28 15:56:03.271139 7f142579c700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
2017-07-28 15:56:03.271145 7f142579c700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication
2017-07-28 15:56:03.271146 7f142579c700 0 librados: client.admin initialization error (2) No such file or directory
Error connecting to cluster: ObjectNotFound

As we can see from the error message, because we used the default user client.admin, Ceph would look for the password of the client.admin user in the following four default paths:

/etc/ceph/ceph.client.admin.keyring :  The actual naming format is: /etc/ceph/<$cluster>.<$type>.<$id>.keyring. 
/etc/ceph/ceph.keyring : Naming format is:/etc/ceph/<$cluster>.keyring. 
/etc/ceph/keyring. 
/etc/ceph/keyring.bin. 

If these four files do not exist, or if the user's client.admin key is not saved in either of them, an error will occur: ERROR: missing keyring In other words, the user's client.admin failed to log on to the Ceph system!

Who was the ancestor of CephX?

Hide the mon of Boss

A conversation:
client.admin: Of course it is me!Any command can be executed after logging into Ceph with my account password!
mon.: oh.
client.admin: Who are you?I have all my account passwords (quietly checked the ceph auth list), why can't I see you?
mon.: Well.
client.admin: Oops, which second-class deleted my secret key file, I can't connect to the cluster anymore!
mon.: Let me help you find your secret key.
client.admin: You?Sure?
mon.: Well.

Client.admin, who thought he had a lot of privileges, suddenly lost the secret key file to save the password and could not access the cluster.And mon. who never appeared, claims to be able to retrieve the secret key of client.admin. Is mon. the real ancestor?
Now, back to the beginning of the story, when the cluster was built, we used ceph-deploy new NodeA NodeB NodeC to generate three files:

ceph.conf
ceph.mon.keyring
ceph-deploy-ceph.log

In addition to ceph.conf, a ceph.mon.keyring file is generated by default, which is unlikely to be used in subsequent cluster interactions because after ceph-deploy mon create-initial, a client.admin user is generated, and subsequent interactions generally use this user.However, the first user generated by the cluster is <mon. >, and the corresponding key file is saved in the deployment directory as ceph.mon.keyring.
Looking at the LOG of ceph-deloy, you can see that when you step ceph-deploy mon create-initial, a log is recorded as follows:

[2017-07-28 16:49:53,468][centos7][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.centos7.asok mon_status
[2017-07-28 16:49:53,557][centos7][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-centos7/keyring auth get client.admin
[2017-07-28 16:49:53,761][centos7][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-centos7/keyring auth get client.bootstrap-mds
[2017-07-28 16:49:54,046][centos7][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-centos7/keyring auth get client.bootstrap-mgr
[2017-07-28 16:49:54,255][centos7][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-centos7/keyring auth get-or-create client.bootstrap-mgr mon allow profile bootstrap-mgr
[2017-07-28 16:49:54,452][centos7][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-centos7/keyring auth get client.bootstrap-osd
[2017-07-28 16:49:54,658][centos7][INFO ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-centos7/keyring auth get client.bootstrap-rgw

In fact, these key files were created by user mon. including the client.admin user and his key.So throughout CephX's history, mon. was the first user to be generated, while all other users were generated by mon. users, or in turn, down.The bootstrap is used to guide the generation of users of the corresponding class, such as bootstrap-osd, which is used to guide the generation of all osd.N users.

CephX usage scenarios

After talking about the generation time of each user, let's see when they used their account and password!

MON

When the entire cluster starts, Monitor starts first, then OSD starts.When Monitor starts, Monitor will start the process with its own secret key file. That is, when Monitor starts, it does not need to authenticate the secret key to any process. Generally speaking, even if the secret key of Monitor is modified, it will not affect the start of Monitor. This is illustrated by a small experiment:

[root@blog cluster]# cat /var/lib/ceph/mon/ceph-blog/keyring
[mon.]
key = AQAr1H1ZAAAAABAAWItgjm4dOKPJx+FX9lVk4Q==
caps mon = "allow *"
[root@blog cluster]# ceph auth get mon.
exported keyring for mon.
[mon.]
key = AQAr1H1ZAAAAABAAWItgjm4dOKPJx+FX9lVk4Q==
caps mon = "allow *"
[root@blog cluster]# vim /var/lib/ceph/mon/ceph-blog/keyring
##### Change part A of the contents of the secret key file to B and restart Monitor
[root@blog cluster]# cat /var/lib/ceph/mon/ceph-blog/keyring
[mon.]
key = AQAr1H1ZBBBBBBAAWItgjm4dOKPJx+FX9lVk4Q==
caps mon = "allow *"
[root@blog cluster]# systemctl restart ceph-mon.target
[root@blog cluster]# ceph auth get mon.
exported keyring for mon.
[mon.]
key = AQAr1H1ZBBBBBBAAWItgjm4dOKPJx+FX9lVk4Q==
caps mon = "allow *"

Monitor's database records all user passwords except mon. Authentication is only really started after Monitor starts. All subsequent users who want to connect to the cluster must first connect to the Ceph cluster through fsid and MON IP. After the authentication, they can access the cluster normally. Here's how to proceed with OSD Startup AuthenticationCheck

OSD

When OSD starts, it first needs log_to_monitors, which is to log on to the cluster with its own account password, which is recorded in the Monitor database, so if they match, OSD can start normally, otherwise, the following error will be reported:

2017-08-01 16:54:51.515541 7f21ea978800 -1 osd.0 30 log_to_monitors {default=true}
2017-08-01 16:54:51.991263 7f21ea978800 1 journal close /var/lib/ceph/osd/ceph-0/journal
2017-08-01 16:54:51.999674 7f21ea978800 -1 ^[[0;31m ** ERROR: osd init failed: (1) Operation not permitted^[[0m
2017-08-01 16:54:52.006620 7f21ea978800 -1 common/HeartbeatMap.cc: In function 'ceph::HeartbeatMap::~HeartbeatMap()' thread 7f21ea978800 time 2017-08-01 16:54:52.001569
common/HeartbeatMap.cc: 44: FAILED assert(m_workers.empty())

Operation not permitted in the log means authentication failed, indicating that the key file carried by osd.0 and the contents recorded by Monitor are inconsistent, causing the OSD to fail to start.At this point, you need to compare the key contents of osd.0 with those of Monitor:

[root@blog ~]# cat /var/lib/ceph/osd/ceph-0/keyring
[osd.0]
key = AQA81H5Zh05jDxAAkvaHBs07K9HYF1uGSPh+rA==
[root@blog ~]# ceph auth get osd.0
exported keyring for osd.0
[osd.0]
key = AQBH1H5Z6pBvDhAAul364EZsRjDy/NqTQh07Yw==
caps mon = "allow profile osd"
caps osd = "allow *"

Indeed, the content is different, at this time, replacing the key part of the key file keyring with the key value obtained by ceph auth get osd.0 will start the OSD normally.

Client

Usually when we execute ceph-s, it is equivalent to opening a client to connect to the Ceph cluster, which by default logs on to the connection cluster using the account password of client.admin, so ceph-s executed normally is equivalent to executing ceph-s --name client.admin --keyring /etc/ceph/ceph.client.admin.keyring.It is important to note that each time we execute Ceph's instructions on the command line, it is equivalent to opening a client, interacting with the cluster, and then closing the client.

Now let's make a very common mistake, which is easy to encounter when you're new to Ceph:

[root@blog ~]# ceph -s
2017-08-03 02:22:27.352516 7fbd157b7700 0 librados: client.admin authentication error (1) Operation not permitted
Error connecting to cluster: PermissionError

The error message is understandable, the operation is not allowed, that is, the authentication failed, because here we are using the default client.admin user and its secret key, indicating that the key content is inconsistent with the Eph cluster record, that is, /etc/Ceph/ceph.client.admin.keyringThe content is most likely left over by a previous cluster or the wrong key is logged, in which case the correct key content can be seen by simply executing the ceph auth list with mon.user:

[root@blog ~]# ceph auth get client.admin --name mon. --keyring /var/lib/ceph/mon/ceph-blog/keyring
exported keyring for client.admin
[client.admin]
key = AQD7F4JZIs9DJxAAZms/NQQQ1YhUpCHRtjygJA==
caps mds = "allow *"
caps mon = "allow *"
caps osd = "allow *"

Detailed caps

Looking closely at the output of the ceph auth list, in addition to the username and the corresponding key content, there is one that starts with caps, which is the CephX subdivision of the permissions of individual users, such as: read, write, execute, and so on:

caps mds = "allow *"
caps mon = "allow *"
caps osd = "allow *"

For different applications (mds/mon/osd), the same read or write permissions have different roles. The r/w/x permissions of these three applications are analyzed in turn below.

Introduction to MON privileges

r Permissions

So the question is, what is the minimum permission to execute ceph-s?That is caps mon = allow r, which is the R permission of the MON.So what exactly did this permission read?The first thing to emphasize is that the data read here comes from MON, not OSD.

A MON, as the state maintainer of a cluster, maintains a cluster set of state diagrams (Cluster Map s) in its database (/var/lib/ceph/mon/ceph-$hostname/store.db), which contain, but are not limited to:
CRUSH Map
OSD Map
MON Map
MDS Map
PG Map
The read permissions here are the permissions to read these Maps, but the real content of these Maps is not meaningful to read, so they are shown in a more friendly output of instructions that contain, but are not limited to:

ceph -s
ceph osd crush dump
ceph osd tree
ceph pg dump
ceph osd dump

As long as you have r access to the MON, you can read all the MON-maintained Map data from the cluster. Macroscopically, you can read the state information of the cluster (but you cannot modify it).
Here is a brief introduction to the validation process. We access the cluster by generating a secret key that only contains the r privilege of the MON:

ceph auth get-or-create client.mon_r mon 'allow r' >> /root/key
ceph --name client.mon_r --keyring /root/key -s ## OK
ceph --name client.mon_r --keyring /root/key osd dump ## OK
ceph --name client.mon_r --keyring /root/key pg dump ## OK
ceph --name client.mon_r --keyring /root/key osd out 0 ## Error EACCES: access denied
ceph --name client.mon_r --keyring /root/key osd set noout ## Error EACCES: access denied

w Permissions

The w privilege is interesting and must be combined with the r privilege to be valid, otherwise, when a single w privilege executes an instruction, it will always access denied.So when we test w privileges, we need to attach r privileges:

ceph auth get-or-create client.mon_rw mon 'allow rw' >> /root/key

At this point, you can clearly read the status of each OSD and each PG as if each Map of the fake cluster were in front of you, but once you have been granted w privileges, you can operate on these entities, such as kicking an OSD (ceph osd rm), repairing a PG (ceph pg repair), modifying the CRUSH structure (ceph osd setcrushmap), and deleting a MON (c)Eph mon rm), and these operations, if only r privileges, cannot be performed (access denied).
Since there are so many instructions that can be executed here, I'll just make a brief summary of the r, w permissions:
r: Reads the state of each component of the cluster (MON/OSD/MDS/CRUSH/PG), but cannot be modified.
rw: Reads and modifies the state of each component of the cluster, and executes various action instructions for the component.

Be careful:
None of the components currently discussed have read and write permissions for cluster objects, that is, you cannot read and write objects from a cluster without rw permissions for MON s alone.

x Permissions

The x privilege for MON s is also interesting because it is only relevant for auth.That is, if you want to execute all auth-related directives such as ceph auth list, ceph auth get, you can execute them with X privileges.However, similar to w privileges, r privileges need to be combined to be effective:

ceph auth get-or-create client.mon_rx mon 'allow rx' >> /root/key
ceph --name client.mon_rw --keyring /root/key auth list

*Permissions

One sentence description: * = rwx
Tips
By writing multiple user keys to the same file, such as / root/key above, you can include:

> [root@blog ~]# cat /root/key
> [client.mon_r]
>	key=AQBtLIJZScuLARAAPQS9ahvU1oZh1Ha/fYhUhQ==
> [client.mon_w]
> key = AQD5LYJZJO2/AxAAWgbuPPUNJ0ugxsqdDD3+sw==
>

When the directive specifies--name, Ceph goes to the file after--keyring to find the key under the corresponding user name section.

profile osd permissions

The official explanation is to give users the right to connect to other OSDs/MONs as one OSD.Grant this privilege to the OSD so that it can handle replica heartbeats and status reports.No more general understanding found yet~

OSD

Compared to the various permissions of MON, the r W of OSD simply understands that the r W permission is the permission to read an object, the w w permission is the permission to write an object, and the x permission is more interesting. You can call any class-read/class-write permission. Here we refer to a passage on Ceph CookBook to introduce class-read/class-write:

Ceph can be extended by creating shared object classes called Ceph Classes. Ceph can load .so classes stored in the OSD class dir.For a class ,you can create new object methods that have the ability to call native methods in Ceph object store ,for example , the objects that you have defined in your class can call native Ceph methods such as read and write.

For example, you can implement custom methods that allow you to read and write objects of a certain type, such as those that start with rbd_data.At present, only the librados layer can be called to use the custom class method, while the upper RBD, RGW, and so on cannot be used.

* Permissions include permissions for admin directives such as ceph tell osd. * in addition to rwx.
This page on the official website (http://docs.ceph.com/docs/kraken/man/8/ceph-authtool/) provides a good introduction to a few examples. Here's a brief description of what the next longer instruction means:

osd = "allow class-read object_prefix rbd_children, allow pool templates r class-read, allow pool vms rwx"

First privilege: object_prefix is a class method that gives read privileges to all objects starting with rbd_children.Read only, not write.
Second permission: Give the pool templates read permission to execute the class-read method.That is, in addition to being able to read objects from this pool, customers can also implement their own system-owned class methods such as obejct_prefix to read objects, such as those with a certain type of characteristics.
Third privilege: Give the pool vms read and write and execute class-read/class-write methods.

Recovery process for losing all keys

With such a lot of theoretical knowledge, one of the second-class Freight Maintenance students was impatient and deleted all the secret keys in one breath. These secret keys contain:

MON : /var/lib/ceph/mon/ceph-hostname/keyring
OSD : /var/lib/ceph/osd/ceph-0/keyring
Client : /etc/ceph/ceph.client.admin.keyring
In summary, all files containing the contents of the secret key have been deleted, and with / etc/ceph/directory deleted cleanly.Can you recover all the secret key files at this time?The answer is: Yes!

Ceph makes an interesting setting for managing the secret key: all account passwords except mon. are stored in the MON database leveldb, but the mon. user's information is not stored in the database, but is obtained by reading the Keyring file in the MON directory at MON startup.So we can even fake a keyring and put it in the MON directory.Then synchronize to each MON node and restart the three MONs.At this point, the MON takes the artificial Keyring and starts working.With the account password of the mon.user, it is easy to get all the key content using the CEPH auth list --name mon. --keyring/var/lib/ceph/mon/ceph-$hostname/keyring directive!

Wait, if you really delete the / etc/ceph/directory, this command above cannot be executed because there is no / etc/ceph/ceph.conf to specify the cluster, then we can get the FSID of the cluster from / var/lib/ceph/osd/ceph-0/ceph_fsid under any OSD directory, and the IP information of the MON is easy to recover.With a simple refactoring of ceph.conf, you can get the correct ceph auth list information by using mon.

Wait a second time. The content of ceph.conf is too concise now. There are many fewer East things than before. Can they be restored?The answer is yes!

Find any OSD that has not been restarted and execute ceph daemon osd.0 config diff so that you can see the difference between the configuration loaded by this OSD boot and the default configuration, and easily restore to the same configuration file as before deletion by careful comparison ~.

Doubt:
I believe there will be students'doubts here. Why not directly restart the cluster after changing cephx in ceph.conf to none? First, it is more costly because OSD restarts (OSD must be restarted, otherwise OSD will commit suicide completely within a period of time when MON restarts).So the above method only restarts the MON and has a small impact.

cephx -> none

When turning off CephX functionality, there is a certain order to follow:
Shutdown: Restart MON ->Restart OSD
On: Restart MON ->Restart OSD
If you do not restart the OSD after turning off CephX, the OSD will hang randomly over time.

Distribution of Keyrings

Usually we can execute ceph-s on the MON node, but when we reach the OSD node, we cannot execute ceph-s because of the missing key file for client.admin.

The secret key save path for MON is: /var/lib/ceph/mon/ceph-hostname/keyring.The secret key save path for OSD is: /var/lib/ceph/osd/ceph_hostname/keyring. The secret key save path for OSD is: /var/lib/ceph/osd/ceph-hostname/keyring.The secret key save path for OSD is: /var/lib/ceph/osd/ceph_osd_id/keyring.
The path to save the key for client.admin is: /etc/ceph/ceph.client.admin.keyring.

In fact, we can even execute on the OSD node (without client.admin.keyring):

ceph -s --name osd.0 --keyring /var/lib/ceph/osd/ceph-0/keyring

To read the state of the cluster.

Configure user permissions that can only access one RBD

Refer to this article for details (https://blog-fromsomedude.rhcloud.com/2016/04/26/Allowing-a-RBD-client-to-map-only-one-RBD/)

Here is the meaning of the creation instructions for secret keys:

ceph auth get-or-create client.myclient
mon
'allow r'
osd
'allow rwx object_prefix rbd_data.103d2ae8944a;
allow rwx object_prefix rbd_header.103d2ae8944a;
allow rx object_prefix rbd_id.myimage'
-o /etc/ceph/ceph.client.myclient.keyring

Here, the object_prefix prefix method gives the user read and write permissions to all related objects of an RBD, which is a more practical application.

summary

This paper describes the generation of CephX in Ceph systems and the dependencies between them.The roles and usage scenarios of each privilege in different applications are described in detail.Finally, we apply the theory to the actual production environment through the example of secret key loss, so that everyone can use CPHX more easily.

Cluster Key Information

mds.node1
	key: AQCdYkFdDqGoLxAAN39t2Vxm0jjtpoMalj3g3g==
	caps: [mds] allow
	caps: [mon] allow profile mds
	caps: [osd] allow rwx
osd.99
	key: AQDosD5dJTyOGxAACSy+2GP8I/VMYe6wGxVVBA==
	caps: [mgr] allow profile osd
	caps: [mon] allow profile osd
	caps: [osd] allow *
client.admin
	key: AQDZrz5dkVL3LBAAdxgaudAL0tSSE9kec+RibQ==
	caps: [mds] allow *
	caps: [mgr] allow *
	caps: [mon] allow *
	caps: [osd] allow *
client.bootstrap-mds
	key: AQDarz5dU826EhAA7UCgJwyTQ35GLc9QV1G+uQ==
	caps: [mon] allow profile bootstrap-mds
client.bootstrap-mgr
	key: AQDarz5d38LaNBAAyL6sG9LnZIjBw/8ssK4lvg==
	caps: [mon] allow profile bootstrap-mgr
client.bootstrap-osd
	key: AQDbrz5dB6MbGxAAPsKr1AncwwG4hEehOfm+yg==
	caps: [mon] allow profile bootstrap-osd
client.bootstrap-rgw
	key: AQDcrz5dNdYxABAAu1lZZMFm+vZEsQ2jw5DnrA==
	caps: [mon] allow profile bootstrap-rgw
client.rbdadmin
	key: AQAAvz5dVCTsMxAAvszhfo3lX+NSCBNdUBt/Wg==
	caps: [mds] allow *
	caps: [mgr] allow *
	caps: [mon] allow *
	caps: [osd] allow *
client.test.node9
	key: AQCzYkFdsd71EhAA2ihWSJ8Y2HApICA7REzLHA==
	caps: [mds] allow *
	caps: [mgr] allow *
	caps: [mon] allow *
	caps: [osd] allow *
mgr.node4
	key: AQDirz5dKPDbMBAADI7T3NeWP0+3CuIeGY3s7g==
	caps: [mds] allow *
	caps: [mon] allow profile mgr
	caps: [osd] allow *