Contents

1 K8s使用外部Ceph集群(配置ceph-csi)

K8s使用外部Ceph集群(配置ceph-csi)

参考: https://juejin.cn/post/7296756504912330767

参考: https://github.com/ceph/ceph-csi?tab=readme-ov-file#overview

参考: https://github.com/ceph/ceph-csi/blob/devel/docs/deploy-rbd.md

参考: https://github.com/ceph/ceph-csi/blob/devel/docs/deploy-rbd.md#deployment-with-kubernetes

参考: https://github.com/ceph/ceph-csi/tree/devel/deploy/rbd/kubernetes

参考: https://www.cnblogs.com/hukey/p/17828946.html

参考: https://juejin.cn/post/7296756504912330767

参考: docs.ceph.com/en/latest/r…

参考: juejin.cn/post/712535…

K8S介绍

略

存储介绍

参考: https://www.cnblogs.com/orange-CC/p/12657920.html

一般的存储服务, 可以提供的存储类型主要有三种:

块存储: 以磁盘块方式提供. 块设备. 只允许一个节点挂载, IO效率最高. (ceph中是 rbd类型)

文件存储: 以文件系统(卷)的方式提供. 文件系统设备. 允许多节点挂载. 多写可以保证顺序性. (ceph中是cephfs)

对象存储: 一般是亚马逊 s3 对象存储标准, 每个文件存储为一个对象, 提供链接地址供外部访问. (ceph中是 cephrgw)

架构图

架构1: 以k8s为上层分布式编排底座.下层是 ceph rbd (块存储) 分布式存储底座为例.

架构2: k8s为上层分布式编排底座.下层是 ceph cephfs (文件存储) 分布式存储底座类似, 不再缀图.

k8s+rbd (rbd块存储方式)

k8s集群假设已经建立, 但是还没有加载csi (container storage interface) 驱动. ceph-csi是ceph作为其csi的一种实现.

参考: https://jimmysong.io/kubernetes-handbook/concepts/csi.html

参考: https://kubernetes.feisky.xyz/extension/volume/csi

参考: https://kubernetes.io/zh-cn/docs/reference/kubernetes-api/config-and-storage-resources/csi-driver-v1/

参考: https://kubernetes.io/zh-cn/docs/concepts/storage/storage-classes/

参考: https://github.com/ceph/ceph-csi

在ceph上创建存储池

root@wangjm-B550M-K-1:~# ceph osd pool create kubernetes
pool 'kubernetes' created

新创建的存储池在使用之前需要初始化，使用rbd工具进行初始化

rbd pool init kubernetes

设置客户端认证

为kubernetes和ceph-csi创建一个新用户

root@wangjm-B550M-K-1:~/k8s/csi/ceph/rbd# ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes'
[client.kubernetes]
        key = AQCdXTtm88bhMRAAvsaRX1ppP70XZojm0fXRdg==

查看集群ID

ceph -s的输出里面有集群的id信息

root@wangjm-B550M-K-1:~/k8s/csi/ceph/rbd# ceph -s
  cluster:
    id:     92046bac-05dd-11ef-979f-572db13abde1
    health: HEALTH_OK
 
  services:
    mon: 4 daemons, quorum wangjm-B550M-K-1,wangjm-B550M-K-2,wangjm-B550M-K-3,jingmin-kube-master1 (age 24h)
    mgr: jingmin-kube-master1.agkrmd(active, since 44h), standbys: wangjm-B550M-K-1.uhkxdb
    osd: 3 osds: 3 up (since 23h), 3 in (since 24h)
    rgw: 2 daemons active (1 hosts, 1 zones)
 
  data:
    pools:   9 pools, 226 pgs
    objects: 262 objects, 7.7 MiB
    usage:   994 MiB used, 714 GiB / 715 GiB avail
    pgs:     226 active+clean

新建k8s命名空间, 调整默认命名空间

kubectl create ns ceph-rbd

kubectl config set-context --current --namespace ceph-rbd

创建ceph-csi使用的configMap

ceph-csi需要一个存储在k8s中的ConfigMap，这个ConfigMap中保存了ceph集群的fsid和monitors的地址

# csi-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    [
      {
        "clusterID": "92046bac-05dd-11ef-979f-572db13abde1",
        "monitors": [
          "192.168.1.1:6789",
          "192.168.1.8:6789",
          "192.168.1.9:6789",
          "192.168.1.10:6789"
        ]
      }
    ]
metadata:
  name: ceph-csi-config

应用yaml文件

root@wangjm-B550M-K-1:~/k8s/csi/ceph/rbd# kubectl apply -f csi-config-map.yaml 
configmap/ceph-csi-config created

创建KMS provider所使用的ConfigMap文件

最新版本的ceph-csi还需要一个额外的ConfigMap对象来定义密钥管理服务(KMS)提供程序的详细信息。如果没有设置KMS，在csi-kms-config-map中放置一个空配置即可

# csi-kms-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    {}
metadata:
  name: ceph-csi-encryption-kms-config

应用yaml文件

root@wangjm-B550M-K-1:~/k8s/csi/ceph/rbd# kubectl apply -f csi-kms-config-map.yaml 
configmap/ceph-csi-encryption-kms-config created

创建一个保存Ceph配置的ConfigMap

最新版本的Ceph-CSI还需要另一个ConfigMap对象来定义Ceph配置，以便添加到CSI容器内的ceph.conf文件中

# ceph-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
  ceph.conf: |
    [global]
    auth_cluster_required = cephx
    auth_service_required = cephx
    auth_client_required = cephx
  # keyring is a required key and its value should be empty
  keyring: |
metadata:
  name: ceph-config

应用yaml文件

root@wangjm-B550M-K-1:~/k8s/csi/ceph/rbd# kubectl apply -f ceph-config-map.yaml
configmap/ceph-config created

创建ceph-csi cephx secret

Ceph-csi需要cephx凭据才能与Ceph集群通信

创建yaml文件

# csi-rbd-secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: ceph-rbd
stringData:
  userID: kubernetes
  userKey: AQCdXTtm88bhMRAAvsaRX1ppP70XZojm0fXRdg==

应用yaml文件

root@wangjm-B550M-K-1:~/k8s/csi/ceph/rbd# kubectl apply -f csi-rbd-secret.yaml
secret/csi-rbd-secret created

配置ceph-csi plugins

创建csi driver

wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csidriver.yaml

kubectl apply -f csidriver.yaml

创建需要的ServiceAccount和RBAC ClusterRole/ClusterRoleBinding


wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml
wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml

vim csi-provisioner-rbac.yaml
:%s/namespace\: default/namespace\: ceph-rbd/g

vim csi-nodeplugin-rbac.yaml
:%s/namespace\: default/namespace\: ceph-rbd/g

kubectl create -f csi-provisioner-rbac.yaml
kubectl create -f csi-nodeplugin-rbac.yaml

创建ceph-csi provisioner 和node plugins

wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml
wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml

vim csi-rbdplugin-provisioner.yaml
:%s/namespace\: default/namespace\: ceph-rbd/g

vim csi-rbdplugin.yaml
:%s/namespace\: default/namespace\: ceph-rbd/g

kubectl apply -f csi-rbdplugin-provisioner.yaml
kubectl apply -f csi-rbdplugin.yaml

由于yaml文件里面的镜像是国外的地址源，因此这里改成自己的镜像地址

（或者也可以配置containerd/cri-o代理，来下载国外镜像）

如果设置过 containerd/cri-o/docker 的代理, 或者国际网络通畅的话, 可以略过.

查看yaml文件里面所使用的镜像

[root@node1 ceph-csi]# grep 'image:'  csi-rbdplugin-provisioner.yaml 
          image: registry.k8s.io/sig-storage/csi-provisioner:v3.6.0
          image: registry.k8s.io/sig-storage/csi-snapshotter:v6.3.0
          image: registry.k8s.io/sig-storage/csi-attacher:v4.4.0
          image: gcr.io/k8s-staging-sig-storage/csi-resizer:v1.9.0
          image: quay.io/cephcsi/cephcsi:canary
          image: quay.io/cephcsi/cephcsi:canary
          image: quay.io/cephcsi/cephcsi:canary
          
[root@node1 ceph-csi]# grep 'image:'  csi-rbdplugin.yaml 
          image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.0
          image: quay.io/cephcsi/cephcsi:canary
          image: quay.io/cephcsi/cephcsi:canary

将上面文件中的镜像替换成aliyun中保存的镜像

sed -i s#registry.k8s.io/sig-storage/csi-provisioner:v3.6.0#registry.cn-hangzhou.aliyuncs.com/postkarte/csi-provisioner:v3.6.0#g csi-rbdplugin-provisioner.yaml

sed -i s#registry.k8s.io/sig-storage/csi-snapshotter:v6.3.0#registry.cn-hangzhou.aliyuncs.com/postkarte/csi-snapshotter:v6.3.0#g csi-rbdplugin-provisioner.yaml

sed -i s#registry.k8s.io/sig-storage/csi-attacher:v4.4.0#registry.cn-hangzhou.aliyuncs.com/postkarte/csi-attacher:v4.4.0#g  csi-rbdplugin-provisioner.yaml

sed -i s#gcr.io/k8s-staging-sig-storage/csi-resizer:v1.9.0#registry.cn-hangzhou.aliyuncs.com/postkarte/csi-resizer:v1.9.0#g csi-rbdplugin-provisioner.yaml

sed -i s#quay.io/cephcsi/cephcsi:canary#registry.cn-hangzhou.aliyuncs.com/postkarte/cephcsi:canary#g csi-rbdplugin-provisioner.yaml

sed -i s#registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.0#registry.cn-hangzhou.aliyuncs.com/postkarte/csi-node-driver-registrar:v2.9.0#g csi-rbdplugin.yaml 

sed -i s#quay.io/cephcsi/cephcsi:canary#registry.cn-hangzhou.aliyuncs.com/postkarte/cephcsi:canary#g csi-rbdplugin.yaml

应用yaml文件

kubectl apply -f csi-rbdplugin-provisioner.yaml 
kubectl apply -f csi-rbdplugin.yaml

创建storageclass

参考: https://github.com/ceph/ceph-csi/blob/devel/examples/rbd/storageclass.yaml

# csi-rbd-sc.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: 92046bac-05dd-11ef-979f-572db13abde1
   pool: kubernetes
   imageFeatures: layering
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: ceph-rbd
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: ceph-rbd
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: ceph-rbd
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
   - discard

应用yaml文件

kubectl apply -f csi-rbd-sc.yaml 
storageclass.storage.k8s.io/csi-rbd-sc created

设为默认storageclass

kubectl patch storageclass csi-rbd-sc -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

查看sc

root@wangjm-B550M-K-1:~/k8s/csi/ceph/rbd# kubectl get sc 
NAME                   PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
csi-rbd-sc (default)   rbd.csi.ceph.com   Delete          Immediate           true                   2m57s

部署应用测试

测试1

(这个是别人的测试, 看起来有点复杂, 没实际去测试)

这里部署一个gitlab进行测试部署参考链接

测试2

这里部署一个mysql数据库测试

新建命名空间, 重设默认命名空间

root@wangjm-B550M-K-1:~/k8s/test/mysql# kubectl create ns test
namespace/test created
root@wangjm-B550M-K-1:~/k8s/test/mysql# kubectl config set-context --current --namespace test
Context "kubernetes-admin@kubernetes" modified.

参考: https://kubernetes.io/zh-cn/docs/tasks/run-application/run-single-instance-stateful-application/

mysql.yaml

root@wangjm-B550M-K-1:~/k8s/test/mysql# cat mysql.yaml 
apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
  - port: 3306
  selector:
    app: mysql
  clusterIP: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
          # 在实际中使用 secret
        - name: MYSQL_ROOT_PASSWORD
          value: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
spec:
  storageClassName: csi-rbd-sc
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

应用

kubectl apply -f ./mysql.yaml

查看状态

root@wangjm-B550M-K-1:~/k8s/test/mysql# kubectl get all
NAME                         READY   STATUS    RESTARTS   AGE
pod/mysql-6666d46f58-x4jlg   1/1     Running   0          3m2s

NAME            TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/mysql   ClusterIP   None         <none>        3306/TCP   3m2s

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mysql   1/1     1            1           3m2s

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/mysql-6666d46f58   1         1         1       3m2s

查看ceph dashboard -> Block -> Images, 已经出现一个新的image (name是 csi打头, 所属pool是kubernetes)

配置ceph中帐号,供cephfs使用

创建名为“cephfs” 卷(文件系统)

参考： https://www.cnblogs.com/hukey/p/17828946.html

参考： https://juejin.cn/post/7155656346048659493

参考： https://docs.ceph.com/en/latest/cephfs/

方式一：创建文件系统并自动分配pool

# Create a CephFS volume named (for example) "cephfs":
ceph fs volume create cephfs

实际上，还会自动创建cephfs.cephfs.data和cephfs.cephfs.metal两个pool.

如果是第一次创建cephfs文件系统，还会自动创建mds服务。

root@wangjm-B550M-K-1:~# ceph -s
  cluster:
    id:     92046bac-05dd-11ef-979f-572db13abde1
    health: HEALTH_OK
 
  services:
    mon: 4 daemons, quorum wangjm-B550M-K-1,wangjm-B550M-K-2,wangjm-B550M-K-3,jingmin-kube-master1 (age 7h)
    mgr: jingmin-kube-master1.agkrmd(active, since 7h), standbys: wangjm-B550M-K-1.uhkxdb
    osd: 3 osds: 3 up (since 4d), 3 in (since 4d)
    rgw: 2 daemons active (1 hosts, 1 zones)
 
  data:
    pools:   9 pools, 226 pgs
    objects: 6.14k objects, 22 GiB
    usage:   69 GiB used, 647 GiB / 715 GiB avail
    pgs:     226 active+clean
 
  io:
    client:   3.0 MiB/s rd, 49 KiB/s wr, 24 op/s rd, 7 op/s wr
 
root@wangjm-B550M-K-1:~# ceph fs volume create cephfs
root@wangjm-B550M-K-1:~# ceph -s
  cluster:
    id:     92046bac-05dd-11ef-979f-572db13abde1
    health: HEALTH_OK
 
  services:
    mon: 4 daemons, quorum wangjm-B550M-K-1,wangjm-B550M-K-2,wangjm-B550M-K-3,jingmin-kube-master1 (age 7h)
    mgr: jingmin-kube-master1.agkrmd(active, since 7h), standbys: wangjm-B550M-K-1.uhkxdb
    mds: 1/1 daemons up, 1 standby
    osd: 3 osds: 3 up (since 4d), 3 in (since 4d)
    rgw: 2 daemons active (1 hosts, 1 zones)
 
  data:
    volumes: 1/1 healthy
    pools:   11 pools, 243 pgs
    objects: 6.16k objects, 22 GiB
    usage:   69 GiB used, 647 GiB / 715 GiB avail
    pgs:     243 active+clean
 
  io:
    client:   4.1 MiB/s rd, 49 KiB/s wr, 33 op/s rd, 7 op/s wr

可以看到 services中多了mds服务。

root@wangjm-B550M-K-1:~# ceph fs ls
name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ]
root@wangjm-B550M-K-1:~# ceph fs volume info cephfs
{
    "mon_addrs": [
        "192.168.1.8:6789",
        "192.168.1.9:6789",
        "192.168.1.10:6789",
        "192.168.1.1:6789"
    ],
    "pools": {
        "data": [
            {
                "avail": 218418446336,
                "name": "cephfs.cephfs.data",
                "used": 0
            }
        ],
        "metadata": [
            {
                "avail": 218418446336,
                "name": "cephfs.cephfs.meta",
                "used": 98304
            }
        ]
    }
}

可以看到这个名为cephfs的cephfs类型的文件系统，对应的存储pool信息。

方式二：手动创建pool和并关联文件系统(未使用)

参考： https://docs.ceph.com/en/latest/cephfs/createfs/#

参考： https://juejin.cn/post/7155656346048659493

目前没采用这种方式, 略.

配置客户端权限

参考： https://docs.ceph.com/en/latest/cephfs/client-auth/

参考： https://docs.ceph.com/en/latest/cephfs/mount-prerequisites/

Before mounting CephFS, ensure that the client host (where CephFS has to be mounted and used) has a copy of the Ceph configuration file (i.e. ceph.conf) and a keyring of the CephX user that has permission to access the MDS. Both of these files must already be present on the host where the Ceph MON resides.

Generate a minimal conf file for the client host and place it at a standard location:
```
# on client host
mkdir -p -m 755 /etc/ceph
ssh {user}@{mon-host} "sudo ceph config generate-minimal-conf" | sudo tee /etc/ceph/ceph.conf
```
Alternatively, you may copy the conf file. But the above method generates a conf with minimal details which is usually sufficient. For more information, see Client Authentication and Bootstrap options.
Ensure that the conf has appropriate permissions:
```
chmod 644 /etc/ceph/ceph.conf
```
Create a CephX user and get its secret key:
```
ssh {user}@{mon-host} "sudo ceph fs authorize cephfs client.foo / rw" | sudo tee /etc/ceph/ceph.client.foo.keyring
```
In above command, replace cephfs with the name of your CephFS, foo by the name you want for your CephX user and / by the path within your CephFS for which you want to allow access to the client host and rw stands for both read and write permissions. Alternatively, you may copy the Ceph keyring from the MON host to client host at /etc/ceph but creating a keyring specific to the client host is better. While creating a CephX keyring/client, using same client name across multiple machines is perfectly fine.

以我这边的实际情况为例（在archlinux 笔记本，内网环境，挂载cephfs文件系统）

[wangjm@jingminarchpc ~]$ sudo mkdir -p -m 755 /etc/ceph


[wangjm@jingminarchpc ~]$ ssh root@192.168.1.8 "sudo ceph config generate-minimal-conf" | sudo tee /etc/ceph/ceph.conf
# minimal ceph.conf for 92046bac-05dd-11ef-979f-572db13abde1
[global]
        fsid = 92046bac-05dd-11ef-979f-572db13abde1
        mon_host = [v2:192.168.1.1:3300/0,v1:192.168.1.1:6789/0] [v2:192.168.1.8:3300/0,v1:192.168.1.8:6789/0] [v2:192.168.1.9:3300/0,v1:192.168.1.9:6789/0] [v2:192.168.1.10:3300/0,v1:192.168.1.10:6789/0]
        
        
[wangjm@jingminarchpc ~]$ cat /etc/ceph/ceph.conf 
# minimal ceph.conf for 92046bac-05dd-11ef-979f-572db13abde1
[global]
        fsid = 92046bac-05dd-11ef-979f-572db13abde1
        mon_host = [v2:192.168.1.1:3300/0,v1:192.168.1.1:6789/0] [v2:192.168.1.8:3300/0,v1:192.168.1.8:6789/0] [v2:192.168.1.9:3300/0,v1:192.168.1.9:6789/0] [v2:192.168.1.10:3300/0,v1:192.168.1.10:6789/0]


[wangjm@jingminarchpc ~]$ ssh root@192.168.1.8 "sudo ceph fs authorize cephfs client.wangjm / rw" | sudo tee /etc/ceph/ceph.client.wangjm.keyring
[client.wangjm]
        key = AQB8R0BmRzHrNBAA4fDPiCkzoVXCgYjpblXzog==

[wangjm@jingminarchpc ~]$ ssh root@192.168.1.8 "sudo ceph fs authorize cephfs client.wangjm2 / rw" | sudo tee /etc/ceph/ceph.client.wangjm2.keyring
[client.wangjm2]
        key = AQAHR0Bm34+NMRAAuBC0qcTrl+GpFv3mylUdgw==

我是在web界面中, cluster->users中添加了wangjm, wangjm2帐号.

也可以在命令行中配置: 需要在ceph中配置类似管理员类型的帐号,

参考: https://github.com/ceph/ceph-csi/blob/devel/docs/capabilities.md

# 使用如下配置, 我不知道是不是有问题, 反正权限不足
USER=wangjm
FS_NAME=cephfs
SUB_VOL=csi
ceph auth get-or-create client.$USER \
  mgr "allow rw" \
  osd "allow rw tag cephfs metadata=$FS_NAME, allow rw tag cephfs data=$FS_NAME" \
  mds "allow r fsname=$FS_NAME path=/volumes, allow rws fsname=$FS_NAME path=/volumes/$SUB_VOL" \
  mon "allow r fsname=$FS_NAME"
  
# 后来用的是这个配置. 
USER=wangjm
FS_NAME=cephfs
ceph auth caps client.$USER   
mgr "allow rw"   
osd "allow rw tag cephfs metadata=$FS_NAME, allow rw tag cephfs data=$FS_NAME"   
mds "allow rw fsname=$FS_NAME"   
mon "allow r fsname=$FS_NAME"

实际上,这是后面发现前面生成的帐号有问题(wangjm这个权限不足, 折回来补上的, 需要修改下权限)

参考: https://docs.ceph.com/en/latest/rados/operations/user-management/#modifying-user-capabilities

参考: https://github.com/ceph/ceph-csi/blob/devel/docs/capabilities.md

参考:https://github.com/ceph/ceph-csi/blob/devel/docs/static-pvc.md#cephfs-static-pvc

之前已经建好了此帐号, 这里使用ceph auth caps命令



#官网提供的, 参考: https://github.com/ceph/ceph-csi/blob/devel/docs/capabilities.md
#USER=wangjm
#FS_NAME=cephfs
#SUB_VOL=csi
#ceph auth get-or-create client.$USER \
#  mgr "allow rw" \
#  osd "allow rw tag cephfs metadata=$FS_NAME, allow rw tag cephfs data=$FS_NAME" \
#  mds "allow r fsname=$FS_NAME path=/volumes, allow rws fsname=$FS_NAME path=/volumes/$SUB_VOL" \
#  mon "allow r fsname=$FS_NAME"

# 不行. 还是显示权限不足.()

# 后来实际采用的(实际上,就是web中建立的,默认的帐号权限)
USER=wangjm
FS_NAME=cephfs
ceph auth caps client.$USER   
mgr "allow rw"   
osd "allow rw tag cephfs metadata=$FS_NAME, allow rw tag cephfs data=$FS_NAME"   
mds "allow rw fsname=$FS_NAME"   
mon "allow r fsname=$FS_NAME"

如果用户还没创建, 就用ceph auth get-or-create命令. 如果用户已经创建, 就用ceph auth caps调整用户权限.

怀疑是k8s中provisoner之类的东东有缓存.

删除重建csi相关内容.

kubectl config set-context --current --namespace ceph-cephfs

#我在当前目录下存放了, 所有cephfs用到的yaml文件
kubectl delete -f ./

kubectl apply -f ./

还是不行, 日志报如下错误

I0514 12:49:51.106806       1 utils.go:199] ID: 22 GRPC request: {}
I0514 12:49:51.106821       1 utils.go:205] ID: 22 GRPC response: {}
I0514 12:50:13.292900       1 utils.go:198] ID: 23 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 GRPC call: /csi.v1.Controller/CreateVolume
I0514 12:50:13.293053       1 utils.go:199] ID: 23 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9","parameters":{"clusterID":"92046bac-05dd-11ef-979f-572db13abde1","csi.storage.k8s.io/pv/name":"pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9","csi.storage.k8s.io/pvc/name":"nginx-cephfs-pvc","csi.storage.k8s.io/pvc/namespace":"test-nginx","fsName":"cephfs","imageFeatures":"layering"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":5}}]}
I0514 12:50:13.294801       1 omap.go:89] ID: 23 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 got omap values: (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volumes.default"): map[]
I0514 12:50:13.332500       1 omap.go:159] ID: 23 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 set omap keys (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volumes.default"): map[csi.volume.pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9:3e803a86-91dc-4d7b-b944-785444f33316])
I0514 12:50:13.341369       1 omap.go:159] ID: 23 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 set omap keys (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volume.3e803a86-91dc-4d7b-b944-785444f33316"): map[csi.imagename:csi-vol-3e803a86-91dc-4d7b-b944-785444f33316 csi.volname:pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 csi.volume.owner:test-nginx])
I0514 12:50:13.341386       1 fsjournal.go:311] ID: 23 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 Generated Volume ID (0001-0024-92046bac-05dd-11ef-979f-572db13abde1-0000000000000001-3e803a86-91dc-4d7b-b944-785444f33316) and subvolume name (csi-vol-3e803a86-91dc-4d7b-b944-785444f33316) for request name (pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9)
E0514 12:50:13.344058       1 volume.go:244] ID: 23 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 failed to create subvolume csi-vol-3e803a86-91dc-4d7b-b944-785444f33316 in fs cephfs: rados: ret=-2, No such file or directory: "subvolume group 'csi' does not exist"
E0514 12:50:13.344079       1 controllerserver.go:102] ID: 23 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 failed to create volume pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9: rados: ret=-2, No such file or directory: "subvolume group 'csi' does not exist"
I0514 12:50:13.361979       1 omap.go:126] ID: 23 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 removed omap keys (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volumes.default"): [csi.volume.pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9]
E0514 12:50:13.362028       1 utils.go:203] ID: 23 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 GRPC error: rpc error: code = Internal desc = rados: ret=-2, No such file or directory: "subvolume group 'csi' does not exist"
I0514 12:50:51.107126       1 utils.go:198] ID: 24 GRPC call: /csi.v1.Identity/Probe
I0514 12:50:51.107219       1 utils.go:199] ID: 24 GRPC request: {}
I0514 12:50:51.107237       1 utils.go:205] ID: 24 GRPC response: {}
I0514 12:51:17.364702       1 utils.go:198] ID: 25 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 GRPC call: /csi.v1.Controller/CreateVolume
I0514 12:51:17.364839       1 utils.go:199] ID: 25 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9","parameters":{"clusterID":"92046bac-05dd-11ef-979f-572db13abde1","csi.storage.k8s.io/pv/name":"pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9","csi.storage.k8s.io/pvc/name":"nginx-cephfs-pvc","csi.storage.k8s.io/pvc/namespace":"test-nginx","fsName":"cephfs","imageFeatures":"layering"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":5}}]}
I0514 12:51:17.366520       1 omap.go:89] ID: 25 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 got omap values: (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volumes.default"): map[]
I0514 12:51:17.395811       1 omap.go:159] ID: 25 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 set omap keys (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volumes.default"): map[csi.volume.pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9:066899f8-7c72-4a3e-b211-b89e1d50b939])
I0514 12:51:17.404285       1 omap.go:159] ID: 25 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 set omap keys (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volume.066899f8-7c72-4a3e-b211-b89e1d50b939"): map[csi.imagename:csi-vol-066899f8-7c72-4a3e-b211-b89e1d50b939 csi.volname:pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 csi.volume.owner:test-nginx])
I0514 12:51:17.404303       1 fsjournal.go:311] ID: 25 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 Generated Volume ID (0001-0024-92046bac-05dd-11ef-979f-572db13abde1-0000000000000001-066899f8-7c72-4a3e-b211-b89e1d50b939) and subvolume name (csi-vol-066899f8-7c72-4a3e-b211-b89e1d50b939) for request name (pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9)
E0514 12:51:17.424478       1 volume.go:244] ID: 25 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 failed to create subvolume csi-vol-066899f8-7c72-4a3e-b211-b89e1d50b939 in fs cephfs: rados: ret=-2, No such file or directory: "subvolume group 'csi' does not exist"
E0514 12:51:17.424495       1 controllerserver.go:102] ID: 25 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 failed to create volume pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9: rados: ret=-2, No such file or directory: "subvolume group 'csi' does not exist"
I0514 12:51:17.441948       1 omap.go:126] ID: 25 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 removed omap keys (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volumes.default"): [csi.volume.pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9]
E0514 12:51:17.442014       1 utils.go:203] ID: 25 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 GRPC error: rpc error: code = Internal desc = rados: ret=-2, No such file or directory: "subvolume group 'csi' does not exist"

显示rados: ret=-2, No such file or directory: "subvolume group 'csi' does not exist"

参考: https://github.com/rook/rook/issues/6183

在这个issue中, 最后有人说, 可以自己手动建一下对应的subvolumegroup.

参考: https://docs.ceph.com/en/latest/cephfs/fs-volumes/#fs-subvolume-groups

ceph fs subvolumegroup create cephfs csi

我试了下, 成功了.

AI0514 12:53:25.445117       1 utils.go:198] ID: 28 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 GRPC call: /csi.v1.Controller/CreateVolume
I0514 12:53:25.445246       1 utils.go:199] ID: 28 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9","parameters":{"clusterID":"92046bac-05dd-11ef-979f-572db13abde1","csi.storage.k8s.io/pv/name":"pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9","csi.storage.k8s.io/pvc/name":"nginx-cephfs-pvc","csi.storage.k8s.io/pvc/namespace":"test-nginx","fsName":"cephfs","imageFeatures":"layering"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":5}}]}
I0514 12:53:25.446973       1 omap.go:89] ID: 28 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 got omap values: (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volumes.default"): map[]
I0514 12:53:25.477948       1 omap.go:159] ID: 28 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 set omap keys (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volumes.default"): map[csi.volume.pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9:54b2a3a1-7505-47c5-8407-64c98a1b44de])
I0514 12:53:25.486406       1 omap.go:159] ID: 28 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 set omap keys (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volume.54b2a3a1-7505-47c5-8407-64c98a1b44de"): map[csi.imagename:csi-vol-54b2a3a1-7505-47c5-8407-64c98a1b44de csi.volname:pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 csi.volume.owner:test-nginx])
I0514 12:53:25.486435       1 fsjournal.go:311] ID: 28 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 Generated Volume ID (0001-0024-92046bac-05dd-11ef-979f-572db13abde1-0000000000000001-54b2a3a1-7505-47c5-8407-64c98a1b44de) and subvolume name (csi-vol-54b2a3a1-7505-47c5-8407-64c98a1b44de) for request name (pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9)
I0514 12:53:25.862740       1 controllerserver.go:475] ID: 28 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 cephfs: successfully created backing volume named csi-vol-54b2a3a1-7505-47c5-8407-64c98a1b44de for request name pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9
I0514 12:53:25.862903       1 utils.go:205] ID: 28 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 GRPC response: {"volume":{"capacity_bytes":10737418240,"volume_context":{"clusterID":"92046bac-05dd-11ef-979f-572db13abde1","fsName":"cephfs","imageFeatures":"layering","subvolumeName":"csi-vol-54b2a3a1-7505-47c5-8407-64c98a1b44de","subvolumePath":"/volumes/csi/csi-vol-54b2a3a1-7505-47c5-8407-64c98a1b44de/11b62e7a-43ba-4c02-9900-5e9bdd449f1f"},"volume_id":"0001-0024-92046bac-05dd-11ef-979f-572db13abde1-0000000000000001-54b2a3a1-7505-47c5-8407-64c98a1b44de"}}

k8s+cephfs(cephfs文件存储方式)

参考： https://github.com/ceph/ceph-csi/blob/devel/README.md

参考： https://github.com/ceph/ceph-csi/blob/devel/docs/deploy-cephfs.md

参考： https://github.com/ceph/ceph-csi/tree/devel/deploy/cephfs/kubernetes

参考： https://juejin.cn/post/7296756504912330767

类似于前面配置rbd块存储的方式。

新建k8s命名空间，调整默认命名空间

kubectl create ns ceph-cephfs

kubectl config set-context --current --namespace ceph-cephfs

创建ceph-csi使用的configMap

ceph-csi需要一个存储在k8s中的ConfigMap，这个ConfigMap中保存了ceph集群的fsid和monitors的地址

# csi-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    [
      {
        "clusterID": "92046bac-05dd-11ef-979f-572db13abde1",
        "monitors": [
          "192.168.1.1:6789",
          "192.168.1.8:6789",
          "192.168.1.9:6789",
          "192.168.1.10:6789"
        ]
      }
    ]
metadata:
  name: ceph-csi-config

应用yaml文件

root@wangjm-B550M-K-1:~/k8s/csi/ceph/cephfs# kubectl apply -f csi-config-map.yaml 
configmap/ceph-csi-config created

创建KMS provider所使用的ConfigMap文件

最新版本的ceph-csi还需要一个额外的ConfigMap对象来定义密钥管理服务(KMS)提供程序的详细信息。如果没有设置KMS，在csi-kms-config-map中放置一个空配置即可

# csi-kms-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    {}
metadata:
  name: ceph-csi-encryption-kms-config

应用yaml文件

root@wangjm-B550M-K-1:~/k8s/csi/ceph/cephfs# kubectl apply -f csi-kms-config-map.yaml 
configmap/ceph-csi-encryption-kms-config created

创建一个保存Ceph配置的ConfigMap

最新版本的Ceph-CSI还需要另一个ConfigMap对象来定义Ceph配置，以便添加到CSI容器内的ceph.conf文件中

# ceph-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
  ceph.conf: |
    [global]
    auth_cluster_required = cephx
    auth_service_required = cephx
    auth_client_required = cephx
  # keyring is a required key and its value should be empty
  keyring: |
metadata:
  name: ceph-config

应用yaml文件

root@wangjm-B550M-K-1:~/k8s/csi/ceph/cephfs# kubectl apply -f ceph-config-map.yaml
configmap/ceph-config created

创建ceph-csi cephx secret

Ceph-csi需要cephx凭据才能与Ceph集群通信

创建yaml文件

注意: 之前ceph-rbd前面的配置, 这里用到的帐号密码是userID和userKey. 但是后面cephfs部署的时候, 创建pvc的时候pending状态. 查了下provisioner的日志. 发现需要adminID. github上rook中好像有人报过类似的问题.

参考: https://github.com/ceph/ceph-csi/blob/devel/docs/deploy-cephfs.md 这里有具体的说明, 可以搜索一下关键字(fsName, adminID, userID)

参考: https://github.com/rook/rook/pull/5448/commits/056e6631344665a2e10526a4023c1aa8fce6a383

参考: https://github.com/rook/rook/pull/5448

参考: https://github.com/rook/rook/pull/5447

参考: https://github.com/rook/rook/issues/5432

# csi-cephfs-secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: csi-cephfs-secret
  namespace: ceph-cephfs
stringData:
  adminID: wangjm
  adminKey: AQB8R0BmRzHrNBAA4fDPiCkzoVXCgYjpblXzog==
  userID: wangjm
  userKey: AQB8R0BmRzHrNBAA4fDPiCkzoVXCgYjpblXzog==

应用yaml文件

root@wangjm-B550M-K-1:~/k8s/csi/ceph/cephfs# kubectl apply -f ./csi-cephfs-secret.yaml 
secret/csi-cephfs-secret created

配置ceph-csi plugins

创建csi driver

wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/cephfs/kubernetes/csidriver.yaml

kubectl apply -f csidriver.yaml

创建需要的ServiceAccount和RBAC ClusterRole/ClusterRoleBinding

wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/cephfs/kubernetes/csi-provisioner-rbac.yaml
wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/cephfs/kubernetes/csi-nodeplugin-rbac.yaml

vim csi-provisioner-rbac.yaml
:%s/namespace\: default/namespace\: ceph-cephfs/g

vim csi-nodeplugin-rbac.yaml
:%s/namespace\: default/namespace\: ceph-cephfs/g

kubectl create -f csi-provisioner-rbac.yaml
kubectl create -f csi-nodeplugin-rbac.yaml

创建ceph-csi provisioner 和node plugins

wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/cephfs/kubernetes/csi-cephfsplugin-provisioner.yaml
wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/cephfs/kubernetes/csi-cephfsplugin.yaml

vim csi-cephfsplugin-provisioner.yaml
:%s/namespace\: default/namespace\: ceph-cephfs/g

vim csi-cephfsplugin.yaml
:%s/namespace\: default/namespace\: ceph-cephfs/g

kubectl apply -f csi-cephfsplugin-provisioner.yaml
kubectl apply -f csi-cephfsplugin.yaml

optional: 由于yaml文件里面的镜像是国外的地址源，因此这里改成自己的镜像地址

（或者也可以配置containerd/cri-o代理，来下载国外镜像）

如果设置过 containerd/cri-o/docker 的代理, 或者国际网络通畅的话, 可以略过.

查看yaml文件里面所使用的镜像

[root@node1 ceph-csi]# grep 'image:'  csi-rbdplugin-provisioner.yaml 
          image: registry.k8s.io/sig-storage/csi-provisioner:v3.6.0
          image: registry.k8s.io/sig-storage/csi-snapshotter:v6.3.0
          image: registry.k8s.io/sig-storage/csi-attacher:v4.4.0
          image: gcr.io/k8s-staging-sig-storage/csi-resizer:v1.9.0
          image: quay.io/cephcsi/cephcsi:canary
          image: quay.io/cephcsi/cephcsi:canary
          image: quay.io/cephcsi/cephcsi:canary
          
[root@node1 ceph-csi]# grep 'image:'  csi-rbdplugin.yaml 
          image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.0
          image: quay.io/cephcsi/cephcsi:canary
          image: quay.io/cephcsi/cephcsi:canary

将上面文件中的镜像替换成aliyun中保存的镜像

sed -i s#registry.k8s.io/sig-storage/csi-provisioner:v3.6.0#registry.cn-hangzhou.aliyuncs.com/postkarte/csi-provisioner:v3.6.0#g csi-rbdplugin-provisioner.yaml

sed -i s#registry.k8s.io/sig-storage/csi-snapshotter:v6.3.0#registry.cn-hangzhou.aliyuncs.com/postkarte/csi-snapshotter:v6.3.0#g csi-rbdplugin-provisioner.yaml

sed -i s#registry.k8s.io/sig-storage/csi-attacher:v4.4.0#registry.cn-hangzhou.aliyuncs.com/postkarte/csi-attacher:v4.4.0#g  csi-rbdplugin-provisioner.yaml

sed -i s#gcr.io/k8s-staging-sig-storage/csi-resizer:v1.9.0#registry.cn-hangzhou.aliyuncs.com/postkarte/csi-resizer:v1.9.0#g csi-rbdplugin-provisioner.yaml

sed -i s#quay.io/cephcsi/cephcsi:canary#registry.cn-hangzhou.aliyuncs.com/postkarte/cephcsi:canary#g csi-rbdplugin-provisioner.yaml

sed -i s#registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.0#registry.cn-hangzhou.aliyuncs.com/postkarte/csi-node-driver-registrar:v2.9.0#g csi-rbdplugin.yaml 

sed -i s#quay.io/cephcsi/cephcsi:canary#registry.cn-hangzhou.aliyuncs.com/postkarte/cephcsi:canary#g csi-rbdplugin.yaml

应用yaml文件

kubectl apply -f csi-cephfsplugin-provisioner.yaml 
kubectl apply -f csi-cephfsplugin.yaml

创建storageclass

参考: https://github.com/ceph/ceph-csi/blob/devel/examples/cephfs/storageclass.yaml

# csi-cephfs-sc.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-cephfs-sc
provisioner: cephfs.csi.ceph.com
parameters:
   clusterID: 92046bac-05dd-11ef-979f-572db13abde1
   fsName: cephfs
   #pool: kubernetes
   #imageFeatures: layering
   csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret
   csi.storage.k8s.io/provisioner-secret-namespace: ceph-cephfs
   csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: ceph-cephfs
   csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
   csi.storage.k8s.io/node-stage-secret-namespace: ceph-cephfs
reclaimPolicy: Delete
allowVolumeExpansion: true
# mountOptions:
#   - context="system_u:object_r:container_file_t:s0:c0,c1"

注意这里 fsName和clusterID等几个parameters必须. 参见: https://github.com/ceph/ceph-csi/blob/devel/docs/deploy-cephfs.md

应用yaml文件

kubectl apply -f csi-cephfs-sc.yaml 
storageclass.storage.k8s.io/csi-cephfs-sc created

如果要设为默认storageclass (一般还是考虑使用rbd作为默认storageclass, 因为性能较好. 除非需要多node访问同一资源的时候,才考虑使用cephfs)

kubectl patch storageclass csi-cephfs-sc -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

并需要删除其他storageclass上的此annotation

查看sc

root@wangjm-B550M-K-1:~/k8s/csi/ceph/cephfs# kubectl get sc
NAME                   PROVISIONER           RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
csi-cephfs-sc          cephfs.csi.ceph.com   Delete          Immediate           true                   87s
csi-rbd-sc (default)   rbd.csi.ceph.com      Delete          Immediate           true                   3d22h

测试cephfs类型的storageclass

参考: https://kubernetes.io/zh-cn/docs/tasks/run-application/run-stateless-application-deployment/

参考: https://leehao.me/k8s-%E9%83%A8%E7%BD%B2-nginx-%E5%85%A5%E9%97%A8/

创建命名空间, 设为当前操作的默认命名空间

kubectl create ns test-nginx

kubectl config set-context --current --namespace test-nginx

还有哪里参考了一下, 反正就是nginx的镜像, pvc, pod,service, ingress那一套

# cat nginx.yaml 
---
##创建pvc
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nginx-cephfs-pvc
  labels:
    pvc: cephfs
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: csi-cephfs-sc
---
##部署应用nginx
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    name: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      name: nginx
  template:
    metadata:
      labels:
       name: nginx
    spec:
      containers:
      - name: nginx
        image: docker.io/nginx
        volumeMounts:
        - mountPath: /usr/share/nginx/html
          name: nginx-data
        ports:
        - containerPort: 80
      volumes:
      - name: nginx-data
        persistentVolumeClaim:
          claimName: nginx-cephfs-pvc
---
##创建service
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    name: nginx
spec:
  #type: NodePort
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
    name: http
    #nodePort: 30080
  selector:
    name: nginx

然后看下pvc是什么状态 pending还是bound.

kubectl get pvc

如果是pending,说明配置的storageclass以及相关的provisoner有问题, 或者说真个csi驱动有问题. 需要排查问题.(比如我就遇到了问题, 参见后面的问题处理章节)

如果是bound状态, 说明已经动态分配了pv.

可以查一下对应的pv

kubectl get pv

如果有装ingress controller的的话, 可以加一下ingress

---
## 创建ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
  - host: book.ole12138.cn
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80

看下ingress状态

root@wangjm-B550M-K-1:~/k8s/test/test-nginx# kubectl get ingress
NAME    CLASS   HOSTS              ADDRESS         PORTS   AGE
nginx   nginx   book.ole12138.cn   192.168.1.100   80      7m52s
root@wangjm-B550M-K-1:~/k8s/test/test-nginx# curl --resolve book.ole12138.cn:80:192.168.1.100 http://book.ole12138.cn
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx</center>
</body>
</html>

参考: https://juejin.cn/post/7271613128361246781

基本没问题了.

问题处理: 配置的cephfs帐号无效, pvc一直pending的问题

第一次测试上面cephfs用作provisoner/storageclass的时候.

请求的pvc一直是pending状态

code = Internal desc = rados: ret=-1, Operation not permitted

参考: https://github.com/ceph/ceph-csi/issues/1818

参考: https://github.com/ceph/ceph-csi/issues/2506

需要在ceph中配置类似管理员类型的帐号

参考: https://github.com/ceph/ceph-csi/blob/devel/docs/capabilities.md

# 使用如下配置, 我不知道是不是有问题, 反正权限不足
USER=wangjm
FS_NAME=cephfs
SUB_VOL=csi
ceph auth get-or-create client.$USER \
  mgr "allow rw" \
  osd "allow rw tag cephfs metadata=$FS_NAME, allow rw tag cephfs data=$FS_NAME" \
  mds "allow r fsname=$FS_NAME path=/volumes, allow rws fsname=$FS_NAME path=/volumes/$SUB_VOL" \
  mon "allow r fsname=$FS_NAME"
  
# 后来用的是这个配置. 
USER=wangjm
FS_NAME=cephfs
ceph auth caps client.$USER   
mgr "allow rw"   
osd "allow rw tag cephfs metadata=$FS_NAME, allow rw tag cephfs data=$FS_NAME"   
mds "allow rw fsname=$FS_NAME"   
mon "allow r fsname=$FS_NAME"

实际上,这是后面发现前面生成的帐号有问题(wangjm这个权限不足, 折回来补上的, 需要修改下权限)

参考: https://docs.ceph.com/en/latest/rados/operations/user-management/#modifying-user-capabilities

参考: https://github.com/ceph/ceph-csi/blob/devel/docs/capabilities.md

参考:https://github.com/ceph/ceph-csi/blob/devel/docs/static-pvc.md#cephfs-static-pvc

参考: https://elrond.wang/2021/08/16/CephFS-subvolume/

参考: https://access.redhat.com/documentation/zh-cn/red_hat_ceph_storage/4/html/file_system_guide/ceph-file-system-subvolumes

之前已经建好了此帐号, 这里使用ceph auth caps命令

USER=wangjm
FS_NAME=cephfs
SUB_VOL_GROUP=group1
SUB_VOL=volume1
ceph auth caps client.$USER \
  mgr "allow rw" \
  osd "allow rw tag cephfs metadata=$FS_NAME, allow rw tag cephfs data=$FS_NAME" \
  mds "allow r fsname=$FS_NAME path=/volumes, allow r fsname=$FS_NAME path=/volumes/$SUB_VOL_GROUP, allow rws fsname=$FS_NAME path=/volumes/$SUB_VOL_GROUP/$SUB_VOL" \
  mon "allow r fsname=$FS_NAME"

不行. 还是显示权限不足.()

恢复默认的帐号权限

USER=wangjm
FS_NAME=cephfs
ceph auth caps client.$USER   
mgr "allow rw"   
osd "allow rw tag cephfs metadata=$FS_NAME, allow rw tag cephfs data=$FS_NAME"   
mds "allow rw fsname=$FS_NAME"   
mon "allow r fsname=$FS_NAME"

怀疑是k8s中provisoner之类的东东有缓存.

删除重建csi相关内容.

kubectl config set-context --current --namespace ceph-cephfs

#我在当前目录下存放了, 所有cephfs用到的yaml文件
kubectl delete -f ./

还是不行, 日志报如下错误

I0514 12:49:51.106806       1 utils.go:199] ID: 22 GRPC request: {}
I0514 12:49:51.106821       1 utils.go:205] ID: 22 GRPC response: {}
I0514 12:50:13.292900       1 utils.go:198] ID: 23 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 GRPC call: /csi.v1.Controller/CreateVolume
I0514 12:50:13.293053       1 utils.go:199] ID: 23 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9","parameters":{"clusterID":"92046bac-05dd-11ef-979f-572db13abde1","csi.storage.k8s.io/pv/name":"pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9","csi.storage.k8s.io/pvc/name":"nginx-cephfs-pvc","csi.storage.k8s.io/pvc/namespace":"test-nginx","fsName":"cephfs","imageFeatures":"layering"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":5}}]}
I0514 12:50:13.294801       1 omap.go:89] ID: 23 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 got omap values: (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volumes.default"): map[]
I0514 12:50:13.332500       1 omap.go:159] ID: 23 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 set omap keys (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volumes.default"): map[csi.volume.pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9:3e803a86-91dc-4d7b-b944-785444f33316])
I0514 12:50:13.341369       1 omap.go:159] ID: 23 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 set omap keys (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volume.3e803a86-91dc-4d7b-b944-785444f33316"): map[csi.imagename:csi-vol-3e803a86-91dc-4d7b-b944-785444f33316 csi.volname:pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 csi.volume.owner:test-nginx])
I0514 12:50:13.341386       1 fsjournal.go:311] ID: 23 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 Generated Volume ID (0001-0024-92046bac-05dd-11ef-979f-572db13abde1-0000000000000001-3e803a86-91dc-4d7b-b944-785444f33316) and subvolume name (csi-vol-3e803a86-91dc-4d7b-b944-785444f33316) for request name (pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9)
E0514 12:50:13.344058       1 volume.go:244] ID: 23 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 failed to create subvolume csi-vol-3e803a86-91dc-4d7b-b944-785444f33316 in fs cephfs: rados: ret=-2, No such file or directory: "subvolume group 'csi' does not exist"
E0514 12:50:13.344079       1 controllerserver.go:102] ID: 23 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 failed to create volume pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9: rados: ret=-2, No such file or directory: "subvolume group 'csi' does not exist"
I0514 12:50:13.361979       1 omap.go:126] ID: 23 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 removed omap keys (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volumes.default"): [csi.volume.pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9]
E0514 12:50:13.362028       1 utils.go:203] ID: 23 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 GRPC error: rpc error: code = Internal desc = rados: ret=-2, No such file or directory: "subvolume group 'csi' does not exist"
I0514 12:50:51.107126       1 utils.go:198] ID: 24 GRPC call: /csi.v1.Identity/Probe
I0514 12:50:51.107219       1 utils.go:199] ID: 24 GRPC request: {}
I0514 12:50:51.107237       1 utils.go:205] ID: 24 GRPC response: {}
I0514 12:51:17.364702       1 utils.go:198] ID: 25 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 GRPC call: /csi.v1.Controller/CreateVolume
I0514 12:51:17.364839       1 utils.go:199] ID: 25 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9","parameters":{"clusterID":"92046bac-05dd-11ef-979f-572db13abde1","csi.storage.k8s.io/pv/name":"pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9","csi.storage.k8s.io/pvc/name":"nginx-cephfs-pvc","csi.storage.k8s.io/pvc/namespace":"test-nginx","fsName":"cephfs","imageFeatures":"layering"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":5}}]}
I0514 12:51:17.366520       1 omap.go:89] ID: 25 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 got omap values: (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volumes.default"): map[]
I0514 12:51:17.395811       1 omap.go:159] ID: 25 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 set omap keys (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volumes.default"): map[csi.volume.pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9:066899f8-7c72-4a3e-b211-b89e1d50b939])
I0514 12:51:17.404285       1 omap.go:159] ID: 25 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 set omap keys (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volume.066899f8-7c72-4a3e-b211-b89e1d50b939"): map[csi.imagename:csi-vol-066899f8-7c72-4a3e-b211-b89e1d50b939 csi.volname:pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 csi.volume.owner:test-nginx])
I0514 12:51:17.404303       1 fsjournal.go:311] ID: 25 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 Generated Volume ID (0001-0024-92046bac-05dd-11ef-979f-572db13abde1-0000000000000001-066899f8-7c72-4a3e-b211-b89e1d50b939) and subvolume name (csi-vol-066899f8-7c72-4a3e-b211-b89e1d50b939) for request name (pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9)
E0514 12:51:17.424478       1 volume.go:244] ID: 25 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 failed to create subvolume csi-vol-066899f8-7c72-4a3e-b211-b89e1d50b939 in fs cephfs: rados: ret=-2, No such file or directory: "subvolume group 'csi' does not exist"
E0514 12:51:17.424495       1 controllerserver.go:102] ID: 25 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 failed to create volume pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9: rados: ret=-2, No such file or directory: "subvolume group 'csi' does not exist"
I0514 12:51:17.441948       1 omap.go:126] ID: 25 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 removed omap keys (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volumes.default"): [csi.volume.pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9]
E0514 12:51:17.442014       1 utils.go:203] ID: 25 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 GRPC error: rpc error: code = Internal desc = rados: ret=-2, No such file or directory: "subvolume group 'csi' does not exist"

显示rados: ret=-2, No such file or directory: "subvolume group 'csi' does not exist"

在这个issue中, 最后有人说, 可以自己手动建一下对应的subvolumegroup.

参考: https://github.com/rook/rook/issues/6183

参考: https://docs.ceph.com/en/latest/cephfs/fs-volumes/#fs-subvolume-groups

ceph fs subvolumegroup create cephfs csi

我试了下, 成功了.

AI0514 12:53:25.445117       1 utils.go:198] ID: 28 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 GRPC call: /csi.v1.Controller/CreateVolume
I0514 12:53:25.445246       1 utils.go:199] ID: 28 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9","parameters":{"clusterID":"92046bac-05dd-11ef-979f-572db13abde1","csi.storage.k8s.io/pv/name":"pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9","csi.storage.k8s.io/pvc/name":"nginx-cephfs-pvc","csi.storage.k8s.io/pvc/namespace":"test-nginx","fsName":"cephfs","imageFeatures":"layering"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":5}}]}
I0514 12:53:25.446973       1 omap.go:89] ID: 28 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 got omap values: (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volumes.default"): map[]
I0514 12:53:25.477948       1 omap.go:159] ID: 28 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 set omap keys (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volumes.default"): map[csi.volume.pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9:54b2a3a1-7505-47c5-8407-64c98a1b44de])
I0514 12:53:25.486406       1 omap.go:159] ID: 28 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 set omap keys (pool="cephfs.cephfs.meta", namespace="csi", name="csi.volume.54b2a3a1-7505-47c5-8407-64c98a1b44de"): map[csi.imagename:csi-vol-54b2a3a1-7505-47c5-8407-64c98a1b44de csi.volname:pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 csi.volume.owner:test-nginx])
I0514 12:53:25.486435       1 fsjournal.go:311] ID: 28 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 Generated Volume ID (0001-0024-92046bac-05dd-11ef-979f-572db13abde1-0000000000000001-54b2a3a1-7505-47c5-8407-64c98a1b44de) and subvolume name (csi-vol-54b2a3a1-7505-47c5-8407-64c98a1b44de) for request name (pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9)
I0514 12:53:25.862740       1 controllerserver.go:475] ID: 28 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 cephfs: successfully created backing volume named csi-vol-54b2a3a1-7505-47c5-8407-64c98a1b44de for request name pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9
I0514 12:53:25.862903       1 utils.go:205] ID: 28 Req-ID: pvc-9e6e7c2a-915c-4440-9ada-7c4f73829ad9 GRPC response: {"volume":{"capacity_bytes":10737418240,"volume_context":{"clusterID":"92046bac-05dd-11ef-979f-572db13abde1","fsName":"cephfs","imageFeatures":"layering","subvolumeName":"csi-vol-54b2a3a1-7505-47c5-8407-64c98a1b44de","subvolumePath":"/volumes/csi/csi-vol-54b2a3a1-7505-47c5-8407-64c98a1b44de/11b62e7a-43ba-4c02-9900-5e9bdd449f1f"},"volume_id":"0001-0024-92046bac-05dd-11ef-979f-572db13abde1-0000000000000001-54b2a3a1-7505-47c5-8407-64c98a1b44de"}}

再查下pvc,是否已经bound到了pv

kubectl get pvc

看到状态已经是bound.

03 K8s使用外部Ceph集群

K8s使用外部Ceph集群(配置ceph-csi)

K8S介绍

存储介绍

架构图

k8s+rbd (rbd块存储方式)

在ceph上创建存储池

新创建的存储池在使用之前需要初始化，使用rbd工具进行初始化

设置客户端认证

查看集群ID

新建k8s命名空间, 调整默认命名空间

创建ceph-csi使用的configMap

应用yaml文件

创建KMS provider所使用的ConfigMap文件

应用yaml文件

创建一个保存Ceph配置的ConfigMap

应用yaml文件

创建ceph-csi cephx secret

创建yaml文件

应用yaml文件

配置ceph-csi plugins

创建csi driver

创建需要的ServiceAccount和RBAC ClusterRole/ClusterRoleBinding

创建ceph-csi provisioner 和node plugins

由于yaml文件里面的镜像是国外的地址源，因此这里改成自己的镜像地址

查看yaml文件里面所使用的镜像

将上面文件中的镜像替换成aliyun中保存的镜像

应用yaml文件

创建storageclass

应用yaml文件

查看sc

部署应用测试

测试1

测试2

配置ceph中帐号,供cephfs使用

创建名为“cephfs” 卷(文件系统)

方式一： 创建文件系统并自动分配pool

方式二： 手动创建pool和并关联文件系统(未使用)

配置客户端权限

k8s+cephfs(cephfs文件存储方式)

新建k8s命名空间，调整默认命名空间

创建ceph-csi使用的configMap

应用yaml文件

创建KMS provider所使用的ConfigMap文件

应用yaml文件

创建一个保存Ceph配置的ConfigMap

应用yaml文件

创建ceph-csi cephx secret

创建yaml文件

应用yaml文件

配置ceph-csi plugins

创建csi driver

创建需要的ServiceAccount和RBAC ClusterRole/ClusterRoleBinding

创建ceph-csi provisioner 和node plugins

查看yaml文件里面所使用的镜像

将上面文件中的镜像替换成aliyun中保存的镜像

应用yaml文件

创建storageclass

应用yaml文件

查看sc

测试cephfs类型的storageclass

问题处理: 配置的cephfs帐号无效, pvc一直pending的问题

评论

发表回复 取消回复

方式一：创建文件系统并自动分配pool

方式二：手动创建pool和并关联文件系统(未使用)

发表回复取消回复