Clever Geek Handbook

将基于Ceph的存储连接到Kubernetes集群的实际示例

容器存储接口（CSI）是用于Kubernetes与存储系统之间通信的统一接口。我们已经简短地讨论了它，但是今天我们将仔细研究CSI和Ceph的组合：我们将展示如何将Ceph存储连接到Kubernetes集群。

本文包含了一些实际示例，尽管为了易于理解而对其进行了一些简化。我们不会考虑安装和配置Ceph和Kubernetes集群。



您是否想知道它是如何工作的？





, Kubernetes, , , kubespray. Ceph — , , . , , 10 /.



, !



Ceph , :



ceph health
ceph -s


RBD :



ceph osd pool create kube 32
ceph osd pool application enable kube rbd


Kubernetes. Ceph CSI RBD. , , Helm.

, ceph-csi-rbd:



helm repo add ceph-csi https://ceph.github.io/csi-charts
helm inspect values ceph-csi/ceph-csi-rbd > cephrbd.yml


cephrbd.yml. ID IP- Ceph:



ceph fsid  #    clusterID
ceph mon dump  #    IP-


cephrbd.yml. PSP (Pod Security Policies). nodeplugin provisioner , , :



csiConfig:
  - clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
    monitors:
      - "v2:172.18.8.5:3300/0,v1:172.18.8.5:6789/0"
      - "v2:172.18.8.6:3300/0,v1:172.18.8.6:6789/0"
      - "v2:172.18.8.7:3300/0,v1:172.18.8.7:6789/0"

nodeplugin:
  podSecurityPolicy:
    enabled: true

provisioner:
  podSecurityPolicy:
    enabled: true


— Kubernetes.



helm upgrade -i ceph-csi-rbd ceph-csi/ceph-csi-rbd -f cephrbd.yml -n ceph-csi-rbd --create-namespace


, RBD !

Kubernetes StorageClass. Ceph.



Ceph kube:



ceph auth get-or-create client.rbdkube mon 'profile rbd' osd 'profile rbd pool=kube'


:



ceph auth get-key client.rbdkube


:



AQCO9NJbhYipKRAAMqZsnqqS/T8OYQX20xIa9A==


Secret Kubernetes — , userKey:



---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: ceph-csi-rbd
stringData:
  #        ,   
  #  Ceph. ID      ,
  #   storage class
  userID: rbdkube
  userKey: <user-key>


: 



kubectl apply -f secret.yaml


StorageClass:



---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: <cluster-id>
   pool: kube

   imageFeatures: layering

   #       
   #   .
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-rbd
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-rbd
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-rbd

   csi.storage.k8s.io/fstype: ext4

reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - discard


clusterID, ceph fsid, Kubernetes: 



kubectl apply -f storageclass.yaml


, PVC (Persistent Volume Claim):



apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc


, Kubernetes Ceph :



kubectl get pvc
kubectl get pv


! Ceph?

:



rbd ls -p kube
rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653  # ,  ,   ID ,


, RBD.

pvc.yaml 2Gi :



kubectl apply -f pvc.yaml


, , .



rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653

kubectl get pv
kubectl get pvc


, PVC . , Kubernetes PVC YAML: 



kubectl get pvc rbd-pvc -o yaml


:



message: Waiting for user to (re-)start a pod to finish file system resize of volume on node. type: FileSystemResizePending



, — .

, . PVC/PV .



Pod, :



---
apiVersion: v1
kind: Pod
metadata:
  name: csi-rbd-demo-pod
spec:
  containers:
    - name: web-server
      image: nginx:1.17.6
      volumeMounts:
        - name: mypvc
          mountPath: /data
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: rbd-pvc
        readOnly: false


PVC:



kubectl get pvc


, .



RBD ( – Rados Block Device), , . , , CephFS.

Ceph Kubernetes CSI CephFS.



Helm-:



helm inspect values ceph-csi/ceph-csi-cephfs > cephfs.yml


cephfs.yml. , Ceph:



ceph fsid
ceph mon dump


:



csiConfig:
  - clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
    monitors:
      - "172.18.8.5:6789"
      - "172.18.8.6:6789"
      - "172.18.8.7:6789"

nodeplugin:
  httpMetrics:
    enabled: true
    containerPort: 8091
  podSecurityPolicy:
    enabled: true

provisioner:
  replicaCount: 1
  podSecurityPolicy:
    enabled: true


, address:port. cephfs , v2.

httpMetrics ( Prometheus ) , nginx-proxy, Kubespray’. , , .



Helm- Kubernetes:



helm upgrade -i ceph-csi-cephfs ceph-csi/ceph-csi-cephfs -f cephfs.yml -n ceph-csi-cephfs --create-namespace


Ceph, . , CephFS . fs :



ceph auth get-or-create client.fs mon 'allow r' mgr 'allow rw' mds 'allow rws' osd 'allow rw pool=cephfs_data, allow rw pool=cephfs_metadata'


, :



ceph auth get-key client.fs


Secret StorageClass.

, RBD:



---
apiVersion: v1
kind: Secret
metadata:
  name: csi-cephfs-secret
  namespace: ceph-csi-cephfs
stringData:
  #     
  adminID: fs
  adminKey: <  >


:



kubectl apply -f secret.yaml


– StorageClass:



---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cephfs-sc
provisioner: cephfs.csi.ceph.com
parameters:
  clusterID: <cluster-id>

  #    CephFS,     
  fsName: cephfs

  # ()  Ceph,      
  # pool: cephfs_data

  # ()      Ceph-fuse
  # :
  # fuseMountOptions: debug

  # ()     CephFS  
  # . man mount.ceph     . :
  # kernelMountOptions: readdir_max_bytes=1048576,norbytes

  #       /  Ceph.
  csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-cephfs
  csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-cephfs
  csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-cephfs

  # ()     ceph-fuse (fuse), 
  #  ceph kernelclient (kernel).
  #   ,      ,
  #    ceph-fuse  mount.ceph
  # mounter: kernel
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - debug


clusterID Kubernetes:



kubectl apply -f storageclass.yaml




, , PVC:



---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-cephfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: csi-cephfs-sc


PVC/PV:



kubectl get pvc
kubectl get pv


CephFS, -. , .



Ceph :



#  
mkdir -p /mnt/cephfs

#     
ceph auth get-key client.admin >/etc/ceph/secret.key

#    /etc/fstab
# !!  ip     
echo "172.18.8.6:6789:/ /mnt/cephfs ceph name=admin,secretfile=/etc/ceph/secret.key,noatime,_netdev    0       2" >> /etc/fstab

mount /mnt/cephfs


, FS Ceph , . , - , .



, CephFS . Kubernetes PVC — , , 7Gi.



: 



kubectl apply -f pvc.yaml


, :



getfattr -n ceph.quota.max_bytes <-->


, , attr.



,



YAML , .

— . Ceph Kubernetes, :



Kubernetes c

RBD

RBD Kubernetes Ceph

RBD Kubernetes CSI

CephFS

CephFS Kubernetes CSI



Kubernetes Kubernetes , CephFS . GET/POST Ceph.



, Ceph. -, .



: , Southbridge, Certified Kubernetes Administrator, .



More articles:


All Articles