Static Volumes#

Assigning PVs#

For security reasons, you might want to use a storage class that does not perform automatic provisioning of PVs. In that case, you want to reference a pre-created volume in the PVC. In nplus, you can do so by setting the volumeName in the values.

Please review values.yaml as an example:

database:
  mounts:
    data:
      volumeName: "pv-{{ .component.fullName }}-data"
nstl:
  mounts:
    data:
      volumeName: "pv-{{ .component.fullName }}-data"

You can also set the environment config volume. Please refer to the environment documentation for that.

helm install \
  --values samples/environment/demo.yaml \
  --values samples/static/values.yaml
  sample-static nplus/nplus-instance

Creating PVs#

https://github.com/ceph/ceph-csi/blob/devel/docs/static-pvc.md

Data Disk:#

Create a pool on your cep cluster
```
ceph osd pool create k-lab 64 64
```
Create a block device pool
```
rbd pool init k-lab
```

Create an image

rbd create -s 50G k-lab/pv-sample-static-database-data
rbd create -s 50G k-lab/pv-sample-static-nstl-data
rbd ls k-lab | grep pv-sample-static-

Resize:

rbd resize --size 50G k-lab/pv-no-provisioner-database-data --allow-shrink

Create a Subvolume (FS)

ceph fs subvolume create cephfs pv-no-provisioner-rs-file --size 53687091200

Get the path of the subvolume

ceph fs subvolume getpath  cephfs pv-no-provisioner-rs-file

Troubleshooting#

kubectl describe pv/pv-no-provisioner-rs-file pvc/no-provisioner-rs-file
kubectl get volumeattachment

PV Manifests#

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-no-provisioner-database-data
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  csi:
    driver: rook-ceph.rbd.csi.ceph.com
    fsType: ext4
    nodeStageSecretRef:
      # node stage secret name
      name: rook-csi-rbd-node
      # node stage secret namespace where above secret is created
      namespace: rook-ceph-external
    volumeAttributes:
      # Required options from storageclass parameters need to be added in volumeAttributes
      clusterID: rook-ceph-external
      pool: k-lab
      staticVolume: "true"
      imageFeatures: layering
      #mounter: rbd-nbd
    # volumeHandle should be same as rbd image name
    volumeHandle: pv-no-provisioner-database-data
  persistentVolumeReclaimPolicy: Retain
  # The volumeMode can be either `Filesystem` or `Block` if you are creating Filesystem PVC it should be `Filesystem`, if you are creating Block PV you need to change it to `Block`
  volumeMode: Filesystem
  storageClassName: ceph-rbd
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-no-provisioner-nstl-data
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  csi:
    driver: rook-ceph.cephfs.csi.ceph.com
    fsType: ext4
    nodeStageSecretRef:
      # node stage secret name
      name: rook-csi-rbd-node
      # node stage secret namespace where above secret is created
      namespace: rook-ceph-external
    volumeAttributes:
      # Required options from storageclass parameters need to be added in volumeAttributes
      clusterID: rook-ceph-external
      pool: k-lab
      staticVolume: "true"
      imageFeatures: layering
      #mounter: rbd-nbd
    # volumeHandle should be same as rbd image name
    volumeHandle: pv-no-provisioner-nstl-data
  persistentVolumeReclaimPolicy: Retain
  # The volumeMode can be either `Filesystem` or `Block` if you are creating Filesystem PVC it should be `Filesystem`, if you are creating Block PV you need to change it to `Block`
  volumeMode: Filesystem
  storageClassName: ceph-rbd
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-no-provisioner-rs-file
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 50Gi
  csi:
    driver: cephfs.csi.ceph.com
    nodeStageSecretRef:
      name: rook-csi-cephfs-secret
      #rook-csi-cephfs-node
      namespace: rook-ceph-external
    volumeAttributes:
      # Required options from storageclass parameters need to be added in volumeAttributes
      clusterID: rook-ceph-external
      fsName: cephfs
      pool: cephfs_data
      staticVolume: "true"
      # rootPath kriegt man per ceph fs subvolume getpath  cephfs pv-no-provisioner-rs-file 
      rootPath: "/volumes/_nogroup/pv-no-provisioner-rs-file/3016f512-bc19-4bfb-8eb2-5118430fbbe5"
      #mounter: rbd-nbd
    # volumeHandle should be same as rbd image name
    volumeHandle: pv-no-provisioner-rs-file
  persistentVolumeReclaimPolicy: Retain
  # The volumeMode can be either `Filesystem` or `Block` if you are creating Filesystem PVC it should be `Filesystem`, if you are creating Block PV you need to change it to `Block`
  volumeMode: Filesystem
  storageClassName: cephfs

Files#

Download all files of this sample

build.sh#

#!/bin/bash
#
# This sample script builds the example as described. It is also used to build the test environment in our lab,
# so it should be well tested.
#

# Make sure it fails immediately, if anything goes wrong
set -e

# -- ENVironment variables:
# CHARTS: The path to the source code
# DEST: The path to the build destination
# SAMPLE: The directory of the sample
# NAME: The name of the sample, used as the .Release.Name
# KUBE_CONTEXT: The name of the kube context, used to build this sample depending on where you run it against. You might have different Environments such as lab, dev, qa, prod, demo, local, ...

# Check, if we have the source code available
if [ ! -d "$CHARTS" ]; then
    echo "ERROR Building $SAMPLE example: The Charts Sources folder is not set. Please make sure to run this script with the full Source Code available"
    exit 1
fi
if [ ! -d "$DEST" ]; then
    echo "ERROR Building $SAMPLE example: DEST folder not found."
    exit 1
fi
if [ ! -d "$CHARTS/instance" ]; then
    echo "ERROR Building $SAMPLE example: Chart Sources in $CHARTS/instance not found. Are you running this script as a subscriber?"
    exit 1
fi

# Set the Variables
SAMPLE="static"
NAME="sample-$SAMPLE"

# Output what is happening
echo "Building $NAME"

# Create the manifest
mkdir -p $DEST/instance
helm template --debug \
     --values $SAMPLES/application/empty.yaml \
     --values $SAMPLES/environment/$KUBE_CONTEXT.yaml \
     --values $SAMPLES/resources/$KUBE_CONTEXT.yaml \
     --values $SAMPLES/static/values.yaml \
     $NAME $CHARTS/instance > $DEST/instance/$SAMPLE.yaml
# Adding the static PV to the manifest
echo -e "\n---\n" >> $DEST/instance/$SAMPLE.yaml
cat $SAMPLES/static/pv-nfs.yaml >> $DEST/instance/$SAMPLE.yaml

pv-nfs.yaml#

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-sample-static-database-data
spec:
  # -- set an empty string must be explicitly set otherwise default StorageClass will be set
  # see https://kubernetes.io/docs/concepts/storage/persistent-volumes/
  storageClassName: ""
  # -- make sure, this PV may only by bound to a specific claim
  claimRef:
    name: sample-static-database-data
    namespace: lab
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 50Gi
  nfs:
    path: /exports/k8s/sample-static-database-data
    server: 10.14.5.4
  persistentVolumeReclaimPolicy: Retain
  # The volumeMode can be either `Filesystem` or `Block` if you are creating Filesystem PVC it should be `Filesystem`, if you are creating Block PV you need to change it to `Block`
  volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-sample-static-nstl-data
spec:
  # -- set an empty string must be explicitly set otherwise default StorageClass will be set
  # see https://kubernetes.io/docs/concepts/storage/persistent-volumes/
  storageClassName: ""
  # -- make sure, this PV may only by bound to a specific claim
  claimRef:
    name: sample-static-nstl-data
    namespace: lab
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 50Gi
  nfs:
    path: /exports/k8s/sample-static-database-data
    server: 10.14.5.4
  persistentVolumeReclaimPolicy: Retain
  # The volumeMode can be either `Filesystem` or `Block` if you are creating Filesystem PVC it should be `Filesystem`, if you are creating Block PV you need to change it to `Block`
  volumeMode: Filesystem

pv.yaml#

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-sample-static-database-data
spec:
  # -- set an empty string must be explicitly set otherwise default StorageClass will be set
  # see https://kubernetes.io/docs/concepts/storage/persistent-volumes/
  storageClassName: "" 
  # -- make sure, this PV may only by bound to a specific claim
  claimRef:
    name: sample-static-database-data
    namespace: lab
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  csi:
    driver: rook-ceph.rbd.csi.ceph.com
    fsType: ext4
    nodeStageSecretRef:
      # node stage secret name
      name: rook-csi-rbd-node
      # node stage secret namespace where above secret is created
      namespace: rook-ceph-external
    volumeAttributes:
      # Required options from storageclass parameters need to be added in volumeAttributes
      clusterID: rook-ceph-external
      pool: k-lab
      staticVolume: "true"
      imageFeatures: layering
      #mounter: rbd-nbd
    # volumeHandle should be same as rbd image name
    volumeHandle: pv-sample-static-database-data
  persistentVolumeReclaimPolicy: Delete
  # The volumeMode can be either `Filesystem` or `Block` if you are creating Filesystem PVC it should be `Filesystem`, if you are creating Block PV you need to change it to `Block`
  volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-sample-static-nstl-data
spec:
  # -- set an empty string must be explicitly set otherwise default StorageClass will be set
  # see https://kubernetes.io/docs/concepts/storage/persistent-volumes/
  storageClassName: "" 
  # -- make sure, this PV may only by bound to a specific claim
  claimRef:
    name: sample-static-nstl-data
    namespace: lab
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  csi:
    driver: rook-ceph.rbd.csi.ceph.com
    fsType: ext4
    nodeStageSecretRef:
      # node stage secret name
      name: rook-csi-rbd-node
      # node stage secret namespace where above secret is created
      namespace: rook-ceph-external
    volumeAttributes:
      # Required options from storageclass parameters need to be added in volumeAttributes
      clusterID: rook-ceph-external
      pool: k-lab
      staticVolume: "true"
      imageFeatures: layering
      #mounter: rbd-nbd
    # volumeHandle should be same as rbd image name
    volumeHandle: pv-sample-static-nstl-data
  persistentVolumeReclaimPolicy: Delete
  # The volumeMode can be either `Filesystem` or `Block` if you are creating Filesystem PVC it should be `Filesystem`, if you are creating Block PV you need to change it to `Block`
  volumeMode: Filesystem

values.yaml#

database:
  mounts:
    data:
      volumeName: "pv-{{ .component.fullName }}-data"
nstl:
  mounts:
    data:
      volumeName: "pv-{{ .component.fullName }}-data"


# # mon:
# #   mounts:
# #     data:
# #       volumeName: "pv-{{ .component.fullName }}-data"
# # pipeliner:
# #   mounts:
# #     data:
# #       volumeName: "pv-{{ .component.fullName }}-data"
# # rs:
# #   mounts:
# #     file:
# #       volumeName: "pv-{{ .component.fullName }}-file"