K8S 1.20+ nfs挂载sc部署，以及错误排查

一、nfs文件服务器部署

# 安装nfs服务器
yum install -y rpcbind nfs-utils
# 创建需要挂载的文件目录
mkdir /mnt/nfs
# 写入nfs挂载配置文件
echo " /mnt/nfs *(insecure,rw,sync,no_root_squash)" > /etc/exports
# 生效配置信息
exportfs -rv
# 启动nfs服务器
systemctl start rpcbind
systemctl start nfs-server
# 设为开机默认启动
systemctl enable rpcbind
systemctl enable nfs-server
# exportfs 检查nfs服务器状态
exportfs

二、直接使用nfs进行挂载

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: data
          mountPath: /usr/share/nginx/html
      volumes:
      - name: data
        nfs:
          path: /mnt/nfs # nfs服务器挂载的地址，可以挂载到子目录，例如：/mnt/nfs/nginx
          server: 10.10.10.10 # nfs服务器地址

三、nfs-client-provisioner挂载StorageClass

nfs-client-provisioner是一个Kubernetes的NFS外部provisioner，本身不提供NFS，需要现有的NFS服务器提供存储，可以直接将nfs与StorageClass进行绑定。

创建namespace

kubectl create ns nfs

创建部署文件

这里使用的是阿里云NAS文件服务器，其实本质也是NFS文件服务器进行演示

cat <<EOF | sudo tee nfs-client-provisioner.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-aliyun-nas
  labels:
    app: nfs-client-provisioner
  namespace: nfs
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"	# 作为默认StorageClass
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-aliyun-nas
  template:
    metadata:
      labels:
        app: nfs-aliyun-nas
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/nfs-subdir-external-provisioner:v4.0.2   # k8s 1.24 版本之后需要用这个镜像，否则会因为SelfLink被移除导致无法
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: aliyun-nas/nfs
            - name: NFS_SERVER
              value: xxxxx.cn-hangzhou.nas.aliyuncs.com # 阿里云NAS文件服务器路径
            - name: NFS_PATH
              value: /k8s_sc # NAS文件服务器挂载目录
      volumes:
        - name: nfs-client-root
          nfs:
            server: xxxxx.cn-hangzhou.nas.aliyuncs.com  # 阿里云NAS文件服务器路径，同`NFS_SERVER`保持一致
            path: /k8s_sc # NAS文件服务器挂载目录，同`NFS_PATH`保持一致
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-aliyun-nas-sc # StorageClass 名称
provisioner: aliyun-nas/nfs # 需要同`nfs-client-provisioner`参数保存一致`PROVISIONER_NAME`进行绑定
parameters:
  archiveOnDelete: "false"
EOF
# 部署
kubectl -n nfs apply -f nfs-client-provisioner.yaml

rbac授权

cat <<EOF | sudo tee rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["get"]
  - apiGroups: ["extensions"]
    resources: ["podsecuritypolicies"]
    resourceNames: ["nfs-provisioner"]
    verbs: ["use"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: nfs
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: nfs
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io
EOF
# 部署
kubectl -n nfs apply -f rbac.yaml

创建完成后效果

[root@k8s-1 nfs]# kubectl -n nfs get pods
NAME                                         READY   STATUS    RESTARTS   AGE
nfs-aliyun-nas-78c5dcdb5b-jck9z   1/1     Running   0          4d

[root@k8s-1 nfs]# kubectl get sc
NAME                                  PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-aliyun-nas (default)   aliyun-nas/nfs   Delete          Immediate           false                  4d

四、问题排查

StorageClass 创建成功，但是PVC一直Pending状态无法

# PVC创建时错误
kubectl -n test describe pvc test_pvc
waiting for a volume to be created, either by external provisioner "zkxy/nfs-client-provisioner" or manually created by system administrator

# nfs-client-provisioner pod 错误日志
kubectl -n nfs logs nfs-aliyun-nas-78c5dcdb5b-jck9z
unexpected error getting claim reference: selfLink was empty, can't make reference

K8S 1.20+版本处理方法，/etc/kubernetes/manifests/kube-apiserver.yaml 添加: - --feature-gates=RemoveSelfLink=false

sed -i -e  's/image:/- --feature-gates=RemoveSelfLink=false\n    image:/g'   /etc/kubernetes/manifests/kube-apiserver.yaml

K8S 1.24版本因为已经移除了selfLink， 添加: - --feature-gates=RemoveSelfLink=false会导致K8S无法启动，可以直接使用以下镜像版本直接解决。

registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/nfs-subdir-external-provisioner:v4.0.2