Почему elasticsearch пишет no persistent volumes available for this claim and no storage class is set для local-storage?

Question

[Антон Пацев]

Добрый день!
Делаю по статье
https://scalablesystem.design/ds101/kubernetes-ins...

Имеется 3 ноды Centos 7.
На каждой создана папка /mnt/disks
mkdir -p /mnt/disks/vdb1

Примонтирован 50ГБ диск в эту папку
mount /dev/vdb1 /mnt/disks/vdb1

Проверяем

mount | grep mnt
/dev/vdb1 on /mnt/disks/vdb1 type xfs (rw,relatime,attr2,inode64,noquota)

Версия Kubernetes - 1.12
git clone https://github.com/kubernetes-incubator/kubespray.git
cd kubespray
cp -rfp inventory/sample inventory/mycluster
cd inventory/mycluster/
declare -a IPS=(10.233.60.72 10.233.60.76 10.233.60.75)
CONFIG_FILE=inventory/mycluster/hosts.ini python3 contrib/inventory_builder/inventory.py ${IPS[@]}
pip install -r requirements.txt

Добавляем в конец файла inventory/mycluster/group_vars/k8s-cluster.yml

helm_version: "v2.9.1"
helm_enabled: true
docker_dns_servers_strict: no
local_volume_provisioner_enabled: true

Запускаем установку

ansible-playbook -u apatsev -i inventory/mycluster/hosts.ini cluster.yml -b

Проверяем наличие StorageClass

kubectl get storageclass
NAME            PROVISIONER                    AGE
local-storage   kubernetes.io/no-provisioner   18m

Проверяем PV

kubectl get pv
NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS    REASON   AGE
local-pv-5bec36e4   49Gi       RWO            Delete           Available           local-storage            2m15s
local-pv-aa880f42   49Gi       RWO            Delete           Available           local-storage            2m15s
local-pv-b6ffa66b   49Gi       RWO            Delete           Available           local-storage            2m15s

Добавляем репо kubernetes-charts-incubator

helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator
"incubator" has been added to your repositories

Запускаем установку elasticsearch

helm install incubator/elasticsearch --name my-release --set data.storageClass=local-storage,data.storage=10Gi

Смотрим PVC

kubectl get pvc
NAME                                     STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-my-release-elasticsearch-data-0     Pending                                                     3m15s
data-my-release-elasticsearch-master-0   Pending                                                     3m15s

Смотрим describe PVC

kubectl describe pvc data-my-release-elasticsearch-data-0
Name:          data-my-release-elasticsearch-data-0
Namespace:     default
StorageClass:  
Status:        Pending
Volume:        
Labels:        app=elasticsearch
               component=data
               release=my-release
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
Events:
  Type       Reason         Age               From                         Message
  ----       ------         ----              ----                         -------
  Normal     FailedBinding  2s (x4 over 31s)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

Смотрим поды:

kubectl get po  --namespace kube-system | grep local-volume-provisioner
local-volume-provisioner-9wmqm                       1/1     Running   0          7m43s
local-volume-provisioner-f9gnn                       1/1     Running   0          7m43s
local-volume-provisioner-zpmdn                       1/1     Running   0          7m43s

Логи:

kubectl logs local-volume-provisioner-9wmqm --namespace kube-system
I1009 08:34:19.214312       1 common.go:259] StorageClass "local-storage" configured with MountDir "/mnt/disks", HostDir "/mnt/disks", BlockCleanerCommand ["/scripts/quick_reset.sh"]
I1009 08:34:19.214560       1 main.go:42] Configuration parsing has been completed, ready to run...
I1009 08:34:19.214991       1 common.go:315] Creating client using in-cluster config
I1009 08:34:19.245635       1 main.go:52] Starting controller
I1009 08:34:19.245659       1 controller.go:42] Initializing volume cache
I1009 08:34:19.249252       1 populator.go:85] Starting Informer controller
I1009 08:34:19.249273       1 populator.go:89] Waiting for Informer initial sync
I1009 08:34:20.249475       1 controller.go:72] Controller started
I1009 08:34:20.250066       1 discovery.go:254] Found new volume of volumeMode "Filesystem" at host path "/mnt/disks/vdb1" with capacity 53659832320, creating Local PV "local-pv-5bec36e4"
I1009 08:34:20.301162       1 discovery.go:280] Created PV "local-pv-5bec36e4" for volume at "/mnt/disks/vdb1"
I1009 08:34:20.301826       1 cache.go:55] Added pv "local-pv-5bec36e4" to cache
I1009 08:34:20.343959       1 cache.go:64] Updated pv "local-pv-5bec36e4" to cache
I1009 08:34:20.353803       1 cache.go:64] Updated pv "local-pv-5bec36e4" to cache

Смотрим df | grep disks у пода local-volume-provisioner-9wmqm

kubectl --namespace kube-system  exec local-volume-provisioner-9wmqm -- df | grep disks
/dev/vda1        9774628 6355168   2961036  69% /mnt/disks
/dev/vdb1       52402180   32944  52369236   1% /mnt/disks/vdb1

Что интересно диск с системой тоже замонтировался

mount |grep /dev/vda1
/dev/vda1 on / type ext4 (rw,relatime,data=ordered)
/dev/vda1 on /var/lib/kubelet/pods/89d44bff-cb9e-11e8-b52a-fa163e47d93f/volume-subpaths/config/elasticsearch/0 type ext4 (rw,relatime,data=ordered)

Как установить/настроить приложение (например, elasticsearch) чтобы оно использовало local-storage?

Answer 1

[Антон Пацев]

похоже что то с elasticsearch

helm install stable/prometheus --name stable-prometheus --set server.persistentVolume.storageClass=local-storage --set alertmanager.persistentVolume.storageClass=local-storage

kubectl get pvc
NAME                             STATUS   VOLUME              CAPACITY   ACCESS MODES   STORAGECLASS    AGE
stable-prometheus-alertmanager   Bound    local-pv-aa880f42   49Gi       RWO            local-storage   9s
stable-prometheus-server         Bound    local-pv-5bec36e4   49Gi       RWO            local-storage   9s

Ошибка в этой строке
$ helm install incubator/elasticsearch --name my-release --set data.storageClass=ssd,data.storage=100Gi

в официальном readme https://github.com/helm/charts/tree/master/incubat...

fix pull request
https://github.com/helm/charts/pull/8294