Codefresh On-Prem: no space left on device errors in build log
Overview
After some time you can start receiving no space left on device
errors in the build logs of your codefresh on premises. The article is only applicable in case of using the built-in cf-builder runtime. We strongly recommend using codefresh runner runtimes for any production workload
Details
For some specific cases our built-in clean-up scripts might not be optimal for your storage size and type of workloads
Exec an interactive shell in all of your cf-builder-xxx pods and check the disk space used in the output of the following command where X is the builder pod number:
kubectl exec -it -ncodefresh cf-builder-X -- sh -c 'df -i | grep /var/lib/docker'
and
kubectl exec -it -ncodefresh cf-builder-X -- sh -c 'df -h | grep /var/lib/docker'
if confirmed the disk is full you can try the following: exec in every builder pod
docker rmi -f $(docker images -f "dangling=true" -q)
wait before the operation completes
check the disk space one more time to make sure you freed up enough space
Notes
If this issue happens quite often, please open a ticket for the support team