One of my servers went down today. Problems started with deployment on jenkins that reported “No space left on device”, although partition was not nearly full. If you ever run into such trouble – most likely you have too many small or 0-sized files on your disk, and while you have enough disk space, you have exhausted all available Inodes. Below is the solution for this problem.
1. check available disk space to ensure that you still have some:
2. check available Inodes
If you have IUse% at 100 or near, then huge number of small files is the reason for “No space left on device” errors.
3. find those little bastards
$ for i in /*; do echo $i; find $i |wc -l; done
This command will list directories and number of files in them. Once you see a directory with unusually high number of files (or command just hangs over calculation for a long time), repeat the command for that directory to see where exactly the small files are.
$ for i in /home/*; do echo $i; find $i |wc -l; done
4. once you found the suspect – just delete the files
$ sudo rm -rf /home/bad_user/directory_with_lots_of_empty_files
You’re done. Check the results with df -i command again. You should see something like this:
Leave a Reply