How many times have we used a program, and then the program bombs out for some unearthly reason. You look at the log, and discover that the log says something along the lines of:

​

• "Cannot create file ...", or

• "System Error: unable to write output", or

• "Program Error: See administrator log"

 

You get the picture - something is holding your system up and the process cannot continue (incidentally, my favorite is the last type of error - which gives no help whatsoever, and usually the administrator log is just as helpful.

​

I had a situation like this recently, whereby, a SALT master server, suddenly stopped working. It was all very odd. I was using the server to configure and install the worker servers, when suddenly some of the servers were set up correctly, and then suddenly the rest of them failed to configure. 

​

Of course, the SALT master output some sort of obscure message that it couldn't do the operation. So we started to look for the problem.

​

I wont go thru all the sordid details, but eventually we found the problem. The disk was full, or was it ? Looking at the output of 'df' this is what we saw:

Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/disk1       8124856 3562544   4142936  47% /
tmpfs            1864424      16   1864408   1% /dev/shm

​

That doesn't look very full - I said to myself. But then my co-worker, suggested I run a different operation:

[root@my-server]# df -i
Filesystem     Inodes  IUsed  IFree IUse% Mounted on
/dev/disk1     524288 524288      0  100% /
tmpfs          466106      5 466101    1% /dev/shm

​

In all my years, of working with linux and unix ( and that is at least 25 years) - I never ever had this happen to me - my inodes were full. In all my years i had taken those inodes for granted, and just thought that when my disk gets full, I can just check the space with the standard 'df -k' and that is good enough. Well you learn something new every day ! - and I wont forget this one in a long time.

​

A quick refresher on Inodes - An inode - is a metadata block of information that describes a filesystem block. From wikipedia, it says:
"A file system relies on data structures about the files, beside the file content. The former are called metadata—data that describe data. Each file is associated with an inode, which is identified by an integer number, often referred to as an i-number or inode number."

In essence, think of it as a metadata block of information describing the real data on the filesystem. 

​

In our case what happened ? We had a lot of SALT job information in many many small files - in fact there were so many small files on the SALT master, that the inode table just filled up . As a result, the physical data of the disk was not full, but the inode table was full, hence, when the SALT process tried to write another file, it couldn't because there was no more indoe table space left.

​

In our case the solution was simple. Stop the SALT master, delete the unnecessary files and then re-start the SALT master program.

​

How can you find which directories contain the most files ? - A quick google search, brought me to this article, with the following piece of code:

​

#!/bin/bash

if [ $# -ne 1 ];then
  echo "Usage: `basename $0` DIRECTORY"
  exit 1
fi

echo "Wait a moment if you want a good top of the bushy folders..."

find "$@" -type d -print0 2>/dev/null | while IFS= read -r -d '' file; do 
    echo -e `ls -A "$file" 2>/dev/null | wc -l` "files in:\t $file"
done | sort -nr | head | awk '{print NR".", "\t", $0}'

exit 0

 

Example output would look like this:

​

Wait a moment if you want a good top of the bushy folders...
1.       1234 files in:  ./media
2.       499 files in:   ./files
3.       195 files in:   ./icons
4.       10 files in:    .

​

​

Another, useful one liner that I like to use when looking for all those really large files that fill up the space - is the following:

find {/path/to/directory/} -xdev  -type f -size +{size-in-kb}k -exec ls -lh {} \; | awk '{ print $9 ": " $5 }' 

​

for example:

​

find / -xdev -type f - size +50000k -exec ls -lh {}\; | awk '{ print $9 ": " $5 }' 

​

Can I increase the number of Inodes on the system ? In a one word answer - No, but there are ways of working around this, and googling the net will provide you with a large number of options. 

 

For example, you can create a new filesystem with the '-N' option, and give it a large Inode table -

​

From the man page:

       -N number-of-inodes
              Overrides  the  default calculation of the number of inodes that should be
              reserved for the filesystem (which is based on
              the number of blocks and the bytes-per-inode ratio).  This allows the user  
              to specify  the  number  of  desired  inodes
              directly.

 

You can find some useful example on this link that can help you set up your filesystem correctly.

​

In Summary: Your disk might not appear full, but there are plenty of things going on in the operating system and in the filesystem itself that might cause things to go wrong. Check every possibility to find what could be filling up your system, or causing your process to fail. 

​

Comments welcome to: markpragerinfo@gmail.com

​