The new forecast server I’m working on for Seasonality users is using the filesystem heirarchy as a form of database instead of PostgreSQL. This will slow down the forecast generation code a bit, because I’m writing a ton of small files instead of letting Postgres optimize disk I/O. However, reading from the database will be lightning fast, because filesystems are very efficient at traversing directory structures.
The problem I ran into was that I was quickly hitting the maximum number of files on the filesystem. The database I’m working on creates millions of files to store its data in, and I was quickly running out of inodes.
Earlier today I installed a fresh copy of Ubuntu on a virtual machine where the final forecast server will reside. Of course I forgot to increase the number of inodes before installing the OS on the new partition. Unfortunately, there is no way to add more inodes to a Linux ext4 filesystem without reformatting the volume. Luckily I caught the problem pretty early and didn’t get too far into the system setup.
To fix the issue, I booted off the Ubuntu install ISO again and chose the repair boot option. Then I had it start a console without selecting a root partition (if you select a root partition, it will mount the partition and when I tried to unmount it, the partition was in use). This let me format the partition with an increased number of inodes using the -N flag in mkfs:
mkfs.ext4 -N 100000000 /dev/sda1
That ought to be enough. 🙂 After that, I was able to install Ubuntu on the new partition (just making sure not to select to format that same partition again, wiping out your super-inode format).
The forecast server is coming along quite well. I’m hoping to post more about how it all works in the near future.