Best strategy for saving huge amount of small files

[phil]

Hello,

we are using a Open-E Storage Server with an Dual XEON 5140 (Dual Core), 4GB RAM and 5TB SATA Disk drives in RAID6 Mode. This server should be used for a custom application which produces a huge amout a small files (1.000.000 20kb files in 2h).

I made some tests, putting all files in the same directory, this slows down the system if you have more that a few thousands files in it. My next step was creating 1.000 subfolders:

0/0/0
0/0/1
...
9/9/9

Based on the last numbers of the file id I saved all files to corresponding folder. But after a while, maybe one hour and about 400.000 files, everything gets slow again....

Maybe I should create 10.000 subfolders? What are your tips?

Btw. is there any option to tune the file system for small files? Can I set options like noatime?

[SeanLeyne]

How are you connecting to the DSS (Via iSCSI volumne, NAS Share, NFS?)?

If via iSCSI, what file system have you used (FAT, NTFS, EXT3)?

[phil]

it is a NAS (I guess samba) share... my application runs on windows and uses \\storage\spool as a "target"

[netsyphon]

There is something about blocksize and filesystem that may help you out more than anything else.

i think open-e uses XFS

[Robotbeat]

Yeah, I think it's XFS, which has variable block size (can be good for small files). Does Open-E have any tuning parameters that can help optimize for many small files? We have a customer who's looking to use the open-e box as a storage pool to store lots (millions) of small (~16KB) files.

[Robotbeat]

BTW, he's interesting in using it as a NAS, not as an iSCSI box (although we'd probably just get him the DSS version).