'No space left on device' inode problem?

Two identical DSS 6 systems here:

Model: Data Storage Software V6
Version: 6.0up97.8101.6325 64bit
Release date: 2012-06-15
Licensed Storage Capacity: unlimited

The first system provides a client-accessible NFS mount; the second system is used as an offsite mirror of the first, not directly accessible to clients.

On each system, we recently added a RAID volume and used it to increase the size of the logical volume. ‘df’ shows an abundance of space available for use:

# df -h /mnt/tmp/
Filesystem Size Used Avail Use% Mounted on
gverc:/GCC_archive 107T 29T 79T 27% /mnt/tmp

However, on the first system, we found that NFS clients were getting errors like this:

# mkdir /mnt/tmp/zzztestzzz
mkdir: cannot create directory `/mnt/tmp/zzztestzzz': No space left on device

I assume it’s an inode issue, because when you remove a number of file/directory objects, the same number of objects can subsequently be created successfully.

I found this KB article ...

http://kb.open-e.com/Expanding-full-...o-it_1063.html

... and tried the suggested solution, but it didn’t work. Or maybe I didn’t do it right.

What should be done to solve this problem? And why might it affect just one of our systems?

You might want to update to the latest version of the DSS V6 w/ build 7337 and perform a Repair Filesystem after as there could have been something that might happened after the increase of the volume re-sizing. It could also be something else but we would need to see the log file from that system and you can submit a support ticket and we can review them.

You need to apply a small update to mount the shares with inode64.
Please also note you must set FSID manually for each share (in webGUI).
See your inbox for update.

thanks for the replies, but now I'm confused about how to proceed. Should I submit a support ticket or apply the update? If the latter, how do I apply the update, exactly, and is there any danger of data loss? Sorry to admit that I have very limited experience with these systems.

you may upload the update in the webGUI at MAINTENANCE-->Software updates. Reboot for it to be applied.
After reboot the system will mount with inode64. You will also need to go to each share and set the FSID for NFS. This is a number, and unique to each share.
If this solves your problem, great. If not, you may open a support ticket.
Also be aware that changing to inode64 it is recommended to stay with it. If in the future if you remove the small update some data may become inaccessible.

okay, thanks. That sounds fairly straightforward, with the exception of setting the FSID. Where/how is that done?

It is set in the webGUI, CONFIGURATION-->NAS RESOURCES, shares. Select the share and you will see a box to enter the ID.

that's what I assumed, but I can't locate such a box. Would it only be available/visible after the update is applied?

you will not see it until the update is applied and the server rebooted.

after successfully applying the update and setting FSIDs for the two shares on the system (FSID 1 and FSID 2, respectively), I remounted the share in question and found that I still get the "No space left on device" error. Did I miss something? Should I try updating the the latest version of DSS 6 followed by a filesystem repair as suggested by Todd? Or ... ?

yes please, do update and then a file repair and see what results are.

today I updated the system to ...
6.0up98.8101.7337 64bit
... and did a filesystem repair. Unfortunately the "no space" behavior persists. I notice that the NFS option to specify the FSID has disappeared. Does the small update need to be reapplied? followed by another filesystem repair? Is there anything else I should try? Or is it time to open a support ticket?

after a system update, the small update needs to be re-applied.

the small update has been re-applied, and the FSIDs are now present/displayed again, but I still get the error. What next?

I suspect the issue is not inodes...
best to open a support case in the morning, so an engineer can take a look at the logs.