BTRFS is the future filesystem of linux, much like ZFS. It will eventually support transparent encryption.
As far as user-interaction for entering keys and such, well, that's probably best left up to the client-side. Really, for ultimate security, the client should encrypt the files themselves.