some general questions..

[Raudi]

At my experience it shows this behaviour:

In BlockIO the storage performance at the iSCSI initiator is 100% the performance of the RAID. Only a few MB will be used for cache.

In FileIO the storage performance at the iSCSI initiator is much faster than the performance of the RAID. The complete free memory will be used for cache so the read and write access to the RAID will be optimized.

In my case i had 6x 750GB S-ATA II in RAID5 and the performance in BlockIO is not usable when access the RAID with more than two Virtual Machines. When i use FileIO the slow S-ATA RAID is much faster.

There will be changes in the future that there will be used more cache in BlockIO. But at this time is FileIO the only soloution when using VMware ESX.

BlockIO might be faster then FileIO, but only when access the RAID with one initiator and one task.

Best regards
Stefan

[netsyphon]

22 disk raid6 with 1tb drives? whoah.

think about rebuild times?

you probably want to divide this up into at least 2-3 arrays rather than 1 giant raid 6.

[warmduscher]

Hi everybody..

@Raudi: my question pointed to your issue :-) i think we will test some scenarios with block and with file IO

@netsyphon: yes, whoah :-) we decided to use as many spindles as we can due to high IO requirements. we discussed rebuilding times and we decided that a rebuild up to 30h is OK. right now we have a 12 disk 5 TB raid6 on a slower controller and it takes up to 10h for a complete rebuild with normal workload.

if this big block is going to be critical we can run all our VMs on an alternative iscsi path with our secondary dss which has replicated volumes from out primary dss

if anything else is going wrong we have an additional backup server with snapshots of all VMs , so data-loss shouldn't be any problem

our vms are mostly development systems, so downtime is not really critical for us.

@shon: we will do some tests with 2x quadcore vs. 2x dualcore xeons, i think its interesting which kind of processor does more IO in multithreaded applications like ESX farming

thanks for the moment.

regards, lutz

[Raudi]

I think the bottleneck woll be the harddrives not the CPU...

A few days ago i made a migration of 6x 750GB to 8x 750GB and this takes nearly one week to rebuild. Perhaps a rebuild after a drive failure is faster than a migration... Until now i didn't had a drive failure.

Best regards
Stefan

[warmduscher]

Hi Raudi,

what kind of controller do you use?
we had similar problems with 3ware products (eg 9550), on high IO situation the r/w performance dropped down to constant 8MB/s for minutes, then rising again for some time, again dropping...

so we switched from 3ware with raid5 to areca raid6 and we noticed a massive perfromance gain compared to 3ware, even with raid6!

regards,

lutz

[To-M]

Found others having issues with performance on the 3Ware 9550 as well, not sure if this will help in your incase but others claimed it did when they updated to latest.

http://www.3ware.com/KB/article.aspx?id=14956

[warmduscher]

Hi Todd,

thanks for that interesting info. a little bit too late for us, we switched to areca mid 2006 :-)

Regards,
Lutz

[Raudi]

what kind of controller do you use?
we had similar problems with 3ware products (eg 9550), on high IO situation the r/w performance dropped down to constant 8MB/s for minutes, then rising again for some time, again dropping...

I use a 3ware 9550SXU-16ML... This sounds not good, but i hope the issue is fixed. Uses here someone a 3ware too and can tell his experience?

Regards
Stefan

[Robotbeat]

We used a 3ware 95xx series card (first half of 2007), and we had horrible performance. We also switched to an Areca controller.