Discussion about speed and observation

Printable View

08-16-2007, 02:19 PM
vkeven

Discussion about speed and observation

Hi , I'm currently doing some testing to get more speed from ISCSI targe of Open-E , I observed some things that I would like to share with Open-E team and people's here.I tested on the same hardware SanMelody,Starwind from rocketvision and Open-E 1.23(2763).

Target

Dell PowerEdge 2800
2GB Ram
PERC 4/SC with write-back enable 33GB disk for boot and RAID5 280GB for Local Data
Broadcom 5703 PCI-X 1GB/s
Windows 2003 server for starwind and sanmelody test
Dual-Xeon 3.6
Apple Xraid with 3TB connect over LSI Logic Fiber 2GB/s ( 7202-LXP ) ( for ISCSI Target )

Local bench speed of this setup is 90MB/s burst rate , and 60MB/s constant read speed ( mesure with HDtach) , for the real world test I copied a 2GB files ( 2097MB exactly ) from built-in data hard disk to AppleXraid and it take 30s so this mean approximatly 69.9MB/s real world speed.

If I use Starwind or SanMelody I can copy my 2GB file over ISCSI target and it take from 35s to 40s ( 52MB/s to 59MB/s ) so it close to the maximum of my hardware capacity.

The things that causing me some questions is that Open-E change speed drasticly if file size is over the maximum ram of my system , this is my observation

If I copy the same 2GB files over Open-E ISCSI Target as soon as I reach approximately 3/4 of the file size the speed change drasticly , it take 1m10s to copy it so 29MB/s. Now if I take a 750MB files it take 15s so 50MB/s now close to other target software on same hardware. Its look like Open-E does'nt used RAM's the same way as other target. I don't know but I think it could be a Swap Area problem , when I create a volume on Open-E the swap space of 4GB is put on same vg as target so if I copy a 2GB file my system ram reach is maximum and start to swap but on same volume as the target, now if I copy a 750MB file the system ram is enough to buffer it entirely so this is probably why the speed is the same as other target software , on its part windows 2003 take his boot drive for swap and Open-E is load enterly in RAM so it cannot use his local drive for swapping purpose. Now my question is , do you think that it could be possible to have swap space locate on different vg that the ISCSI target is using? This theory could be apply on any RAM size and FC or ISCSI Target so 1GB Ram mean that file over 750MB in size will start to causing some speed issue also.

Sorry for my english and thanks for your time!:)
Keven
08-16-2007, 02:33 PM
vkeven

I repeat the test with 1.5GB file ( 1519MB ) and it take 30s ( 50.6MB/s ) so I think that my theory is correct , if I follow this theory this mean that you need to have enough RAM to buffer every simultaneous writing to all target if you want the maximum speed of your hardware. I think that possibility to put swap space on a separate drive like SAS 15K could be a great way to reach constant good speed without putting 16GB of RAM in your target server, Open-E do you think that its possible?

Keven
Have a good day!:cool:

PS: I'm Open to do some test.
08-16-2007, 04:28 PM
netsyphon

PS - me too..
08-16-2007, 07:17 PM
vkeven

Finally I put 5GB of RAM in my server but the same problem happen so I delete my iscsi target , deleted my iscsi volume , recreate every thing from scratch , used internal Dell Raid on 280GB target in Basic and GPT , used 2TB volume size on Xraid in Basic Disk and GPT, used 2.5TB in GPT and I could not reproduce the problem so the only options that I did'nt test was the initialize when you create an ISCSI volume, as soon as I recreated a volume with initialize fonction TADA problem come back , I created a 8GB volume with initialize function turn on and from this point my target become sluggish and irregular when I used it , if I create a volume without initialize everything is working like is suppose to so now I don't know where to look at:(
08-16-2007, 07:26 PM
vkeven

It's look like I'm talking with myself ( Yes I admit that I'm a little crazy):D

Ok now I have 2 volume with ISCSI target both locate on my AppleXraid over fiber

1 x 8GB with initialize and 1 x 8GB without initialize both are mount on windows 2003 with MS ISCI initiator ( I tried also with startport ) one is H:(not initialize) and one is G:(initialize) , if I copy on G: its become sluggish during copy and windows go from 40s to 2min to 40s to 1M for evaluation copy time, if I used H: windows write 45s and it take 35s to 40s steady.

Suggestion,does initialize is really need?what is initialize exactly , I can probably look into source code but I would like obtain some feedback from Open-E:confused:
08-16-2007, 07:45 PM
vkeven

OK now I think that initialize fonction is not working properly or at least its causing some slownness issue , I retry my test above with internal dell 280gb raid 5 ( 5gb initialize and 5 gb not initialize ) same result initialize = slow and sluggish . Oh good , my customer badly need his new storage server and think that I would need to go with SAN/IQ from lefthand:mad:
08-16-2007, 07:54 PM
To-M

And the Price will go up $$$$$ big time (DSS $1250.00 USA and SAN/IQ 20x that amount). I have some info for you on your tests and some of your questions but you are in 10GB speed mode :).

Can you send me logs from the DSS and are you using the Demo-CD version 1.23?
If so email me so I can give you the FTP access for a different version.

If you are using the USB then email me so I can give a special version that should change your stats :)

Email: todd.maxwell@open-e.com
08-16-2007, 08:02 PM
vkeven

Quote:

Originally Posted by To-M

And the Price will go up $$$$$ big time (DSS $1250.00 USA and SAN/IQ 20x that amount). I have some info for you on your tests and some of your questions but you are in 10GB speed mode :).

Can you send me logs from the DSS and are you using the Demo-CD version 1.23?
If so email me so I can give you the FTP access for a different version.

If you are using the USB then email me so I can give a special version that should change your stats :)

Email: todd.maxwell@open-e.com

Money is not the point here ( I just sent you a PM about that ) , customer that play with 16TB of data in general have enough money to buy from lefthand or isilon or any other big player but I prefer to put my money into and more human company.

I used 1.23 Demo and I'm already having FTP access so just point me into right folder:p
08-16-2007, 08:05 PM
netsyphon

hmm.. ill do similar tests with the init and non-init and see what i get.

also, maybe you could use iozone for benchmark data?

my command is as follows:
iozone -Razb c:\pro1000mt_test_9650se-wbcache-ncq-softinitiator.xls -f G:\tesfile.lo1

maybe open-e engineers can give us a baseline of what we should expect with performance? personally i want to be able to saturate my ethernet before i saturate my disks but it seems i cant saturate either!
08-16-2007, 08:35 PM
vkeven

Quote:

Originally Posted by netsyphon

hmm.. ill do similar tests with the init and non-init and see what i get.

also, maybe you could use iozone for benchmark data?

my command is as follows:
iozone -Razb c:\pro1000mt_test_9650se-wbcache-ncq-softinitiator.xls -f G:\tesfile.lo1

maybe open-e engineers can give us a baseline of what we should expect with performance? personally i want to be able to saturate my ethernet before i saturate my disks but it seems i cant saturate either!

The best speed I can reach with one gigabyte link was 65MB/s writing and 60MB/s reading but apple xraid could nod permit more than that with only one fiber connection so I'm happy with that , the thing that I miss in Open-E is a built-in benchmark for connection between drive and Open-E server , with this thing we could be able to know if the problem come from network or physical connection and also evaluate which speed we can expect from our system before doing ISCSI and/or FC Target , less problem and more happy camper. Open-E?
08-16-2007, 08:36 PM
To-M

Then by all means SPEND away :) - love to see cost at the end of the day beyond that 16TB. LeftHand is a great product and how do we know this :), we just have good friends their :).

Anyway go to the /DEMO-CD/DSS_ver_1.23/InEngineeringPhase and test.

But I would like for you both to wait until next week for 1.30 of the Demo-CD, then run your tests. 1.30 finial release for USB will be tomorrow or Monday.

Any way you can send log file??
08-16-2007, 08:49 PM
vkeven

Quote:

Originally Posted by To-M

And the Price will go up $$$$$ big time (DSS $1250.00 USA and SAN/IQ 20x that amount). I have some info for you on your tests and some of your questions but you are in 10GB speed mode :).

Can you send me logs from the DSS and are you using the Demo-CD version 1.23?
If so email me so I can give you the FTP access for a different version.

If you are using the USB then email me so I can give a special version that should change your stats :)

Email: todd.maxwell@open-e.com

I can't find where to view that log:confused:
08-16-2007, 08:54 PM
vkeven

Oh my sorry stupid question , just sent
08-16-2007, 08:59 PM
To-M

Nope - try again - I do not see any attachments.
08-16-2007, 09:04 PM
netsyphon

remember im just trolling the DSS forum, i have iscsi-r3 ent.. is that going to be updated as well?

maybe not using ext3 and using XFS?
08-16-2007, 09:05 PM
To-M

Not a problem - but we would like to wait with newer release of 1.30.

Also iSCSI-R3 will have update as well, next following week.
08-16-2007, 09:13 PM
To-M

I will ask our engineers to see if we can have this built-in benchmark for connection between drive and DSS server. Packets and NICs can be viewed from the Status > Hardware section on the bottom.