Printable View
OPEN-E:
Intel SR2550, Xeon 5320, 6GB RAM, LSI Raid controller, 512MB w/BBWC 8x72GB SAS 10k drives, RAID 5, Intel 10GBE PCIE Ethernet
SWITCH:
DLink DXS-3220
VMWARE:
Intel ST1550, ESX4.0, Dual XEON 5320, 24GB RAM, NetXen 10GBE Ethernet
I'm only getting 15meg/sec from a VM across to DSS.
Anyone have a suggestion on where to start? I submitted a ticket a few days ago, and haven't heard anything.
I would probably stick a single port gigabit card in the server and DSS using point to point cable (no switch) and test that. This way you can split the problem in half by eliminating the 10g cards and switch. If your perfomance increases your can assume its not the DSS box system itself and its down to 10g drivers and the switch.
You say 15 meg/sec. How are you testing this? What are you using and what methods? Any Load during your tests? Some more details will allow some of us to better advise you what to look for.
I have a very similar setup in terms of base technology and I have been through the highs and lows of 10GbE technology with VMWare and DSS.
Testing it with Crystal Disk Mark. And right now just trying a sequential read. This is a VERY long journey starting with OF, and not getting 10GBE to run properly, now moving to a supported platform with even worse performance.Quote:
Originally Posted by 1parkplace
I've now eliminated the switch, and I'm running direct card to card, with the same performance.
I've switched out the NetXen on the ESX side: I'm now running a Chelsio card. Same exact problem.
No load whatsoever. I only have 1 Server 2003 VM, 1 vmdk. This is just to get baseline.
hi,
do you have already done this settings?
http://forum.open-e.com/faq.php?faq=...please_give_me
do you activate direct cache access in bios on your mainboard for dss and vmware?
(in some mainboards you find crystal beach to set up direct cache access)
greetings
rogerk
yes I did.Quote:
Originally Posted by rogerk
can you use insteed of VMWare an Windows 2003 or 2008 installtion to access DSS?
We use Citrix XEN Server and DDS6 with 10GB from Intel with good performance.
have tested with sqlio and these parameters?
sqlio.exe -kW -s20 -frandom -o64 -b4 -BH -LS -Fparam.txt
size of testfile 10MB
you should get roundabout : 50k IO/s and 200MB/s
if not i think there is an Hardware Error or some misconfig on VMWare site.
greetings
rogerk
If you read the release notes, Chelsio and Netxen are iffy choices for 10gbe NICs. I'm using intel based 2-port 10gig cards and they work great. I had major problems with NetXen cards which I was using before these.
Are you using Block I/O or File I/O for your Volumes? Either way, check the following and retest:
From the Console: Ctrl + Alt + W -> Enter Admin Password -> Tuning Options -> iSCSI Daemon Options -> Target Options -> Pick your target
MaxRecvDataSegmentLength: 8192
MaxBurstLength 262144
MaxXmitDataSegmentLength: 8192
FirstBurstLength: 65536
DataDigest: None
MaxOutstandingR2T: 1
InitialR2T: Yes
ImmediateData: No
HeaderDigest: None
Wthreads: 8
Have you tried to enable jumbo frames on the switch / DSS / ESX? I found this article relating to this : http://www.interworks.com/blogs/kcul...s-vmware-esx-4
I am not using Jumbo Frames, however you can try it...
No Jumbo frames....I should still be able to push 300meg /sec without.Quote:
Originally Posted by 1parkplace
My DSS box will be here in a few days (finally!). I am going the 10 gb route myself on my dss box and found this thread quite interesting. What I don't understand is why would you NOT enable jumbo frames? The investement to 10 gbe stuff is still enough over 1 gbe, why would you not want to eek out every ounce of performance that 10 gbe has to offer, especially when all it takes is a few keystrokes and mouse clicks?
Block I/O or File I/O?
Did you apply those tuning options and retest?
Aha well, my DSS can't pass enough data to saturate the link, even with non-jumbo frames.Quote:
Originally Posted by dweberwr
The other issue is that I have a tap into the iSCSI vlan, and I'd rather not have to set jumbo frames if I don;t have to on any management box that may need to be on there.
There's also the ease of migration....with little downtime.
-D
file i/o of course.Quote:
Originally Posted by 1parkplace
Yup those tuning options killed the iSCSI connection. set them all back and I was able to reconnect.
-D
interesting... Those are the settings I use and all works for me. not sure why they wouldnt work for you. I was originally given those settings by an Open-E tech.
I switched to Block I/O recently, fixed a lot of problems for me. This was from advice from Open-E, they said they certified DSS with VMWare with Block I/O, not File I/O.
I deleted all my volumes and recreated as Block I/O and then VMWare saw all my LUNs and the performance was still on par.
Block I/O is better. Fewer caching layers (I think), lower latency. In some circumstances, that means cached I/O can be a little slower, but if you have a fast back-end (a decent RAID card), it works fine. In fact, in some cases, File I/O just gets in the way.Quote:
Originally Posted by 1parkplace
I agree. Believe it or not... my machines have a great hardware backend and 32GB of RAM each, but File I/O just couldn't cut it.Quote:
Originally Posted by Robotbeat
I have a simple setup with 2 ESX servers and 2 DSS servers, each ESX targets both DSS and has an even split of VMFS volumes on each DSS.
After switching to Block I/O... Storage vMotion, vMotion, etc all sped up greatly(10x faster in some cases), barely increased "High Disk Latency" counter on the ESX servers during storage moves.
Overall I am much happier using Block I/O.
Hey 1parkplace
before you changed to block I/O did you get and cmd_abort (1143) errors?
did going to Block I/O fix this?
Not 100% sure, I know I used to get those before DSS v6 and also with the NetXen 10GB cards.Quote:
Originally Posted by symm
The upgrade to v6 and the intel 10gb happened at the same time, so not sure which fixed my problems.
I would assume it would probably remedy that situation though, since it takes caching in RAM out of the equation. As long as you have the spindles to keep up with the data load and your RAID controller is solid, you shouldn't have those problems.
Quote:
Originally Posted by 1parkplace
Thanks for answer!