OK, i figured out how to get fast speeds with Open-E and VMWare VSphere ESXi 4 and thought I would share:

I'm not going to go over the basic setup stuff, but this assumes you can setup your volume/lun and connect esxi to it using 2 paths.

Step1: configure the Open-E iSCSI target TCP/IP settings
most of this i got from http://forum.open-e.com/showthread.php?t=1542
On the open-E box, hit Ctrl-Alt-W, log in to the console, go to tuning options, iSCSI daemon options, target options, <your iscsi target>.
Here are the settings I made:
MaxRecvDataSegmentLength=65536
MaxBurstLength=1047552 (the recommended 16776192 knocked out my connection)
MaxXmitDataSegmentLength=65536
FirstBurstLength=523776
MaxOutstandingR2T=8
InitialR2T=No
ImmediateData=Yes

Step 2: configure the VMWare iSCSI connector TCP/IP settings
Once you have connected vmware to the open-e lun
From the console, or via esxcli, or via ssh, edit etc/vmware/vmkiscsid/iscsid.conf using VI editor.
Here are the settings I made, some of these you can make in VIClient under properties of iSCSI storage adapter, advanced settings. I made those first, then checked this file to make sure they are all set:
discovery.sendtargets.iscsi.MaxRecvDataSegmentLeng ht = 65536
node.session.iscsi.FirstBurstLength = 523776
node.session.iscsi.MaxBurstLength = 1047552
node.conn[0].iscsi.MaxRecvDataSegmentLength = 65536
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmedateData = Yes
node.conn[0].tcp.window_size = 65536

Step 3: configure Multipathing to use Round Robin
http://www.kb.open-e.com/download/92/

Step 4: fix IOPS
At this point I was getting pretty good speeds, about 135 MB/s, but I was supposed to have 2XGB ethernet speed, with 2 paths configured in VMWare and two NICs bonded in Open-E... Then I found this thread: http://communities.vmware.com/thread...5&amp;tstart=0
And in there someone links to this article:
http://virtualgeek.typepad.com/virtu...e-vsphere.html
Where the key part of the article (for me anyways) was doing this:
You can reduce the number of commands issued down a particular path before moving on to the next path all the way to 1, thus ensuring that each subsequent command is sent down a different path. In a Dell/EqualLogic configuration, Eric has recommended a value of 3.

You can make this change by using this command:

esxcli --server <servername> nmp roundrobin setconfig --device <lun ID> --iops <IOOperationLimit_value> --type iops
so I ran that (the lun ID was the thing that starts with eui.0000blah blah blah), setting my IOPS to 1 (from the default 1000) and got up to 225MB/s speed reading and writing sequentially. Random writes are still a bit slow, 58MB/s, but maybe this is RAID5 related.

Now when I turn replication on, both sequential and random writes go down to 10MB/s, so I need to sort that out still, but hopefully this is helpful for anyone having a hard time setting up ESXi with Open-E.

Cheers,

Hugh