Neat comparison of the different scsi target implementations

[Robotbeat]

If Open-E ever supports IPoIB Connected Mode (IPoIB-CM), then IPoIB should be pretty fast. Right now, I think they only support Datagram Mode (DM). The difference is that the max MTU for CM is about 64kB, but only 2kB for DM, so obviously there is much less overhead for Connected Mode.

We had a customer asking today if we supported 40Gbps Infiniband. We do (not tested), but since we don't have IPoIB-CM, then we are still rather slow (10GbE is faster than 40Gb IPoIB-DM).

It could also be that Open-E does support IPoIB-CM, but I haven't seen any evidence, yet.

[mscooper]

If Open-E ever supports IPoIB Connected Mode (IPoIB-CM), then IPoIB should be pretty fast. Right now, I think they only support Datagram Mode (DM). The difference is that the max MTU for CM is about 64kB, but only 2kB for DM, so obviously there is much less overhead for Connected Mode.

We had a customer asking today if we supported 40Gbps Infiniband. We do (not tested), but since we don't have IPoIB-CM, then we are still rather slow (10GbE is faster than 40Gb IPoIB-DM).

It could also be that Open-E does support IPoIB-CM, but I haven't seen any evidence, yet.

Does that mean at the moment if you want more than 100mb/s using open-e than infiniband is not really an option at the moment? I have not bought my hardware yet so I have options but Infiniband is significantly cheaper than 10gb ethernet.

[To-M]

mscooper and Robotbeat and everyone else!

Finally we have Infiniband support for CM (Connected mode).

We would like to inform you about new "small update" available for all DSS V6. Please send in a support ticket asking for this this small update for the DSS V6 only!.

This is a new option "Connected Mode" for IPoIB. Until now we were supporting only Datagram mode.

After applying small update you may change the desired mode under hardware console tools (ALT+CTRL+W) -> Infiniband tuning.

In order to get the best performance for IPoIB please to switch to "Connected Mode" and change Jumbo Frames to 65520.

Jumbo Frames can be changed under Hardware Console Tools (ALT+VTRL+W) -> Tuning Options -> Jumbo Frames.


Datagram vs Connected modes

In datagram mode, the IB UD (Unreliable Datagram) transport is used
and so the interface MTU has is equal to the IB L2 MTU minus the
IPoIB encapsulation header (4 bytes). For example, in a typical IB
fabric with a 2K MTU, the IPoIB MTU will be 2048 - 4 = 2044 bytes.

In connected mode, the IB RC (Reliable Connected) transport is used.
Connected mode takes advantage of the connected nature of the IB
transport and allows an MTU up to the maximal IP packet size of 64K,
which reduces the number of IP packets needed for handling large UDP
datagrams, TCP segments, etc and increases the performance for large
messages.

In connected mode, the interface's UD QP is still used for multicast
and communication with peers that don't support connected mode. In
this case, RX emulation of ICMP PMTU packets is used to cause the
networking stack to use the smaller UD MTU for these neighbors

[xkeithr]

IPoIB-CM!! That's fantastic news. I've just submitted a ticket and cannot wait to test it out. My Windows servers thank you.

Any chance of SRP support in the future (to speed up the Vmware side of things)?

Thanks again.

[To-M]

Just hit you with the update - check your email - If we see allot of good out come from this then we will look into and start on that, give us some time and lets see how the results are - so far we had good news on it.

[Robotbeat]

This is really good to hear, although it almost comes too late!

[biohazard]

Please integrate SRP Support in DDS V6. We looking forward to design our new Storage for our ESX Server with Open-E but without SRP-Support there is no chance!

I have made several tests with IPoIB and ISCSI and i get only 300MB/s over SDR Infiband. Thats to less! Our Storageservers are capable to deliver more than 2GB/s.

Thanks!