Infiniband...

[chris74]

Hi, I've been tasked with setting up a storage network based on infiniband. Its a challenge for me as I'm new to this technology but I've been doing my research and in the process of educating myself I've stumbled upon open-e.

My project requires the creation of storage servers to use with CentOS/Xen hypervisors and as such I'm looking at using infiniband cards with the Mellanox ConnectX-2 chipset (MT25408) which is a QDR card, along with a QDR switch. I'm also considering the LSI MegaRAID SAS 9260-16i controller.

I note that open-e doesnt support SRP which is a shame and I can see that open-e appears lacking somewhat in its infiniband support. I'd like to hear from anyone who has had success with open-e and IPoIB and would appreciate any helpful advice or comments. I'm also interested in hearing about how replication works with open-e as I plan to pair up each storage server for resilience. Can it replicate over IB? Do I need a second interface?

Thanks for your time.

[enealDC]

Yeah, I'd like to see SRP support as well, but until it gets into SCST, we won't see it in Open-E.
OpenE supports IPoIB so yes, you can push replication traffic over it. It *should* be treated just like another eth[*] int.
I would suggest you look into Fibre Channel if you have a budget for IB. You'll get great performance the the FC target is very stable. 4GB FB HBAs and switches are going for cheap on ebay these days.

[SeanLeyne]

10GBe is also a very good and reasonably inexpensive (compared to IB) option. An Intel dual-port CX4 adapter should cost around $700 USD. Intel also has 10GbaseT adapters (which use Cat6e cables) bt I don't know the pricing off-hand.

[enealDC]

Don't even think about 10G over Cat6...
Read http://catalog2.corning.com/CorningC...LAN-803-EN.pdf

[Gr-R]

Don't even think about 10G over Cat6...
Read http://catalog2.corning.com/CorningC...LAN-803-EN.pdf

Great find !

[SeanLeyne]

Don't even think about 10G over Cat6...
Read http://catalog2.corning.com/CorningC...LAN-803-EN.pdf

eneal,

While the technical details are factual, the tone of content of the doc shows a clear bias towards the fiber-optic which is the core of Cornings business -- they need to blast Cat6 since they need to sell fiber!

1 - the OP was referring to an install where the cable length in the 6 to 20 feet (2 to 6 metres). At those distances, none of the issues riased in the doc really have any significance.

2 - the posted topic is about IP over IB, IB connections don't support very long distances so worrying about cables of more than 50 metres is not appropriate.

3 - the need for server side connection of over 10GB/s will be several years away. 40/100GB/s connections are only being discussed for data centre backbones, where there are 100s of servers which would be connected to intermediate switches, which would need to be tied together using 40/100 GB/s connections. When you consider that a *single* HP ProCurve A12500 switch can support **512** 10GB/s connections (with intermediate 1 Gb/s switches which could connect at 10GB/s to that switch, more than 5120 1Gb/s connections could be handled that that single switch), the need for interconnects to another switch is something that very few installs would ever need to worry about.


Sean

[r3vo]

Yeah, I'd like to see SRP support as well, but until it gets into SCST, we won't see it in Open-E.
OpenE supports IPoIB so yes, you can push replication traffic over it. It *should* be treated just like another eth[*] int.

just my two cents, cause you said should be treated...
indeed, I tested 2 Dssv6 with Infiniband and the interfaces are treated like another eth[x].
I tried also to replicate a volume between the two boxes which was no problem at all.
I couldn't make any detailed performancetests due to time reasons, and I only got 4hdds (sas) in a raid5, but I know that the speed of replication was very fast.
Unfortunately I only got 2 infiniband cards (without switch), so I wasn't able to perform a failover scenario.

[chris74]

Thanks for the replies. As I suspected, there doesnt appear to be anyone running IB with Open-E - mainly because its just too slow using IPoIB and iSCSI. I'm guessing they dont support SRP because it isnt a recognised standard. Its a shame because the benefit of using RDMA over IB coupled with the low cost of the hardware makes for very exciting possibilities when building a SAN on a budget.

IB is faster and much cheaper than 10GbE. All the drivers and software required to run IB/SRP initiators and targets on Linux are fully available. Its just a shame that no-one has yet bundled it all into a software package such as Open-E. When someone finally does this, I'm sure they will make a lot of money.

We are committed to using IB in this cloud and building all the hardware ourselves - so its just a case of spending the time looking at the OFED and SCST target drivers and going through an extended period of configuration and testing to get things just right.

[enealDC]

Open-E has moved to SCST and the status of the SRP target for infiniband is not clear for that project.
If you google, "ofed SRP target" you get a link to a howto document, but I'm not sure what kind of mileage you'll get.
Someone on this thread has already commented on IPoIB for replication and says it seems fast. As I mentioned earlier in the thread, they should be treated like regular eth* interfaces.

[enealDC]

Actually, I mispoke, It's not the SRP target that's uncertain it's the iSER target. Sorry for the confusion.