Infiniband with iSCSI

[To-M]

Qlogic – does not support using a crossover or regular cable to directly connect without a switch.
Voltaire – stated it will work but they do not support this as well. Although they will tell you what you can do to make it work.

Voltaire states you need one of the hosts systems to have the Subnet manager code enabled to do this. Also this can not be installed on our OS so this
will be out of our reach to support as well. Best to use a switch. The reason this is not supported by the manufactures is because this is still in testing and
the code is not mature yet. So it can be done just not supported from Qlogic and Voltaire. They did state that you can use a regular cable not crossover.

You can contact them on there support numbers, Voltaire was more receptive on giving more information but still did not want to support.

Information on Subnet Manager infrastructure ("OpenSM").
http://infiniband.sourceforge.net/SM/overview.htm

Information about Infiniband.
http://www.infinibandta.org/itinfo/

[Crosscreek]

Thanks. I spoke to someone at SmallTree Communications he told me the same exact thing.

To make matters worse, I'm using Macintosh clients and there is no subnet management software for the Mac. And the only Mac IB drivers are dated and unsupported.

What turned me on to the idea of IB is that the cards can be purchaced online for dirt cheap. Someone on ebay is currently selling Voltaire HCA 400ex cards for $175 each.

But if I have to buy a switch, there is no cost benefit over 10gbe.

[To-M]

Not sure about the EBay deal as I have been burned before. Thanks for the update - I wish we had more to offer with IB but yes you would be better of with a switch for your usage but not by cost. Go for the 10GB. IMHP

[Robotbeat]

I just saw this thread. Anyone have any luck getting a infiniband HCA-to-HCA connection working?

We'd do this only for testing purposes, so one end could be a redhat server or something (with the other being open-e).

[webguyz]

Anyone know of a Infiniband for Dummies book or web site? Tying to understand whats needed for a small network connecting multiple servers to an Open-e SAN using infiniband. Looking at the inifinband spec (which makes my eyes roll into the back of my head as soon as I start reading it) it appears that some kind of software has to be running on a intelligent host (server) connected to the inifinband switch.

Can anyone tell me (speaking slowly) what would be needed to connect 1 windows 2008 server to 1 Open-E DSS using an infiniband switch. Other then infiniband adapters in each box, is there some software that has to be running? Or does infiniband work like tcp/ip and there is not central host managing the show?

Thanks!!

[Robotbeat]

Well, the open-e box can only use IP over Infiniband at the moment. Since you're using IP, it should be pretty simple.

I haven't done it, but we are looking to get some infiniband hca's in our lab soon (I don't work for open-e) to test it out, both with open-e's solution (IPoIB) and also with some other solution (random linux) using a sort of RDMA protocol (iSER or SRP or NFS-RDMA something else).

Unfortunately, with IP over Infiniband, you still have to use a lot of cpu to do all the IP processing. With RDMA, you can bypass the cpu for most transactions, so you're able to get much closer to the theoretical throughput of your interface. The downside is that you need to run iSER or something like that, instead of something that all computers basically support now (like NFS or SMB or iSCSI).

Another upside is that you can get a SDR (10Gbps) or DDR (20Gbps) infiniband card for less than 10Gb ethernet or 4Gb fibrechannel. You also can find QDR (40Gbps) cards out now that only are $1000 (and that's for a dual-port card, too). The Infiniband switches aren't any more than 10GbE per port.

[webguyz]

Well, the open-e box can only use IP over Infiniband at the moment. Since you're using IP, it should be pretty simple.

I checked Mellanox site and they have Windows drivers IPoIB for their cards so If I were to go with their hardware I would need:

Mellanox card with IPoIB drivers for SAN clients
a Infinband Switch
Mellanox card in Open-E Server (are IP over IB drivers in the latest firmware?)

Would that be enough to allow clients over ISCSI to access my Open-E SAN using Infinband or did I miss something.

Thanks!
Andy

[Robotbeat]

Here's a tutorial:
http://www.ibswitches.com/Docs/Infin...20Tutorial.pdf