Switchless 10GbE setup with 2 DSS and 3 ESX Hosts

Hey guys,

my question is based on this blog entry:

http://blog.open-e.com/is-it-possibl...lable-cluster/

The example uses 2 DSS and 2 ESX Hosts - is it also possible to have a switchless config with one more ESX Host? So 2 DSS (for High Availability) and 3 ESX Hosts. I've attached s simple cabling sketch with just the storage net (without management network or pingnode...) and the necessary 10GbE NICs.

http://img706.imageshack.us/img706/3...switchless.jpg

We have a config with 1 Open-E DSS and 3 ESX Hosts, connected with a 1GB Ethernet (including Multipathing 4x 1GB, which creates a big cabeling overhead) and want to upgrade to a second DSS with the 10GbE config - but before we can do so, I want to make sure that this configuration works.

Best,
Norman

We had a customer have a similar setup with only 2 ESX systems (when the AUX and VIP was not able to see each other then issues happened when they upgraded to the latest build 5626), we did not certify it for your setup. But you will need to make sure that the AUX and Ping nodes will be able to communicate or it wont work. If you are going to use this type of setup please wait until the next release as the Aux ports assigned with the VIP currently with the 5626 version will not work. Next release is ETA near the end of the month or sooner. But even so I would strongly test before you implement in your production and test with pulling a NIC and using the Failover.

Hey Todd,

thanks for the quick reply.
I think I understand what tell me in general (yes it works/should work, no we do not recommend it/tested it) - thats fine.
But could you please clarify the details a bit? What do you mean with AUX and VIP? Regarding this picture here http://blog.open-e.com/wp-content/up...Mware-ESX4.png AUX means the 'Auxiliary connections', right? But VIP?

Thanks in advance,
Norman

VIP=Virtual IP

From the WebGUI, go to Setup>network>Failover you can find Virtual IP there.

Virtual IP Settings
With this function you can view the MAC address and assign virtual IP settings for your chosen interface. The virtual IP is shared between the failover nodes. For example, primary node has the IP address 192.168.1.1, while secondary node's address is 192.168.1.2. The virtual IP address is 192.168.1.3. In this situation, the primary node (the active node) will be available under 192.168.1.3. In case of primary node failure, the secondary node will take over the 192.168.1.3 address to ensure all connections can continue to the same address.

The Virtual IP needs to be unique within its network environment and the same on both nodes (Primary and Secondary). Changes to virtual IP settings can be done only when failover configuration is not active.

Also Virtual IP can be in a different subnet work than the node IPs, e.g. 192.168.2.1 as opposed to 192.168.1.1 and 192.168.1.2.

AUX= Auxiliary

Auxiliary connection
This option is used to configure the interfaces on which the Failover sends UDP broadcast/unicast traffic. At least two interfaces needs to be specified. If you have chosen to use unicast when configuring failover, you need to inidicate the IP address for the network card on the other node which should respond to the packets sent from the interface you are currently configuring.

_______________
Alaa Souqi

Switchless 10GbE setup.., good question, thanks for the great answers, very helpful

Quote:

---

Originally Posted by ericsocialadr

Switchless 10GbE setup.., good question, thanks for the great answers, very helpful

---

Yes, defnitly helpful! :o)

Thanks, Todd & Alaa

Hey Todd, Alaa,

a little follow-up question: we're planning with the adapter 'Intel X520-T2' (2 Port 10GBASE-T, RJ-45) and I found out the Chipset it uses (Intel 82599EB) is in your Hardware Compatibility Lists.

Are there drivers included in the Open-E DSS (latest version) for the Intel X520-T2?/Is this adapter compatible (even if it is not 100% tested, which is indicated by the 'DI'-Tag in the list, I guess...)?

Best and thanks,
Norman

Those drivers are included in the newest DSS V6 build 5626. This information can be found in the readme_5626.txt:

UPDATED:
- Updated drivers for Intel's Ethernet cards 10Gb/s (ixgbe - v3.3.9-NAPI)
- Updated drivers for Intel's Ethernet cards 1Gb/s (e1000e - v1.3.10a-NAPI, e1000 - v8.0.30-NAPI)

BR,
Tomek

Would the same setup work for NFS with the NFS Failover module?

Presumably the DSS volumes should be populated with high rpm SAS rather than SATA drives to fully utilize the 10GbE bandwidth?

Would anyone be able to share any performance data for such a setup?

Thanks,
Kok-Hong

Hey guys,

I'm currently setting up this 10GbE switchless Config in an test environment and triped over the following:

"For each interface which has enabled Virtual IP address it is required to have enabled Auxiliary connection." - this is an error message you get when you try to start Failover. The problem is: when you do not have a switch in the line, there is no physical link between the members of a Virtual IP Pool. I thought the link would be possible via the vSwitch in ESX, but it looks like that this isn't the case.

Anyway, it turned out the setup is relativly complicated to configure and to wire. Todd, you said you have a customer who is using this setup. Any chance to get a few more infos from him, e.g. are they also using VMware ESX? Do you think there's a chance to get in contact with them?

All the best and glad to hear from you guys.
Best,
Norman

Oh no, reading helps...

Sorry Todd, the correct question is: when will the next release you recommended comes out?

But that's also wrong, taking a look at the webiste helps... I'll now test build 5794 and will be in touch again if necessary.

Best,
Norman

Have a nice weekend. And sorry for the noise.

Hi Norman

I am testing NFS and iscsi failover with 10GB switchless with 3 esxi4 server and
also parallel 1GB Nics over a switch as bond (active-backup) and standby nic in esx vmkernel. The test seems to work. I changed today to new release and will test again.
In the moment I simulate everything with 1GB nics but everything worked, next week I will try to test with 10GB.

What is the status of your tests ?

If you have interest for a direct contact to share experience about this 10GB test and network diagram, you send send me a PM over this forum and I will contact you

Hi shechtl

We are very much interested in Your 10Gbps switchless configuration of iSCSI failover (2x DSS in iSCSI failover) and 3 esx4 servers.

Can You share with us Your network diagram of mentioned configuration?

you can find a diagram at this link
http://www.megaupload.com/?d=FR6FHVAQ

but its not completed and I did my test with only 2 x ESXi and only with 1GB nics
until i get my 10Gbit nics next week, but it should work also with 3 esx.

With the 10GB /1GB bond I will have some part of a nic failover. If someone will also verify, test and post his results here will be nice

Quote:

---

With the 10GB /1GB bond I will have some part of a nic failover. If someone will also verify, test and post his results here will be nice

---

It is not recommended to bond different chipsets to the same bond.

Quote:

---

Originally Posted by Gr-R

It is not recommended to bond different chipsets to the same bond.

---

ok if its not allowed, but
I will use Intel 1Gb and Intel 10GB Nic and also I will use active-backup so the 1GB nic
is normaly only standby

On the linux bonding project I could found nothing that it is not allowed to bond nic with different speed and hardware ?

different chipsets/speeds in a bond can be unpredictable, as it forces different drivers into the bond.
I'm not saying its not allowed, but since it can be unstable, we do not recommend you do this with any bond.
It is also better to bond ports that are on the same card.

Hello,

We are trying to get the switchless setup to work with Xenserver 6.1 and and Active Active open-e setup.... till now without success.

We see the virtual ip switching etc etc but the server looses connection when a swithover occurs since it does not know to use the second network port. If we pull the first ethernet cable switch over occurs, how can we do this automatically?

Anyone has a walkthrough how to get 2-3 Xenserver hosts to work over all 10Gbe ?

Thanks a lot in advance!