Question about bonding

I am hoping somebody can clear up my confusion. I am new to iscsi and understaning the whole deal about bonding nic's, so bear with me in my explanatio. I am looking at building a iscsi san box with 8 gbe ports on it. I will have at least 4 servers connecting to this san, maybe 2 more. My thoughts were to put a quad port card in all of the servers, connected to a dedicated gigabit switch for the san. What I was wanting to do was bond all 8 ports on the san and have each servers quad card bonded as well. At this time, I am not doing high availability or clustering. Eachserver will be connecting to its own target. Will this work to get the maximum performance out of the san and servers?

You have some flexibility here.
A couple of things you should know. Bond's provide great fault tolerance but suck at providing additional performance. I would suggest you look into multipathing your targets across two or more interfaces.
IMO you need two switches and you don't really need more than two interfaces per initiator.

The hardest thing to wrap your head around bonding is that is doesn't multiply your bandwidth. If you have 4 gigabit cards bonded together you will NOT get 4 gigabit of bandwidth. You will get load balancing across all 4 cards, but your performance will never be more then 1 gigabit, but you will have simultaneous traffic across all 4 channels.

Multipathing allows simultaneous access across 2 channels (or more) if you use Least Queue method of MPIO so it appears your getting more bandwidth but your really splitting the load

We have dual 10g cards in our 2 DSS's and just a single dual ethernet card dedicated for ISCSI in each server doing MPIO. We have 6 servers right now and never really strain anything and get good performance under Hyper-V

Hey webguyz - are you running File or Block I/O under Hyper-V? I am running Block but I was curious what you were doing.

Ok, perhaps bonding wasn't the correct word. Seems like every manufacturer has a different definition bonding... So what I really am looking for is link aggregation. That would be 802.3ad?

Quote:

---

Originally Posted by cphastings

Hey webguyz - are you running File or Block I/O under Hyper-V? I am running Block but I was curious what you were doing.

---

I am curious about this as well as I am running hyper-v.

Doing File I/O but without write caching because of replication for autofailover. Took a perfomance hit when I had to turn it off. I tried it with Block I/O and couldn't see that much difference in performance.. Nothing scientific, just using HDtune Pro and the results were close in File I/O and Block I/O. I had write cache on the volumes at the time I was testing.

Hey dweberwr

Link aggregation is still bonding, I have also heard it called LUG.
so depending on who you are talking to, when you add 2 or more nic ports together it's still bonding.

802.3ad is a mode of bonding, it has been knows to provide the best performance with DSS, but all this depends on your network.

other bonding modes are:
active-backup
broadcast
balance-xor
balance-tlb
balance-rr

I agree with webguyz about bonding, It does not make the nics faster it increases the throughput.

So in my scenario then, where I want to have all 8 gig ports "bonded" using 802.3ad,lacp, will that create a single logical interface to give me the most throughput.?

lets try explaining it another way.

Lets say one of your servers is talking to the DSS and no others. The very most throughput you will get is 1 gigabit. Lets say a second server starts talking as well, now you have 2 nics talking, but each only gives you a max of 1 gigabit. A 3rd server starts talking and now you have 3 gig of data flowing, but the most anyone of them will give you is 1 gigabit. Think of a 2 lane highway as opposed to a 6 lane highway. All the cars go the same speed (55mp or 1 gigabit), but more of them can get from point A to B simultaneously as opposed to the 2 lane highway (where cars still only go 55mph or 1 gig) but less of them can get from point A to B at the same time.

Quote:

---

Originally Posted by dweberwr

So in my scenario then, where I want to have all 8 gig ports "bonded" using 802.3ad,lacp, will that create a single logical interface to give me the most throughput.?

---

any type of bonding (lacp, or 802.3 specifically) will use some type of xor/xand that only works best when you have:

lots of sources and lots of destinations
lots of sources and a single destination
lots of different types of traffic

the algorithm (depending upon the switch) uses the least significant bits of either the source/dest mac or the source/dest ip to determine which interface to use. Some can even use TCP ports.

If your goal is to increase I/Os and throughput, I'd suggest you look into MPIO. I have several VI clusters and this has worked very well for us. We see roughly a 30 - 40% improvement across two interfaces.

Quote:

---

Originally Posted by enealDC

any type of bonding (lacp, or 802.3 specifically) will use some type of xor/xand that only works best when you have:

lots of sources and lots of destinations
lots of sources and a single destination
lots of different types of traffic

the algorithm (depending upon the switch) uses the least significant bits of either the source/dest mac or the source/dest ip to determine which interface to use. Some can even use TCP ports.

If your goal is to increase I/Os and throughput, I'd suggest you look into MPIO. I have several VI clusters and this has worked very well for us. We see roughly a 30 - 40% improvement across two interfaces.

---

The main function of this san will be to host virtual machines on 3 or 4 host servers. It will also be used to host disk to disk backups and long term file storage. My primary concern is having enough performance to run the virtual machines, about 25-30 when all said and done. So to get the best performance I want to do what then?

I should clarify my last question, what specific setting in DSS do I want then, to provide the best throughput and performance? I was thinking about it last night at home and finally understood some concepts and examples that were mentioned here.

If it were me I would probably try creating 2 802.3ad bonds of 4 gigabit ports each to create 2 channels so I could use MPIO on the servers.

On the servers use MPIO. Not sure if MPIO will work with bonded NICs or not at the server. We just have dual gig cards so MPIO is simple.

Quote:

---

Originally Posted by webguyz

If it were me I would probably try creating 2 802.3ad bonds of 4 gigabit ports each to create 2 channels so I could use MPIO on the servers.

On the servers use MPIO. Not sure if MPIO will work with bonded NICs or not at the server. We just have dual gig cards so MPIO is simple.

---

So then in this example, esentially having 2 4gb ports on the dss box, am I limited in how many servers can connect back to the box? I'm guessing not since it sounds like it is just 2 separate , channels if you will.

Hey dweberwr

The best thing to do is to run some performance test, with different settings (bonds).
Depending on your network (switch, nic, routers etc) your results will vary.

I don't actually have a dss box, Going to be eventually, in the process of trying to get all my ducks in a row before it gets here.

Quote:

---

Originally Posted by dweberwr

I don't actually have a dss box, Going to be eventually, in the process of trying to get all my ducks in a row before it gets here.

---

In that case I would suggest a dual port 10G card in the DSS instead of messing around with all those bonded ports. Supermicro has a nice one for less then $500. I think the quad port gigabit Intels are aound $400 each.

Quote:

---

Originally Posted by webguyz

In that case I would suggest a dual port 10G card in the DSS instead of messing around with all those bonded ports. Supermicro has a nice one for less then $500. I think the quad port gigabit Intels are aound $400 each.

---

Do you mean 10 g on just the dss or on the servers too? I can't afford a switch with nothing but 10 g ports on it.

I would put the dual 10g card in the DSS and use MPIO with dual gigabit adapters in the servers.

A Dell 6224 with four 10G CX4 uplink ports costs $1,800.00 ($1,600.00 with just 2 CX4 ports) and you can use the 24 gigabit ports for 12 MPIO server links. I would dedicate this switch to doing ISCSI only.

Couple this with the SuperMicro AOC-STG-i2 dual 10g card (less the $500.00) and you have a cost effective ISCSI SAN infrastructure that should be able to support several virtualized servers and up to 2 DSS's if your doing failover.

SAN's are not cheap, but ultimately the way to go. :D

Quote:

---

Originally Posted by webguyz

I would put the dual 10g card in the DSS and use MPIO with dual gigabit adapters in the servers.

A Dell 6224 with four 10G CX4 uplink ports costs $1,800.00 ($1,600.00 with just 2 CX4 ports) and you can use the 24 gigabit ports for 12 MPIO server links. I would dedicate this switch to doing ISCSI only.

Couple this with the SuperMicro AOC-STG-i2 dual 10g card (less the $500.00) and you have a cost effective ISCSI SAN infrastructure that should be able to support several virtualized servers and up to 2 DSS's if your doing failover.

SAN's are not cheap, but ultimately the way to go. :D

---

We are a cisco shop here, so whatever we would get switch wise would have to be cisco. However the cisco switch I would be getting has 2 10 gb module ports on it, its a 24 port 3560-e. We have one of those already on our core and it is working well. I do believe this is the way to go.

Or after doing some looking around it looks like I can source some used/refurbishe 10 gb or even 20 gb infiniband equipment for about the same price as it will to go ethernet, would that be a beter route?