Advice on Hardware Configuration and when to go to multiple controllers
I have been working on configuring a box (will need 4 total of varying size) for some upgraded storage capacity for a VMWare VSphere Essentials 4.0 box running a few virtual boxes with applications, one of which is SQL based, but, none of which are too intensive. The box needs a lot of storage (up to 48TB). Some of the storage will be for adding space for active applications, but, most of it will be for backup storage from other boxes. In looking at the hardware, I was trying to stay Intel as much as possible (for warranty considerations). I was also trying to spread across as many drives as possible as seems to always be recommended. I find when you get to the chassis with 24+ drive bays, you also move into the need to either employ either a SAS/SATA Expander or another controller to get enough outputs to feed the backplanes/drives. Most of the chassis I have been looking at have a backplane using a mini-SAS (x4) to feed a row of 4 drives. So, on the cheap, it would appear to be easy to buy a controller and add expanders to feed up to the theoritcal maximum of the controller. However, it would seem logical that spreading that demand across multiple controllers would give you better performance (not to mention some potential hardware failure redundancy). So, I guess I am looking for some advice on controller configuration for Open-E when you get beyond 24 drives. In the same regard, I am looking for advice on the configuration as a whole. The big concern at this point is that the Intel RS2WG160 controller (based on the LSI 2108) is not yet in the Open-E supported hardware list (unless I am overlooking it). Is there another Intel SAS/SATA RAID controller card that anyone is using successfully? In the supported hardware list, there is no SAS and the SATA RAID cards are just generic. Intel has a recipe on their ESAA site for an Open-E box using a their SR2612UR server and an AXXROMBSASMR controller. So, I know that some of their controllers must work. Any insight is greatly appreciated.
Box 1 (smaller; This box can't get to 48TB internally without going to 24x 2TB SATA drives and where there is some per)
1x Chenbro RM51924 (24-Bay 3.5)
1x Intel S5520HC Server Motherboard
1x Intel Xeon E5620 2.40 GHz Processor Quad-core
3x KVR1333D3D4R9S/4GI (12GB triple-channel mode)
1x Intel RS2WG160 SAS RAID Controller w/Intel AXXRSBBU7 RAID Controller Battery
21x Western Digital WD2003FYYS 2 TB RE4 SATA/300 (inexpensive storage)
3x 15K.7 ST3600057SS 600 GB (more performance for the application allocated space)
1x Open-E DSS V6 16TB + Storage Extension +32TB + Annual Basic Support
(NOTE: Related to my opening question, the RS2WG160 gives me 4x Mini-SAS (x4) outputs and this Chenbro backplane needs 6x Mini-SAS inputs, so, I need to either add an Intel RES2SV240 SAS Expander, or another RS2WG160 controller to fill all the slots).
Box 2 (Bigger)
1x Chenbro RM91250 (50-Bay 3.5)
1x Chenbro RM51924 (24-Bay 3.5)
1x Intel S5520HC Server Motherboard
1x Intel Xeon E5620 2.40 GHz Processor Quad-core
3x KVR1333D3D4R9S/4GI (12GB triple-channel mode)
1x Intel RS2WG160 SAS RAID Controller w/Intel AXXRSBBU7 RAID Controller Battery
21x Western Digital WD2003FYYS 2 TB RE4 SATA/300 (inexpensive storage)
10x 15K.7 ST3600057SS 600 GB (more performance for the application allocated space)
1x Open-E DSS V6 32TB + Storage Extension +16TB + Annual Basic Support
(NOTE: Related to my opening question, the RS2WG160 gives me 4x Mini-SAS (x4) outputs and this Chenbro backplane needs 12x Mini-SAS inputs, so, I need to either add at least two Intel RES2SV240 SAS Expanders, or possibly up to three RS2WG160 controllers to fill 48 slots).
Having a mini-SAS connection for 4 HDDs is complete overkill.
No hard drive exists which can saturate a single 6 Gpbs channel, this is something which only a handfull of SSDs can do -- and only under very selective conditions (sequential reads). The highest transfer rate that I have seen is less than 200MBs (1.8 Gpbs), so 4 HDDs could be supported (almost) by a one of the 4 channels in the mini-SAS connection.
Further, remember that SAS expanders work on the basis on taking a 4-channel connection from the main RAID adapter and allowing multiple devices to to share that connection, so your are ultimately going to run into a limit of the speed of 4 channels (4 x 6Gpbs = 24 Gbps = 3 MBs). So, using expanders will not explicitly increase your overall performance.
If you want to still pursue a 4-channel connection to 4 HDDs approach, you should think in terms of increasing the number of expanders (or using multiple controllers -- although that has it's own issue).
Personally, for a near fully populated chassis (48 HDDs) I perfer to see 4 expanders -- each connected to 3 backplane channels.
Sean
P.S. You might want to have a look at SuperMicro they have a couple of some 4U chassis (vs. 9U in the Chenbro chassis) which support 36 or 45 3.5" HDDs! So, you could have a lot of rack space.
Appreciate the quick replies. Glad to know that controller chip is supported. SeanLeyne, I understand what you are saying about the data transfer rates of current hard drives. Thanks for the detail. As opposed to thinking that I was gaining performance with the expanders, I was worried about losing it by trying to carry too much traffic down a single 4-lane. I'm having a bit of a tough time visualizing exactly what you are saying about connecting the "4 expanders -- each connected to 3 backplane channels." To take a stab at it, for a 48-drive solution, you would generally recommend one big controller with say 4 mini-SAS 4-lane outputs. Each of the mini-SAS connections would go to a 4-port expander (1-in 3-out) and each of the 3 expander outputs would then connect to 3 backplane channels (each channel commonly correlates to a row of four 3.5" drives). That means that at it's heaviest load, the traffic from 12 drives are getting hauled between the expander and the controller (the smallest pipe) in one 4-lane at up to (4x6Gbps) 24Gbps. Since any one drive isn't likely to exceed 1.8Gbps, the total of their traffic wouldn't exceed (12x1.8Gbps) 21.6Gbps. Am I understanding that correctly? May I ask you what controllers/expanders/chassis you are using? I am pretty set on Intel, but, it would sure be nice to look at what else it out there. I did see SuperMicro had some pretty slick chassis designs. I will definately take a closer look. Thanks again!
Your understanding of the expander usage, as I tried to convey, is correct.
Regarding the use of a single controller:
My own experience is only using a single controller (largest HDD array is 12 HDDs), and I would be concerned about the ability to manage all of the HDDs effectively (allowing me to create RAID groups using the optimal number of HDDs) without having to predetermine which HDDs would be connect to which controller (in a multi-controller scenario).
By the same token, there are valid reasons to justify multiple controllers (depending on number of HDDs);
- some controllers have a limit to the number of RAID sets/groups which can be created (I know that Adaptec controllers do).
- you may want a larger amount of cache memory than available from a given controller
- you may not need a cache controller for some HDDs
- you may want a fancy controller which does hybrid SSD caching, but only for some HDDs
- you may have exceptional throughput requirements which are more than the controller can handle (is only possible in large sequential transfers, or if you are building an array of SSDs)
Regarding Intel:
- Over the years Intel has created RAID controllers using their own chips or have simply rebranded/reused controller chips. Just because it says "Intel" doesn't mean that it is Intel"
- The most of the current Intel controllers (and the expander which you pointed out) are in fact based on LSI chips, so any controller based the same chip would perform the same and be as reliable
- There are other good/excellent RAID controllers available, all off which have good/excellent support. There are plenty of features which need to be considered beyond just the brand name of the controller (hybrid SSD RAID support is an important one -- could allow for lower SATA speed HDDs to be used instead of expensive high speed SAS drives -- for high read/low write applications)
- If you are concerned about warranty support than I would suggest that you simply buy a backup controller. If you run into a real problem, controller failure, the only way your are going to get up-and-running quickly is by simply swapping out the card, and worrying about the warranty return/swap on the card separately. That is what we have done for our 2 SAN/storage systems, waiting even 24 hours for a replacement is much too long for an outage.
Advice on Hardware Configuration and when to go to multiple controllers
Reply With Quote
