When will software RAID hotswap work?

So, when will software RAID hotswap work? Or, is it just one of those things that is just infeasible?

We'd like to use this for our lower-end boxes (1U, 4 hard drive), so we don't have to waste money or slots with a raid controller. At this low end, software raid and hardware raid are pretty close in performance (not counting controller cache), and one of the boxes we use has a special connector that hooks up from the hot-swap back-plane to the motherboard, so it'd be kind of a pain to have to get another kind of cable to hook up to a hardware raid controller.

We dont have any plans for this as I believe it is hardware related.

In linux, it is possible to do a software raid hot-swap without rebooting or making the array unavailable, but you have to do it manually (i.e. log in as an administrator and run some commands). This is fine and preferable if you only have 4 drives (with no hot spare), where you're going to manually add a drive anyways. Perhaps this can be added to the mile-long list of feature requests?

I believe there's a kernel patch that supports hot-plugging and "warm-plugging" (whatever the heck that means). It's called libata. I guess I'll just have to wait until it's included in the mainstream kernel.

http://ubuntuforums.org/archive/index.php/t-239597.html

Hmmm... Maybe hot-plugging/swapping IS supported under the mainline kernel? I haven't tried yet with the new version of Open-E. I'll try it out and tell you if it works. Who knows? Perhaps the new kernel version will allow it to "Just Work (R)."

Here's a link to someone who seems to be entirely satisfied with hot-swapping and software RAID under linux:
http://www.earth.li/~noodles/blog/20...-on-linux.html

Thanks for the research - I will forward to our engineers. Hopefully we can do this. Please give us time as we are booked with request.

You're welcome, Todd.

Heh, no dice. I have 3 drives plugged in, which DSS sees. If I unplug one of them and do a system rescan, for some reason it only sees ONE drive (the middle one, if you're curious). If I reboot so that I only have two drives and then plug the third one in and do a rescan, for some reason it still only sees ONE drive (oddly). So, I guess I would highly recommend only plugging in and unplugging drives once the system is OFF if you're using the software RAID.

Someday, though...

(Robust software raid support would be great so that someday I can combine lots of iscsi volumes into one big volume using the iscsi initiator built-in to DSS. There are probably better ways to do this, though.)

I meant "unit rescan," not "system rescan."

After looking at this again, I realized that I hadn't set the BIOS to set up the sata drives as AHCI. I think they were set up to look like IDE drives or something like that. Anyways, AHCI supports hotswap, whereas regular IDE drives do not, so there's still a good chance that hotplugging could work once I set the BIOS properly on Monday.

Alright! Well, after switching to AHCI mode in the BIOS settings, hotswap seems to work. When I have 3 drives, pull one out and do a unit rescan, only two drives show up. When I put it back in and do a unit rescan, three show up! Now, I haven't tried this with a software raid, but it seems this should work, now!

This will be great if it works, since we won't have to spend money for a hardware RAID controller on the smaller boxes.

Thanks for the update on this - keep in mind that we don't have the expansion capabilities like the HW RAID does. Though we are working to have this but I have no ETA as to when.

Let me know this works with the RAID setup, I would like to pass this around to others.

I have made a RAID 5 array with 3 drives (2+1).

If I click Setup->S/W RAID->MD0, I'm given options to select an array member as faulty. There's also a place to mark a unit for removal, but it is greyed out. Perhaps I need to mark it as faulty first? This is what the help UI says:

Quote:

---

Function: Manager

Using this function you can manage the RAID array.

Available operations:
RAID 0:

The design of this RAID does not permit management of it. No units can have failed status. If any of them fail, the whole array will be disabled.

RAID 1:

* To mark a unit as Faulty, check the appropriate option (in the F column) and click Apply.
* To delete any unit from an array, check the appropriate option (in the R column) and click Remove.

RAID 5:

* To mark a unit as Faulty, check the appropriate option (in the F column) and click Apply.
* To delete any unit from an array, check the appropriate option (in the R column) and click Remove.

RAID 6:

* To mark a unit as Faulty, check the appropriate option (in the F column) and click Apply.
* To delete any unit from an array, check the appropriate option (in the R column) and click Remove.

RAID notation:

* PR - priority in array This represents the priority of a spare unit which will be added to the array if another unit is marked as Faulty. The higher the priority, the sooner this unit will be used.
* F - faulty The unit can be removed from the array.
* R - hot remove This unit can be removed from the array without shutting down the system.
* ST - characterizes the state of a unit in the array, which can be:

* A - active
* * - unit number within the array
* S - spare or spare rebuilding This means that the unit is free and can be added to an array, or is free and is currently rebuilding.

Limitations:

* It is not possible to set any unit as Faulty, if the matrix is degraded or during resync/rebuild.
* When using RAID 1 and RAID 5, it is possible to mark only one disk from among the active disks as faulty. This regulation is not valid for spare units in an array.

Note
Only one disk from within the active group in an array can be marked as Faulty or Removed.

Location
SETUP -> S/W RAID -> UnitMD[#] -> Function: Manager.

---

I am going to try to do this manual remove method first, while accessing files stored on an iscsi volume on this array. I'll let you know what happens and if I can rebuild it without rebooting!

It works!

Before just tearing the drive out of there, I manually selected the unit as "F--faulty" in the s/w raid manager. This caused the raid to become degraded. Then, I selected the unit as "R--hot remove". After this, I physically removed it. Nothing bad happened! Throughout this whole process, I had been playing a long audio file stored on an iscsi volume on this raid array. I did a unit rescan and found that, indeed, the drive was no longer there, but the other two were still there, and the array was fine. It was just degraded but still clean.

Then, I put the drive back in the slot and did a unit rescan. It showed up as "available"! So, I added it as a spare for MD0, which was in a degraded state. It immediately started rebuilding. Rebuilding is faster than initially building, for some reason (50MB/s vs. 20-40MB/s).

So, I'm still listening to the same audio file that I was at the beginning at this process without any hiccups or errors.

Tomorrow, I'm going to just remove the drive immediately without warning the software. We'll see how that goes!!!

Can you tell me what the hardware specs are?

Motherboard
Memory
CPU

Thanks for doing this, I think we will post on the Knowledge Base.

I think the most important thing to do is to make sure AHCI is enabled. I'm using a 1U Supermicro SuperServer that has a 4-slot hot swap backplane and built-in SATA controller on the motherboard ("pretend" hardware RAID, aka BIOS-aware software RAID, but doesn't do much good for linux, since for linux it's both unnecessary and probably slower than linux's sw raid). I'll give you more precise information tomorrow.

BTW, it should be possible to do software RAID expansion under linux. Apparently, it's a new feature under the 2.6.17 kernel:
http://scotgate.org/2006/07/03/growi...5-array-mdadm/

Here's a wiki:
http://linux-raid.osdl.org/index.php/Growing

Here's a howto guide for MythTV:
http://www.mythtv.org/wiki/RAID#Soft..._5_with_XFS.29

Oh, yeah, I used DSS build 3278, which is the one right before Atlanta.

We are working on the software RAID expansion this year. You can create another SW RAID and then add this to the existing Volume groups - poor man's expansion :)

RAID expansion would be nice, but we don't really require this in a software RAID.

Anyway, I just tried pulling a drive without warning the software ahead of time. It works great, with no interruptions (I was again streaming a media file while I pulled the drive). Right now, it is rebuilding. So, it seems like this software raid hot-swap works just as well as hardware raid.

My system is a Supermicro 1U 5015P superserver:
http://www.supermicro.com/products/s...S-5015P-TR.cfm

1GB of RAM, 3 GHz pentium 4 (dual-core), with 3 500 GB SATA Seagate drives.

This motherboard uses the Intel® ICH7R SATA chipset. You must disable the motherboard's SATA RAID and enable AHCI mode.

Also, in order for the open-e system to see a change in the number of disks, you have to do a unit rescan. After removing a disk and inserting it back again, it gets a new "Unit" name. For instance, if it originally was "Unit S002," it may become "Unit S003" when put back in. Then, after doing that unit rescan, you just add it as a hot spare for the appropriate (degraded) RAID array, then it will immediately start rebuilding. For me, rebuilding starts out at over 50MB/s at first and eventually at the end is closer to 30MB/s.

Robotbeat, Thanks for the help!

We appreciate all your great work!