two 2 weeks ago we had an totaly corrupted DSS6 Failover System.
Our Hardware:
2 identical Storage Server with Open-E DSS6
1xIntel X5550
24GB RAM
2 1GB Intel NIC on board
1 DualPort 10GBE CX4 Intel
16 1TB HDD from Western Digital RE3
1 ICP5165BR Raid Controller (Adaptec) latest bios build!
RAID Config: Raid6
We use 2 Citrix XEN Server to access DSS6 over 10GBE CX4.
On Citrix XEN are 15 Windows 2003 Server.
The DSS6 Storage Server are in Failover mode with multiple Targets.
so long...
one day, a HDD fails in our Primary DSS6.
So we changed the HDD against a new one of same type.
the change was done with hot pluging in a running system.
anything was fine. the raidcontroller start the rebuild for raid 6.
until this our data was not! corrupted.
suddenly we noticed that some VM on XEN run into Bluescreens.
at this moment the rebuild status shows ca. 30% und the adaptec shows no error!
DSS shows now error! anything ok...
and from now by every new 1%build more VMs have heavy Problems.
in this moment we realize that the rebuild destroys our data.
we run the manuell failover and start replay the backups to DSS Slave.
after replay the backup we don't resume the manuell FAILover to invetigate Primary System "Error" .
3 Days later a HDD fails in the second DSS6, the adaptec starts an automated rebuild (of course during the night...) after noticed that the rebuild status was ca. 30%.
and again many VM's crashed und our data was corrupted again.
so we make a call to adaptec, send them log files.
we make call to open-e and send them logfiles, too.
there was no error in the logfiles!
adaptec have no idea, and i think they w'ont find anything...
my opinion is that there is a bug in the latest firmware build from the ICP5165BR.
data became corrupted only a rebuild, and rebuild is only done by raidcontroller...
have someone else problems like our problem?
and for those who have the same raidcontroller be aware!
i do have the same controller icp5165br (15753)
since it is in my secondary node i cannot tell if the same issue occurs, but in last failover (quick update of primary node) had no issue with it, even after 2 harddisk failures in r10
but my overall experience with adaptec controllers is rather bad... some random kernel panics in an older controller, crapy management software and such
Ouch!
Your controller rebuild settings were probably set to High.
With RAID5/6, it is ESSENTIAL that you run scrub checks routinely. We run them twice a month.
We use Areca Controllers..
@enealDC
there is an automated check once a month.
what do you mean with rebuild settings to high?
there no setting on the adaptec that allow me to change something like this.
we changd the controller on both dss machines.
because of compatibility of raiddata we use on the first DSS another adaptec (51645).
on the second we use now areca.
but after 3 weeks we have no idea wy this happend.
no answer from open-e and no answer from adaptec.
i got an answer from open-e,
they say it is driver related. the driver for the IPC Raid Controller in the actuell dss6 release is buggy and old. the driver will be changed in the next dss6 release.
so be carefull with raid 6 rebuild and this controller!
corrupted data caused by rebuild raid6
Reply With Quote
