Exomium S200 and DSS V6 and LSI 9265i and Cachecade -> Data corruption

Hi,

habe here an SuperMicro with LSI based certified OPEN-E Exomium S200 box with a LSI 9265i (with batt. backup and Cachecade 2.0 Pro).
Cachecade SSDs are 4*Samsung 843T Pro in a RAID1.
After 14 days of usage, I can confirm that the latency improved, unfortunately in case of data consistency the opposite happened.
I had a uncorrectable disk directory structure error in a Win7 X86 (any kind of chkdsk was not able to repair it), a correctable in SLES 11 SP2 Ext3, and a uncorrectable at our main Mac OS X 10.8.4. server. After disabling to cachecade the last one disappeared (this saved a lot of time and effort) so I'am quite sure that the problem was just a not consistent Cachecade cache.

For sure I use the lastet DSS V6 and LSI firmware.

After that I opened a ticket at LSI, after a few day the suggested to test the previous firmware release if the data corruption occurres too.
I refused this, because this is our main production data box with 20TB, after that LSI closed the ticket.
Think about to run something like this and find inconsistent data after 14 days, do you have a backup strategy which can fix that?

I have currently a discussion with the hardware vendor about the product and have to make sure that LSI Cachecade is supported by OPEN-E and the driver in OPEN-E fullfill the requirements of LSI for Cachecade. OPEN-E firmware is 6.0up98.8101.7337 64bit, 9265i is 23.16.0-0018.

Can you please confirm this?

Do yo have any success story with CacheCade and OPEN-E without corrupt data? Or is it usual that the data are corrupt using LSI Cachecade?

Thanks a lot in advance

Henri

We have been working with the LSI Cachecade for a long time and have not heard of such an incident, we have the drivers inside of the DSS V6 and V7 but the drivers do not corrupt the data, this would be on the hardware side. The best would be to use the Snapshot function of the DSS and you can create tasks so that the Snapshot will be available for 3 weeks, so in the event your hardware has issues you can retrieve the data.

We have many customers who runs LSI with Cashcade, and everything is working normal. It could be the Firmware build with your current controller causing this issue, or there is an issue with one of the disk(s).

In case if the data get corrupted on the Primary box, if the replication service kept running then the mirror will get corrupted too. Having Snapshots, Backup in such cases should help ( in some cases ).

Also you may find the following Forum link and KB useful for you:

http://forum.open-e.com/showthread.p...achecade-2-Pro
http://kb.open-e.com/How-to-dramatic...ance_1293.html

Hi,

thanks for the replies.

I found in the meantime 3 more corrupted filesystems (1*HFS+, 2*SLES 11 SP2). Unfortunately I find this corruption only after rebooting the VMs.
We have our production data on this box, so a snapshot and revert to snapshot for invoicing data, Mail server data etc. is maybe not the best idea.
I added a 8 SSD drives to the box and moved the performance critical vmdks to this SSD datastore, the performance is now okay.
Currently I see not way to perform more alpha tests with LSI cachecade, it took too much time and effort to repair the data, the risk to lose data
is too high. To revert to a full backup after 3 weeks or more is usually not an option, to restore the data file by file and verify what's happened in between is maybe possible, but takes a lot of time.

Maybe anybody else can work with LSI to bring the software to a beta stage.

We will see who will complain in the next months.

Thanks
Henri

Hi,

just got the message, that the issue was caused by an outdated MegaRAID driver, Revision 6.12 in the latest version of OPEN-V6,
which does not support CacheCade.

Can you confirm this?

Thanks
Henri

I'm sure if there is a new driver we can provide this in a small update (for Premium Support only), but is there release notes from LSI stating this that if you don't have the latest driver this can happen?

Hi To-M,

me intention was only to benefit by the performance improvement by Cachecade, "just to use".

The hardware vendor wrote me, the data corruption was caused by a outdated driver in OPEN-E (confirmed by OPEN-E). The driver 6.12 was released by LSI mid 2011, LSI CacheCade in Mar 2012. Therefor CacheCade is not supported at the driver version. LSI told him, if disabling Cachecade helped at data corruption, this is a indication of the kind of problems, both OPEN-E and LSI has confirmed the problem.

Just want to know if you can confirm this too (from the OPEN-E standpoint of view) and when a LSI CacheCade compatible version will be released by OPEN-E?

Currently I'am stuck at V6 because I have enabled the NFS HA feature. Is this in the meantime available in V7 (then I can upgrade)?
Sorry, had more than enough trouble with this stuff.

Thanks
Henri

to be precise, Cacheade Pro 2.0 released in Mar 2012.

Henri

I don't believe this is true.

We have 1000's of V6 and V7 in production and we would have updated the LSI Megaraid SAS drivers from the LSI release notes if this was true.

I have checked our support DB and have not found any records of this associated with any support tickets of this issue. What was the ticket that Open-E stated the same as LSI?

We don't make the driver and the Cachecade is an addon feature to certain LSI controllers. Normally the firmware would control the Cachecade along with some parts of the driver to perform the algorithms it uses for hot data.

I would also like to know the case w/ LSI where they stated this, you can submit a support ticket to Open-E with the contact information.

We have several webinars about the LSI Cachecade and I did one with the LSI engineer in March 28th of 2012 and nothing was mentioned about a out dated driver that can cause corruption.

Again we don't make the drivers, the vendors do so DSS would not be involved here. If there was corruption then it could be where the SSD for the cache was a RAID 0 and not a RAID 1 thus if losing the SSD in a RAID 0 then yes you can have corruption for the Writes at that time the SSD went bad but not for the Reads.

Hi Todd,

I wrote a main to Janusz Bak, in copy to support@open-e.com, with the statement of my hardware vendor.

It's in german (can read it too?), maybe Janusz can clarify this.

Thanks again
Henri

Yes, saw the email and saw what LSI stated that the release was in March of last year, so we did have the driver then as I did the video with LSI last year in March of the 28th. Again we get the drivers from LSI and have the small updates for them. Looks like you got the small updates already from Janusz.

Hi Henri,

I've also had massive problems with data-corruption on a production system. My setup is two DSS 7 boxes as an ISCSI SAN in an active/active fail-over / replication cluster. I think I've now narrowed the cause down to LSI firmware v23.16.0-0018 having a problem with the CacheCade write cache. With the cache turned off, or read only there is no issue. As soon as I set up a write cache my VM's start catastrophically corrupting, particularly after write intensive operations - e.g. Windows Updates. I lost about 20 production servers due to this - complete disaster.

Todd - I opened a support ticket #1035676 on 11th July with this issue so I'm surprised you say you aren't aware of this. I have had CacheCade up and running with no issues prior to this firmware upgrade, so it's true that it's generally very stable, but I'd strongly advise anyone running a CacheCade write cache not to upgrade to the latest firmware.

I'd agree with Todd that it's unlikely to be a driver issue as all the caching is handled by the RAID card and is invisible to the host system. However it seems odd that if you Google "23.16.0-0018 corruption" this thread is one of only two hits you get so I do wonder if there is something about the combination of this firmware and DSS that causes an issue. One would think that if the problem was more general LSI would be aware of the issue and would have pulled the firmware by now.

I'm still testing, but the problem does seem to go away if I revert to an older LSI firmware.

Tim

Henri,

This case of yours is being investigated by the QA team and we did contact LSI here in the USA and they do not know of any issues, they also wanted us or you to reproduce this and let them know. We have a huge customer base using them and I am not defending them just saying until we can prove it is LSI firmware and reproduce the issue then we can inform them. Also did you submit a support ticket with LSI about this? Again we get the drivers from them and until it is reproducible then I cant say it is for sure the driver or firmware.

Hi Todd,

I can reproduce this. Roll back to the 23.12.0.0013 firmware and all is ok. If I move to the 23.16.0-0018 firmware the corruption starts happening again.

If there is anything I can do to help your investigations let me know ASAP, I'd be glad to help.

Tim

Quote:

---

Originally Posted by HenriHoffmann

Hi,

habe here an SuperMicro with LSI based certified OPEN-E Exomium S200 box with a LSI 9265i (with batt. backup and Cachecade 2.0 Pro).

---

Hello Henri,

I would like to correct your statement.
Exomium S200 is not certified server with Open-E.

Regarding your case, our Development Team is working on it at the moment.

Tim,

I am sending this link to the LSI guys, did you show LSI your test results showing them this is issue? Seems our guys after testing believe they have reproduced the same issue.

Thanks for the update Todd.

My conversation with LSI ended when they told me I should replace my SSDs with ones on their approved hardware list. Frankly I've not found their first line support very helpful. If their first line support aren't listening to their customers problems but are trying to find excuses for not helping then it doesn't surprise me it takes them weeks for them to recongise a problem with their firmware. Hopefully you'll be able to escalate this to someone who will take notice.

Glad you guys have been able to reproduce the issue, I was starting to think I was alone in having this problem.

Would appreciate any updates on this.

Thanks,

Tim

FYI - LSI informed me that they have a new firmware release for the LSI 9265 > 5.8 codeset that was released on 9-23-2013

Hi Todd,

I moved the most important VMs to a additional SSD Raid5 Array and updated the DSS V6 to "6.0up99.8101.8328 64bit", even with the small update I ended at a old LSI driver version in V6 (2011 or so).
Is it now safe to reenable CacheCade after updating the LSI FW to the most current level?
Btw. if you confirm that the S200 box from xtivate.de is not certified, I will ask the guy why the sold me a S200 and 200.

Thanks
Henri

Hi Henri,

We have 100's + different builds in testing so I am not sure about this build that you have "8328" as we try to stay with the official releases, currently there is still build "7337" now that is the latest build for V6. V7 release that came out has the latest drivers, small update for V6 would have to be created. So far I have not seen issues with the latest firmware from LSI for both V6 and or V7 so I would assume they have resolved it. I did not see xtivate.de on our list where they have done there certification with us as one of the Manufactures that are listed.
http://www.open-e.com/partners/certified-systems/

LSI Have released a new firmware 23.18.0.0014 (MR 5.8 P1) Nov 26, 2013 and the release notes include the following fixes:-

SCGCQ00525647 (DFCT) - Cachecade 2.0 with MR 5.7 fw peformance is much lower than MR5.6
SCGCQ00552847 (DFCT) - Data Integrity issue found with CacheCade when CacheCade disassociation is initiated after running IO

I note they have also removed MR 5.6 & 5.7 from their download archive (the problem one and the preceding one). I resolved my problem by rolling back to MR 5.6 which seems to be stable with Cachecade write cache on.

Is anyone brave enough to try the new version and see if the problem is resolved in this? Not sure if I want to risk it on production servers after all the problems this caused the last time I upgraded.

Tim

Hi,

we also have the problems described here. Open-e DSS V7, A-A-failover, iSCSI, LSI with CacheCade and MR 5.8. Massive data corruption, corresponding to write IO.

We'd like to set CacheCade to read-only, as a first workaround. Any hint how to do this?

Regards,

Robert

you should be able to change the caching using the megaRAID Storage manager.

Hi Gr-R,

well, I'm able to disable caching completely. There's no read-only.

We found a P1 patch for this release. It can be infused without a reboot. We'll try this.

Well, after upgrading the LSI new firmware ... MR _5.9_ which includes the P1 patch, we haven't new data loss. We're still busy with restoring damaged systems. But all damages that we still found, arise in the time before the upgrade.

Robert