DSS6.0up10 Replication problem I/O errors

Printable View

10-29-2009, 08:05 AM
eppo78

DSS6.0up10 Replication problem I/O errors

Hey all,

I'm having serious issues with replicating 1TB FC volumes to my destination DSS.
I'm running DSS6.0up10.8101.3719 64bit on the following hardware;

SuperMicro X7DWN+ Motherboard
64GB Memory
Qlogic QLE2462
Areca ARC-1680 4096MB cache Firmware V1.47 (Raidcontroller)
24 SEAGATE ST3500620SS (SAS 500GB 7200RPM)
1 10GB Areca Volume (DSS6 OS)
1 10990GB Areca Volume (DSS6 DATA)

I have created a FC volume of 1024GB with replication and initialization enabled and gave the volume LUN 0.

I have severely tested the performance with IOMeter 1TB datastores in VMware ESX4 and it is outstanding:) . DO NOT use a 2TB datastore in VMware, you WILL get I/O errors!

However when I create a replication task with a 1GB nic set at 60MB for the 1TB volume to a mirror server, the source server hangs after a period of time.

I get the following errors from mail and the console:
mail: kernel:journal commit I/O error
console: arcmsr0: abort device command of scsi id = 0 lun = 1

You would think this is a raidcontroller issue but I replaced the controller for a new one including the memory and because I severely abjused the system for 3 days continuously using IOMeter, I think it would have crashed then if it was a hardware issue.

So this issue only occurs when the replication task is running.

Has anyone had the same problem?
Is it possible for anyone running DSS6 with a Areca 1680 to test if they can reproduce the same issue?

Your Help Would Be Greatly Appreciated!

(I already send multiple mails to support about this issue, but without response so far)
10-29-2009, 09:33 AM
eppo78

Oh yeah... Because the server frooze up I had to do a hard powercycle.

The result was a corrupted systemvolume that I was not able to repair, all my data was lost :mad:
10-29-2009, 01:53 PM
symm

try using LUN 1
in esx 3.5 it did not like lun 0

Are you replicating through the Qlogic QLE2462?
if not what type of nic are you using
10-29-2009, 02:54 PM
eppo78

Quote:

Originally Posted by symm

try using LUN 1
in esx 3.5 it did not like lun 0

Are you replicating through the Qlogic QLE2462?
if not what type of nic are you using

I don't think it is ESX related as I have the same problem with a empty volume without a LUN assigned to it.

As far as I know you can only replicate through a nic ( Intel 82575EB internal)
Thanks for you reply though.

Eddo
10-29-2009, 02:56 PM
eppo78

The same issue occurs while replicating a 500GB volume...
10-30-2009, 12:30 AM
symm

Did you try sending the logs to support?

did the initilization finish before you started the replication ?

BTW it does not seem like a vmware issue, but here is a link to the post concerning LUN 0

http://kb.vmware.com/selfservice/mic...8601&stateId=0 0 44972208
10-30-2009, 06:50 AM
eppo78

I did wait until initilization finished, I even waited to put data on it until the initial replication would finish, but it never does.

Thanks for the LUN 0 info.
10-30-2009, 06:52 AM
eppo78

I always send my logfiles with the mails I send to support. I'm still waiting for an anwser...
10-30-2009, 11:48 AM
symm

keep us updated with your progress
10-30-2009, 05:54 PM
JFleck

Sounds the same as my issue.

You are not alone with this issue. I've also submitted a ticket and am awaiting a reply. I have two devices that I believe are identical in hardware and software. Replication from device A to Device B seems to be working without a problem. However, Replication from Device B to Device A seems to fail and cause both units to hang and reports a Journal committ error. fortunately I haven't lost any data, but it does require a hard power cycle (reboot command from console doesn't seem to do it) and it trashes any of the replications that I did have. I am using the default 40mb for my replication tasks.
11-02-2009, 07:24 AM
eppo78

Quote:

Originally Posted by JFleck

You are not alone with this issue.

Thanks for your reply! I'm sorry to hear you have the same problem..
Are you using a supermicro chassis?
Are you using a Areca Raid Controller
What kind of disks are you using?

I'm starting to think the problem could be power or disk related. On this areca faq there was someone with the same abort device commands and solved it by upgrading the power supply.

I attached 8 of my 24 disk to a 500watt PS, Ik takes a lot longer for the system to crash:(
I know the power supply is not broken because I swapped the entire chassis.
Maybe the power output is not sufficient?

Still no word from support, In really hope they can point me in the right direction. I'm wasting time chasing a ghost and my manager is not amused:(
11-02-2009, 08:34 AM
stardf

Did you have tested this problem with Areca 1680 and FW 1.46? Same error ?
11-02-2009, 09:51 AM
eppo78

Quote:

Originally Posted by stardf

Did you have tested this problem with Areca 1680 and FW 1.46? Same error ?

No have not tried that yet. I will test this after my current test, I have left 8 HDD's in the system and created a raidset, if it is a power problem then chances are that the issue will no longer occur.
11-03-2009, 06:45 AM
eppo78

Quote:

Originally Posted by eppo78

I have left 8 HDD's in the system and created a raidset, if it is a power problem then chances are that the issue will no longer occur.

The system crashed again so I think I can savely say it is not a power problem.
No word from support yet..
11-03-2009, 05:14 PM
JFleck

Updates

I am using the Areca Controller Card (ARC-1680) in my system as well. I have recently upgraded the firmware to ver 1.47 and am still seeing the problems. Strange thing is that I have two devices that are identical other than the fact that one is failing with that error and the other does not.

Drives are ST31500341AS 1.5TB drives. Have a total of 16 in the system. Support has asked me to open up remote support but can't seem to get that to work right now, I am waiting on a response from them still.

Controller Name ARC-1680
Firmware Version V1.47 2009-07-16
BOOT ROM Version V1.47 2009-06-25
SAS Firmware Version 4.7.3.0

Main Processor 1.2GHz IOP348 C1
CPU ICache Size 32KBytes
CPU DCache Size 32KBytes/Write Back
CPU SCache Size 512KBytes/Write Back
System Memory 2048MB/533MHz/ECC
Current IP Address 10.1.8.101
11-04-2009, 06:45 AM
eppo78

Problem Solved! :d [

I managed to solve the problem!

There is a bios setting on my motherboard (X7DWN+) that sets the 'max payload size'
for PCI-Express devices. This setting was default on 256, when I had set this to 128 the problem was gone.

Areca support (as good as the cards are :) ) has confirmed that the Areca raidcontrollers use a default payload setting of 128.

Replication works fine now! :D
11-04-2009, 01:30 PM
eppo78

Recall Problem NOT solved!

Quote:

Originally Posted by eppo78

I managed to solve the problem!
Replication works fine now! :D

500Gb replication was OK but when I setup the system for 1024GB replication the system crashed after 3.5 hours :mad:

....
11-04-2009, 02:56 PM
eppo78

I've swapped my motherboard for a new Supermicro X8DTN+ with 8GB DDR3 just in case.
The system crashed again...

I downgraded the Areca firmware/bios to 1.46 and setup replication for 1024GB....

I could really use a break..
11-05-2009, 07:15 AM
eppo78

Preliminary tests are successful

Quote:

Originally Posted by stardf

Did you have tested this problem with Areca 1680 and FW 1.46? Same error ?

Thanks again, I should have tried this earlier :o

I've downgraded the firmware/bios to 1.46 and setup replication for 1024GB. The system past 2 initial replications, no crashes.

The release notes of Areca firmware/bios 1.47 state that there has been a modification to the SendAbortTask, maybe thats the issue?

I'll keep on testing and keep you updated on my progress.
If it is still working after a week I'll say the problem is fixed ;)
11-09-2009, 03:55 PM
stardf

No problem :cool:

I had this problem for a while :rolleyes:
11-10-2009, 11:55 AM
eppo78

Quote:

Originally Posted by stardf

No problem :cool:

I had this problem for a while :rolleyes:

Ahh so you watched me suffer ;)

System is still running without problems, case closed.

Thanks for your input.