Error messages and system hangs

Hi

We habe to DSS v6 Machines running with iSCSI Failover

Every few days we get some error messages:

Code:

2009/09/08 03:23:01 kernel:CPU 1 2009/09/08 03:23:01 kernel:Pid: 1175, comm: bonding Tainted: G D 2.6.27.10-oe64-00000-g9b2116f #12 2009/09/08 03:23:01 kernel:RIP: 0010:[<ffffffff802961b8>] [<ffffffff802961b8>] filp_close+0x18/0xa0 2009/09/08 03:23:01 kernel:RSP: 0000:ffff8800ad9d9f48 EFLAGS: 00010246 2009/09/08 03:23:01 kernel:RAX: fffffffffffffff7 RBX: 0008ab5abe171500 RCX: 0000000000000003 2009/09/08 03:23:01 kernel:RDX: 0000000000000000 RSI: ffff88013e73a9c0 RDI: 0008ab5abe171500 2009/09/08 03:23:01 kernel:RBP: 0000000000000003 R08: 0000000000000003 R09: 00000000ffdfead8 2009/09/08 03:23:01 kernel:R10: ffff8800ad9d8000 R11: 0000000000000000 R12: 0000000000000000 2009/09/08 03:23:01 kernel:R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 2009/09/08 03:23:01 kernel:FS: 0000000000000000(0000) GS:ffff88013f871dc0(0063) knlGS:00000000f7d666c0 2009/09/08 03:23:01 kernel:CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b 2009/09/08 03:23:01 kernel:CR2: 0000000009f66268 CR3: 000000007fb73000 CR4: 00000000000006a0 2009/09/08 03:23:01 kernel:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2009/09/08 03:23:01 kernel:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2009/09/08 03:23:01 kernel:Process bonding (pid: 1175, threadinfo ffff8800ad9d8000, task ffff88013b798b50) 2009/09/08 03:23:01 kernel:Stack: ffff88013e73a9c0 0000000000000003 0008ab5abe171500 ffffffff802962cc 2009/09/08 03:23:01 kernel:0000000000000003 00000000ffdfead8 0000000000000000 ffffffff80228282 2009/09/08 03:23:01 kernel:0000000000000000 0000000000000000 0000000000000000 0000000000000000 2009/09/08 03:23:01 kernel:Call Trace: 2009/09/08 03:23:01 kernel:[<ffffffff802962cc>] ? sys_close+0x8c/0xf0 2009/09/08 03:23:01 kernel:[<ffffffff80228282>] ? ia32_sysret+0x0/0xa 2009/09/08 03:23:01 kernel: 2009/09/08 03:23:01 kernel: 2009/09/08 03:23:01 kernel:Code: ff 66 90 89 f2 be 41 02 00 00 e9 f4 fd ff ff 66 66 66 90 48 83 ec 18 48 89 1c 24 48 89 6c 24 08 48 89 fb 4c 89 64 24 10 45 31 e4 <48> 83 7f 28 00 48 89 f5 74 50 48 8b 47 20 48 85 c0 75 35 48 89 2009/09/08 03:23:01 kernel:RSP <ffff8800ad9d9f48> 2009/09/08 03:23:01 kernel:PGD 203067 PUD 204067 PMD 0 2009/09/08 03:23:01 kernel:CPU 1 2009/09/08 03:23:01 kernel:Pid: 1175, comm: bonding Tainted: G D 2.6.27.10-oe64-00000-g9b2116f #12 2009/09/08 03:23:01 kernel:RIP: 0010:[<ffffffff802961b8>] [<ffffffff802961b8>] filp_close+0x18/0xa0 2009/09/08 03:23:01 kernel:RSP: 0000:ffff8800ad9d9db8 EFLAGS: 00010246 2009/09/08 03:23:01 kernel:RAX: ffff8800bdbd2810 RBX: ffffffffffffad80 RCX: 0000000000000003 2009/09/08 03:23:01 kernel:RDX: ffff8800bdbd2800 RSI: ffff88013e73a9c0 RDI: ffffffffffffad80 2009/09/08 03:23:01 kernel:RBP: 0000000000000002 R08: ffff8800ad9d8000 R09: 0000000000000000 2009/09/08 03:23:01 kernel:R10: 0000000000000200 R11: 0000000022100000 R12: 0000000000000000 2009/09/08 03:23:01 kernel:R13: ffff88013e73a9c0 R14: 0000000000000001 R15: 0000000000000001 2009/09/08 03:23:01 kernel:FS: 0000000000000000(0000) GS:ffff88013f871dc0(0000) knlGS:0000000000000000 2009/09/08 03:23:01 kernel:CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b 2009/09/08 03:23:01 kernel:CR2: ffffffffffffada8 CR3: 000000012a5e8000 CR4: 00000000000006a0 2009/09/08 03:23:01 kernel:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2009/09/08 03:23:01 kernel:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2009/09/08 03:23:01 kernel:Process bonding (pid: 1175, threadinfo ffff8800ad9d8000, task ffff88013b798b50) 2009/09/08 03:23:01 kernel:Stack: 0000000000000001 0000000000000002 ffff8801299ad200 ffffffff80237390 2009/09/08 03:23:01 kernel:000000000000000b ffff88013b798b50 0000000000000000 0000000000000000 2009/09/08 03:23:01 kernel:0000000000000000 ffffffff80237a9b ffff88013df8e000 ffff880000000000 2009/09/08 03:23:01 kernel:Call Trace: 2009/09/08 03:23:01 kernel:[<ffffffff80237390>] ? put_files_struct+0x70/0xc0 2009/09/08 03:23:01 kernel:[<ffffffff80237a9b>] ? do_exit+0x17b/0x8b0 2009/09/08 03:23:01 kernel:[<ffffffff8069cf27>] ? oops_end+0x87/0x90 2009/09/08 03:23:01 kernel:[<ffffffff8069cab9>] ? error_exit+0x0/0x51 2009/09/08 03:23:01 kernel:[<ffffffff802961b8>] ? filp_close+0x18/0xa0 2009/09/08 03:23:01 kernel:[<ffffffff802962cc>] ? sys_close+0x8c/0xf0 2009/09/08 03:23:01 kernel:[<ffffffff80228282>] ? ia32_sysret+0x0/0xa 2009/09/08 03:23:01 kernel: 2009/09/08 03:23:01 kernel: 2009/09/08 03:23:01 kernel:Code: ff 66 90 89 f2 be 41 02 00 00 e9 f4 fd ff ff 66 66 66 90 48 83 ec 18 48 89 1c 24 48 89 6c 24 08 48 89 fb 4c 89 64 24 10 45 31 e4 <48> 83 7f 28 00 48 89 f5 74 50 48 8b 47 20 48 85 c0 75 35 48 89 2009/09/08 03:23:01 kernel:RSP <ffff8800ad9d9db8>

Code:

2009/09/08 04:54:02 kernel:CPU 4 2009/09/08 04:54:02 kernel:Pid: 14719, comm: check_cs Tainted: G D 2.6.27.10-oe64-00000-g9b2116f #12 2009/09/08 04:54:02 kernel:RIP: 0010:[<ffffffff802961b8>] [<ffffffff802961b8>] filp_close+0x18/0xa0 2009/09/08 04:54:02 kernel:RSP: 0000:ffff8800bdae7f48 EFLAGS: 00010246 2009/09/08 04:54:02 kernel:RAX: ffffffffffffffef RBX: 004000001a010045 RCX: 0000000000000004 2009/09/08 04:54:02 kernel:RDX: 0000000000000000 RSI: ffff88013d1243c0 RDI: 004000001a010045 2009/09/08 04:54:02 kernel:RBP: 0000000000000004 R08: 0000000000000004 R09: 00000000fffc9488 2009/09/08 04:54:02 kernel:R10: ffff8800bdae6000 R11: 0000000000000000 R12: 0000000000000000 2009/09/08 04:54:02 kernel:R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 2009/09/08 04:54:02 kernel:FS: 0000000000000000(0000) GS:ffff88013f8d49c0(0063) knlGS:00000000f7e306c0 2009/09/08 04:54:02 kernel:CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b 2009/09/08 04:54:02 kernel:CR2: 00000000080905e0 CR3: 00000000bdba3000 CR4: 00000000000006a0 2009/09/08 04:54:02 kernel:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2009/09/08 04:54:02 kernel:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2009/09/08 04:54:02 kernel:Process check_cs (pid: 14719, threadinfo ffff8800bdae6000, task ffff88013d7b3390) 2009/09/08 04:54:02 kernel:Stack: ffff88013d1243c0 0000000000000004 004000001a010045 ffffffff802962cc 2009/09/08 04:54:02 kernel:0000000000000004 00000000fffc9488 0000000000000000 ffffffff80228282 2009/09/08 04:54:02 kernel:0000000000000000 0000000000000000 0000000000000000 0000000000000000 2009/09/08 04:54:02 kernel:Call Trace: 2009/09/08 04:54:02 kernel:[<ffffffff802962cc>] ? sys_close+0x8c/0xf0 2009/09/08 04:54:02 kernel:[<ffffffff80228282>] ? ia32_sysret+0x0/0xa 2009/09/08 04:54:02 kernel: 2009/09/08 04:54:02 kernel: 2009/09/08 04:54:02 kernel:Code: ff 66 90 89 f2 be 41 02 00 00 e9 f4 fd ff ff 66 66 66 90 48 83 ec 18 48 89 1c 24 48 89 6c 24 08 48 89 fb 4c 89 64 24 10 45 31 e4 <48> 83 7f 28 00 48 89 f5 74 50 48 8b 47 20 48 85 c0 75 35 48 89 2009/09/08 04:54:02 kernel:RSP <ffff8800bdae7f48> 2009/09/08 04:54:02 kernel:CPU 4 2009/09/08 04:54:02 kernel:Pid: 14719, comm: check_cs Tainted: G D 2.6.27.10-oe64-00000-g9b2116f #12 2009/09/08 04:54:02 kernel:RIP: 0010:[<ffffffff802961b8>] [<ffffffff802961b8>] filp_close+0x18/0xa0 2009/09/08 04:54:02 kernel:RSP: 0000:ffff8800bdae7db8 EFLAGS: 00010246 2009/09/08 04:54:02 kernel:RAX: ffff88013b352010 RBX: db5abe1715001600 RCX: 0000000000000032 2009/09/08 04:54:02 kernel:RDX: ffff88013b352000 RSI: ffff88013d1243c0 RDI: db5abe1715001600 2009/09/08 04:54:02 kernel:RBP: 0000000000000002 R08: ffff88013d8e8ff0 R09: 000013519eba9e10 2009/09/08 04:54:02 kernel:R10: ffff88002802baf0 R11: ffffffff803fde80 R12: 0000000000000000 2009/09/08 04:54:02 kernel:R13: ffff88013d1243c0 R14: 0000000000000001 R15: 0000000000000001 2009/09/08 04:54:02 kernel:FS: 0000000000000000(0000) GS:ffff88013f8d49c0(0000) knlGS:0000000000000000 2009/09/08 04:54:02 kernel:CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b 2009/09/08 04:54:02 kernel:CR2: 00000000080905e0 CR3: 000000012ae23000 CR4: 00000000000006a0 2009/09/08 04:54:02 kernel:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2009/09/08 04:54:02 kernel:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2009/09/08 04:54:02 kernel:Process check_cs (pid: 14719, threadinfo ffff8800bdae6000, task ffff88013d7b3390) 2009/09/08 04:54:02 kernel:Stack: 0000000000000001 0000000000000002 ffff88013e0f4740 ffffffff80237390 2009/09/08 04:54:02 kernel:000000000000000b ffff88013d7b3390 0000000000000000 0000000000000000 2009/09/08 04:54:02 kernel:0000000000000000 ffffffff80237a9b ffff88013df8e000 ffff880000000000 2009/09/08 04:54:02 kernel:Call Trace: 2009/09/08 04:54:02 kernel:[<ffffffff80237390>] ? put_files_struct+0x70/0xc0 2009/09/08 04:54:02 kernel:[<ffffffff80237a9b>] ? do_exit+0x17b/0x8b0 2009/09/08 04:54:02 kernel:[<ffffffff8069cf27>] ? oops_end+0x87/0x90 2009/09/08 04:54:02 kernel:[<ffffffff8069cab9>] ? error_exit+0x0/0x51 2009/09/08 04:54:02 kernel:[<ffffffff802961b8>] ? filp_close+0x18/0xa0 2009/09/08 04:54:02 kernel:[<ffffffff802962cc>] ? sys_close+0x8c/0xf0 2009/09/08 04:54:02 kernel:[<ffffffff80228282>] ? ia32_sysret+0x0/0xa 2009/09/08 04:54:02 kernel: 2009/09/08 04:54:02 kernel: 2009/09/08 04:54:02 kernel:Code: ff 66 90 89 f2 be 41 02 00 00 e9 f4 fd ff ff 66 66 66 90 48 83 ec 18 48 89 1c 24 48 89 6c 24 08 48 89 fb 4c 89 64 24 10 45 31 e4 <48> 83 7f 28 00 48 89 f5 74 50 48 8b 47 20 48 85 c0 75 35 48 89

This is the primary machine, the second takes over the shares and the primary system hangs. We have to do a reset to get this machine online again.

Where´s the problem?

Thanks

Here are some other error messages we get:

Code:

2009/09/08 11:52:08 kernel:CPU 2 2009/09/08 11:52:08 kernel:Pid: 22750, comm: apache2ctl Tainted: G D 2.6.27.10-oe64-00000-g9b2116f #12 2009/09/08 11:52:08 kernel:RIP: 0010:[<ffffffff802961b8>] [<ffffffff802961b8>] filp_close+0x18/0xa0 2009/09/08 11:52:08 kernel:RSP: 0000:ffff88007fb99ea8 EFLAGS: 00010246 2009/09/08 11:52:08 kernel:RAX: ffff88013f24c010 RBX: db5abe1715003e40 RCX: 0000000000000100 2009/09/08 11:52:08 kernel:RDX: ffff88013f24c000 RSI: ffff880129471680 RDI: db5abe1715003e40 2009/09/08 11:52:08 kernel:RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000000000002 2009/09/08 11:52:08 kernel:R10: 0000000000000008 R11: ffffffff803fde80 R12: 0000000000000000 2009/09/08 11:52:08 kernel:R13: ffff880129471680 R14: 0000000000000001 R15: 0000000000000001 2009/09/08 11:52:08 kernel:FS: 0000000000000000(0000) GS:ffff88013f871740(0000) knlGS:0000000000000000 2009/09/08 11:52:08 kernel:CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b 2009/09/08 11:52:08 kernel:CR2: 00000000f7f6dacc CR3: 0000000000201000 CR4: 00000000000006a0 2009/09/08 11:52:08 kernel:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2009/09/08 11:52:08 kernel:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2009/09/08 11:52:08 kernel:Process apache2ctl (pid: 22750, threadinfo ffff88007fb98000, task ffff88013d79b890) 2009/09/08 11:52:08 kernel:Stack: 0000000000000001 0000000000000002 ffff88013e76e500 ffffffff80237390 2009/09/08 11:52:08 kernel:0000000000000000 ffff88013d79b890 0000000000000000 0000000000000000 2009/09/08 11:52:08 kernel:0000000000000000 ffffffff80237a9b 00000000fffffff2 0000000000000000 2009/09/08 11:52:08 kernel:Call Trace: 2009/09/08 11:52:08 kernel:[<ffffffff80237390>] ? put_files_struct+0x70/0xc0 2009/09/08 11:52:08 kernel:[<ffffffff80237a9b>] ? do_exit+0x17b/0x8b0 2009/09/08 11:52:08 kernel:[<ffffffff80238244>] ? do_group_exit+0x34/0xa0 2009/09/08 11:52:08 kernel:[<ffffffff80228282>] ? ia32_sysret+0x0/0xa 2009/09/08 11:52:08 kernel: 2009/09/08 11:52:08 kernel: 2009/09/08 11:52:08 kernel:Code: ff 66 90 89 f2 be 41 02 00 00 e9 f4 fd ff ff 66 66 66 90 48 83 ec 18 48 89 1c 24 48 89 6c 24 08 48 89 fb 4c 89 64 24 10 45 31 e4 <48> 83 7f 28 00 48 89 f5 74 50 48 8b 47 20 48 85 c0 75 35 48 89 2009/09/08 11:52:08 kernel:RSP <ffff88007fb99ea8>

Code:

2009/09/08 06:19:54 kernel:CPU 6 2009/09/08 06:19:54 kernel:Pid: 23675, comm: apache2ctl Tainted: G D 2.6.27.10-oe64-00000-g9b2116f #12 2009/09/08 06:19:54 kernel:RIP: 0010:[<ffffffff802961b8>] [<ffffffff802961b8>] filp_close+0x18/0xa0 2009/09/08 06:19:54 kernel:RSP: 0000:ffff88002f4c7ea8 EFLAGS: 00010246 2009/09/08 06:19:54 kernel:RAX: ffff88013d356810 RBX: db5abe1715005cc0 RCX: 0000000000000003 2009/09/08 06:19:54 kernel:RDX: ffff88013d356800 RSI: ffff88013ebd9080 RDI: db5abe1715005cc0 2009/09/08 06:19:54 kernel:RBP: 0000000000000002 R08: ffff88002f4c6000 R09: 0000000000000000 2009/09/08 06:19:54 kernel:R10: ffff880028073af0 R11: ffff88013fa43858 R12: 0000000000000000 2009/09/08 06:19:54 kernel:R13: ffff88013ebd9080 R14: 0000000000000001 R15: 0000000000000001 2009/09/08 06:19:54 kernel:FS: 0000000000000000(0000) GS:ffff88013f9afbc0(0000) knlGS:0000000000000000 2009/09/08 06:19:54 kernel:CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b 2009/09/08 06:19:54 kernel:CR2: 00000000f7f6eacc CR3: 0000000000201000 CR4: 00000000000006a0 2009/09/08 06:19:54 kernel:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2009/09/08 06:19:54 kernel:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2009/09/08 06:19:54 kernel:Process apache2ctl (pid: 23675, threadinfo ffff88002f4c6000, task ffff88007fab2790) 2009/09/08 06:19:54 kernel:Stack: 0000000000000001 0000000000000002 ffff88013d07dd00 ffffffff80237390 2009/09/08 06:19:54 kernel:0000000000000000 ffff88007fab2790 0000000000000000 0000000000000000 2009/09/08 06:19:54 kernel:0000000000000000 ffffffff80237a9b 00000000fffffff2 0000000000000000 2009/09/08 06:19:54 kernel:Call Trace: 2009/09/08 06:19:54 kernel:[<ffffffff80237390>] ? put_files_struct+0x70/0xc0 2009/09/08 06:19:54 kernel:[<ffffffff80237a9b>] ? do_exit+0x17b/0x8b0 2009/09/08 06:19:54 kernel:[<ffffffff80238244>] ? do_group_exit+0x34/0xa0 2009/09/08 06:19:54 kernel:[<ffffffff80228282>] ? ia32_sysret+0x0/0xa 2009/09/08 06:19:54 kernel: 2009/09/08 06:19:54 kernel: 2009/09/08 06:19:54 kernel:Code: ff 66 90 89 f2 be 41 02 00 00 e9 f4 fd ff ff 66 66 66 90 48 83 ec 18 48 89 1c 24 48 89 6c 24 08 48 89 fb 4c 89 64 24 10 45 31 e4 <48> 83 7f 28 00 48 89 f5 74 50 48 8b 47 20 48 85 c0 75 35 48 89 2009/09/08 06:19:54 kernel:RSP <ffff88002f4c7ea8>

Could be problems with memory
what version and how much RAM do you have in your system?

May also want to send logs to support to see what they say.
keep us posted

Have 4 GB in each machine.
I change the memory and try it again.

What did support say about the logs?
did you try to run a memory test?
Is it only happening on the primary machine ?

Haven´t sent the logs to the support yet.
A memory test did not show any errors.
And yes, it´s only at the primary machine.

That looks very similar to an error we got a few weeks ago. I recommend you open a ticket with support and have them look at the logs.

Opened a ticket.
Could you tell me what the error was in your case?

I got this on one of my DSS units (configured for iSCSI failover). I basically lost web access to the unit after this error occurred, attempted to reboot via console, and was unable to do that (it appeard to hang on the shutdown process). I had to go to the datacenter and power it off and back on. The iSCSI replication took over so everything was presented from our second DSS, although we had some issues with that as well. It was a long night to get everything back up.

Code:

2009/08/31 00:03:01|------------[ cut here ]------------ 2009/08/31 00:03:01|CPU 4 2009/08/31 00:03:01|Pid: 22488, comm: iscsi-scstd Not tainted 2.6.27.10-oe64-00000-g9b2116f #12 2009/08/31 00:03:01|RIP: 0010:[<ffffffffa01cad78>] [<ffffffffa01cad78>] session_free+0x138/0x140 [iscsi_scst] 2009/08/31 00:03:01|RSP: 0000:ffff88007faed848 EFLAGS: 00210286 2009/08/31 00:03:01|RAX: ffff880106f69088 RBX: ffff880106f68000 RCX: 0000000000000000 2009/08/31 00:03:01|RDX: ffff880106f69000 RSI: 00000000ffffffff RDI: ffff880106f68000 2009/08/31 00:03:01|RBP: 000005d8210f0040 R08: ffff880106f68000 R09: ffff880115f8e920 2009/08/31 00:03:01|R10: ffff88012e6ee4d0 R11: 0000000000000000 R12: ffff88011efa2e20 2009/08/31 00:03:01|R13: ffff88007faedb78 R14: 0000000000000000 R15: ffff88011efa2e08 2009/08/31 00:03:01|FS: 0000000000000000(0000) GS:ffff88012f8fd9c0(0063) knlGS:00000000f7cbb6c0 2009/08/31 00:03:01|CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b 2009/08/31 00:03:01|CR2: 00000000093c2000 CR3: 000000007fada000 CR4: 00000000000006a0 2009/08/31 00:03:01|DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2009/08/31 00:03:01|DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2009/08/31 00:03:01|Process iscsi-scstd (pid: 22488, threadinfo ffff88007faec000, task ffff880116d00290) 2009/08/31 00:03:01|Stack: ffff880106f68000 ffffffffa01cb0d9 0000200000000001 0000000000000000 2009/08/31 00:03:01|0100000200010000 ffff88011c2d8000 000005d8210f0040 0000000100000000 2009/08/31 00:03:01|000005d8210f0040 0000080000000000 0000000000000800 ffff88011efa2e00 2009/08/31 00:03:01|Call Trace: 2009/08/31 00:03:01|[<ffffffffa01cb0d9>] ? session_add+0x359/0x480 [iscsi_scst] 2009/08/31 00:03:01|[<ffffffffa01c97b8>] ? ioctl+0x368/0x430 [iscsi_scst] 2009/08/31 00:03:01|[<ffffffff8069c592>] ? __down_read+0x12/0xa0 2009/08/31 00:03:01|[<ffffffff804256c8>] ? __down_write_trylock+0x48/0x60 2009/08/31 00:03:01|[<ffffffff804256c8>] ? __down_write_trylock+0x48/0x60 2009/08/31 00:03:01|[<ffffffffa001d663>] ? di_read_unlock+0x73/0x130 [aufs] 2009/08/31 00:03:01|[<ffffffffa001c497>] ? h_d_revalidate+0x4c7/0x6b0 [aufs] 2009/08/31 00:03:01|[<ffffffff8069c592>] ? __down_read+0x12/0xa0 2009/08/31 00:03:01|[<ffffffff804256c8>] ? __down_write_trylock+0x48/0x60 2009/08/31 00:03:01|[<ffffffff804256c8>] ? __down_write_trylock+0x48/0x60 2009/08/31 00:03:01|[<ffffffff80425701>] ? __up_read+0x21/0xb0 2009/08/31 00:03:01|[<ffffffffa001d663>] ? di_read_unlock+0x73/0x130 [aufs] 2009/08/31 00:03:01|[<ffffffffa001c497>] ? h_d_revalidate+0x4c7/0x6b0 [aufs] 2009/08/31 00:03:01|[<ffffffffa001ce09>] ? aufs_d_revalidate+0x249/0x4f0 [aufs] 2009/08/31 00:03:01|[<ffffffff804256c8>] ? __down_write_trylock+0x48/0x60 2009/08/31 00:03:01|[<ffffffff80425701>] ? __up_read+0x21/0xb0 2009/08/31 00:03:01|[<ffffffff80425701>] ? __up_read+0x21/0xb0 2009/08/31 00:03:01|[<ffffffffa0004a57>] ? aufs_read_unlock+0x37/0x60 [aufs] 2009/08/31 00:03:01|[<ffffffffa001ce09>] ? aufs_d_revalidate+0x249/0x4f0 [aufs] 2009/08/31 00:03:01|[<ffffffff80425701>] ? __up_read+0x21/0xb0 2009/08/31 00:03:01|[<ffffffff802af5c1>] ? mntput_no_expire+0x21/0x120 2009/08/31 00:03:01|[<ffffffff802aa487>] ? __d_lookup+0xb7/0x150 2009/08/31 00:03:01|[<ffffffff80274ea2>] ? __alloc_pages_internal+0x92/0x430 2009/08/31 00:03:01|[<ffffffff80281cc1>] ? handle_mm_fault+0x431/0x840 2009/08/31 00:03:01|[<ffffffff802d0bef>] ? compat_sys_ioctl+0x31f/0x3f0 2009/08/31 00:03:01|[<ffffffff80284efe>] ? do_munmap+0x26e/0x310 2009/08/31 00:03:01|[<ffffffff80425701>] ? __up_read+0x21/0xb0 2009/08/31 00:03:01|[<ffffffff80228282>] ? ia32_sysret+0x0/0xa 2009/08/31 00:03:01| 2009/08/31 00:03:01| 2009/08/31 00:03:01|Code: 48 c7 41 08 00 02 20 00 48 c7 83 a8 10 00 00 00 01 10 00 48 8b bb c8 10 00 00 e8 d4 74 0c e0 48 89 df e8 cc 74 0c e0 5b 31 c0 c3 <0f> 0b eb fe 66 66 66 90 41 57 45 31 ff 41 56 41 55 49 89 f5 be 2009/08/31 00:03:01|RSP <ffff88007faed848>

Looks very similar to our Problem
How do you resolved this problem?

Hey jiassic and chkohlruss

the errors that you posted are different,
jiassic may be drive related
he is showing these:
2009/08/31 00:03:01|[<ffffffff8069c592>] ? __down_read+0x12/0xa0
2009/08/31 00:03:01|[<ffffffff804256c8>] ? __down_write_trylock+0x48/0x60
2009/08/31 00:03:01|[<ffffffff804256c8>] ? __down_write_trylock+0x48/0x60
2009/08/31 00:03:01|[<ffffffffa001d663>] ? di_read_unlock+0x73/0x130 [aufs]
2009/08/31 00:03:01|[<ffffffffa001c497>] ? h_d_revalidate+0x4c7/0x6b0 [aufs]
2009/08/31 00:03:01|[<ffffffff8069c592>] ? __down_read+0x12/0xa0
2009/08/31 00:03:01|[<ffffffff804256c8>] ? __down_write_trylock+0x48/0x60
2009/08/31 00:03:01|[<ffffffff804256c8>] ? __down_write_trylock+0x48/0x60
2009/08/31 00:03:01|[<ffffffff80425701>] ? __up_read+0x21/0xb0
2009/08/31 00:03:01|[<ffffffffa001d663>] ? di_read_unlock+0x73/0x130 [aufs]

while chkohlruss
you have different errors
2009/09/08 06:19:54 kernel:[<ffffffff80237390>] ? put_files_struct+0x70/0xc0

2009/09/08 06:19:54 kernel:[<ffffffff80237a9b>] ? do_exit+0x17b/0x8b0

2009/09/08 06:19:54 kernel:[<ffffffff80238244>] ? do_group_exit+0x34/0xa0

2009/09/08 06:19:54 kernel:[<ffffffff80228282>] ? ia32_sysret+0x0/0xa

Best is to send the entire logs to suport to be sure.
Did you guys hear back from them?

Support sent me a patch file, which I believe is an update to SCST (version 1.0.1.1). I have not actually implemented the patch due to the necessity of shutting down all of my VM's to do the upgrade. That's a pretty hefty maintenance window to try to schedule.

Symm, thanks for the info. How do you find out what these error messages refer to?

I looked a little deeper and found that our RAID controller (3ware 9690SA) does a scheduled verify every night at 12:00 AM, and the event log in the RAID controller shows that the job kicks off at 12:03 - the same time as the error message timestamp above. It does this every night. Is it possible that the increased drive usage as it verifies the RAID5 parity information would cause the DSS error?

Support says it is the RAM.
We´ve done several memtests, no errors found.
We´ve change the RAM with a new one, still got the error messages.

jisaac

errors that I have seen in past life (telecom)
we used linux and raid for storage

What build of DSS 6 are you running? We were running build 3535 and seeing similar issues were our SAN would act like it just froze up. Support had us upgrade to build 3537 and problem solved. The issue was with the system cache. A reboot would fix it for a while and them it would freek out again. High data transfer agrivates this issus.

6.0up06.8102.3535 64bit

Like I said earlier, support sent me a patch file which I haven't had the downtime to implement yet. I'll try to get that done this weekend.

We are running 6.0up04.8101.3530 64bit
Are there any updates i can donwload and install?

We have reinstalled and reconfigured the primary server and it seems that the errors are gone now.