Error messages and system hangs
Hi
We habe to DSS v6 Machines running with iSCSI Failover
Every few days we get some error messages:
Code:2009/09/08 03:23:01 kernel:CPU 1 2009/09/08 03:23:01 kernel:Pid: 1175, comm: bonding Tainted: G D 2.6.27.10-oe64-00000-g9b2116f #12 2009/09/08 03:23:01 kernel:RIP: 0010:[<ffffffff802961b8>] [<ffffffff802961b8>] filp_close+0x18/0xa0 2009/09/08 03:23:01 kernel:RSP: 0000:ffff8800ad9d9f48 EFLAGS: 00010246 2009/09/08 03:23:01 kernel:RAX: fffffffffffffff7 RBX: 0008ab5abe171500 RCX: 0000000000000003 2009/09/08 03:23:01 kernel:RDX: 0000000000000000 RSI: ffff88013e73a9c0 RDI: 0008ab5abe171500 2009/09/08 03:23:01 kernel:RBP: 0000000000000003 R08: 0000000000000003 R09: 00000000ffdfead8 2009/09/08 03:23:01 kernel:R10: ffff8800ad9d8000 R11: 0000000000000000 R12: 0000000000000000 2009/09/08 03:23:01 kernel:R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 2009/09/08 03:23:01 kernel:FS: 0000000000000000(0000) GS:ffff88013f871dc0(0063) knlGS:00000000f7d666c0 2009/09/08 03:23:01 kernel:CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b 2009/09/08 03:23:01 kernel:CR2: 0000000009f66268 CR3: 000000007fb73000 CR4: 00000000000006a0 2009/09/08 03:23:01 kernel:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2009/09/08 03:23:01 kernel:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2009/09/08 03:23:01 kernel:Process bonding (pid: 1175, threadinfo ffff8800ad9d8000, task ffff88013b798b50) 2009/09/08 03:23:01 kernel:Stack: ffff88013e73a9c0 0000000000000003 0008ab5abe171500 ffffffff802962cc 2009/09/08 03:23:01 kernel:0000000000000003 00000000ffdfead8 0000000000000000 ffffffff80228282 2009/09/08 03:23:01 kernel:0000000000000000 0000000000000000 0000000000000000 0000000000000000 2009/09/08 03:23:01 kernel:Call Trace: 2009/09/08 03:23:01 kernel:[<ffffffff802962cc>] ? sys_close+0x8c/0xf0 2009/09/08 03:23:01 kernel:[<ffffffff80228282>] ? ia32_sysret+0x0/0xa 2009/09/08 03:23:01 kernel: 2009/09/08 03:23:01 kernel: 2009/09/08 03:23:01 kernel:Code: ff 66 90 89 f2 be 41 02 00 00 e9 f4 fd ff ff 66 66 66 90 48 83 ec 18 48 89 1c 24 48 89 6c 24 08 48 89 fb 4c 89 64 24 10 45 31 e4 <48> 83 7f 28 00 48 89 f5 74 50 48 8b 47 20 48 85 c0 75 35 48 89 2009/09/08 03:23:01 kernel:RSP <ffff8800ad9d9f48> 2009/09/08 03:23:01 kernel:PGD 203067 PUD 204067 PMD 0 2009/09/08 03:23:01 kernel:CPU 1 2009/09/08 03:23:01 kernel:Pid: 1175, comm: bonding Tainted: G D 2.6.27.10-oe64-00000-g9b2116f #12 2009/09/08 03:23:01 kernel:RIP: 0010:[<ffffffff802961b8>] [<ffffffff802961b8>] filp_close+0x18/0xa0 2009/09/08 03:23:01 kernel:RSP: 0000:ffff8800ad9d9db8 EFLAGS: 00010246 2009/09/08 03:23:01 kernel:RAX: ffff8800bdbd2810 RBX: ffffffffffffad80 RCX: 0000000000000003 2009/09/08 03:23:01 kernel:RDX: ffff8800bdbd2800 RSI: ffff88013e73a9c0 RDI: ffffffffffffad80 2009/09/08 03:23:01 kernel:RBP: 0000000000000002 R08: ffff8800ad9d8000 R09: 0000000000000000 2009/09/08 03:23:01 kernel:R10: 0000000000000200 R11: 0000000022100000 R12: 0000000000000000 2009/09/08 03:23:01 kernel:R13: ffff88013e73a9c0 R14: 0000000000000001 R15: 0000000000000001 2009/09/08 03:23:01 kernel:FS: 0000000000000000(0000) GS:ffff88013f871dc0(0000) knlGS:0000000000000000 2009/09/08 03:23:01 kernel:CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b 2009/09/08 03:23:01 kernel:CR2: ffffffffffffada8 CR3: 000000012a5e8000 CR4: 00000000000006a0 2009/09/08 03:23:01 kernel:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2009/09/08 03:23:01 kernel:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2009/09/08 03:23:01 kernel:Process bonding (pid: 1175, threadinfo ffff8800ad9d8000, task ffff88013b798b50) 2009/09/08 03:23:01 kernel:Stack: 0000000000000001 0000000000000002 ffff8801299ad200 ffffffff80237390 2009/09/08 03:23:01 kernel:000000000000000b ffff88013b798b50 0000000000000000 0000000000000000 2009/09/08 03:23:01 kernel:0000000000000000 ffffffff80237a9b ffff88013df8e000 ffff880000000000 2009/09/08 03:23:01 kernel:Call Trace: 2009/09/08 03:23:01 kernel:[<ffffffff80237390>] ? put_files_struct+0x70/0xc0 2009/09/08 03:23:01 kernel:[<ffffffff80237a9b>] ? do_exit+0x17b/0x8b0 2009/09/08 03:23:01 kernel:[<ffffffff8069cf27>] ? oops_end+0x87/0x90 2009/09/08 03:23:01 kernel:[<ffffffff8069cab9>] ? error_exit+0x0/0x51 2009/09/08 03:23:01 kernel:[<ffffffff802961b8>] ? filp_close+0x18/0xa0 2009/09/08 03:23:01 kernel:[<ffffffff802962cc>] ? sys_close+0x8c/0xf0 2009/09/08 03:23:01 kernel:[<ffffffff80228282>] ? ia32_sysret+0x0/0xa 2009/09/08 03:23:01 kernel: 2009/09/08 03:23:01 kernel: 2009/09/08 03:23:01 kernel:Code: ff 66 90 89 f2 be 41 02 00 00 e9 f4 fd ff ff 66 66 66 90 48 83 ec 18 48 89 1c 24 48 89 6c 24 08 48 89 fb 4c 89 64 24 10 45 31 e4 <48> 83 7f 28 00 48 89 f5 74 50 48 8b 47 20 48 85 c0 75 35 48 89 2009/09/08 03:23:01 kernel:RSP <ffff8800ad9d9db8>Code:2009/09/08 04:54:02 kernel:CPU 4 2009/09/08 04:54:02 kernel:Pid: 14719, comm: check_cs Tainted: G D 2.6.27.10-oe64-00000-g9b2116f #12 2009/09/08 04:54:02 kernel:RIP: 0010:[<ffffffff802961b8>] [<ffffffff802961b8>] filp_close+0x18/0xa0 2009/09/08 04:54:02 kernel:RSP: 0000:ffff8800bdae7f48 EFLAGS: 00010246 2009/09/08 04:54:02 kernel:RAX: ffffffffffffffef RBX: 004000001a010045 RCX: 0000000000000004 2009/09/08 04:54:02 kernel:RDX: 0000000000000000 RSI: ffff88013d1243c0 RDI: 004000001a010045 2009/09/08 04:54:02 kernel:RBP: 0000000000000004 R08: 0000000000000004 R09: 00000000fffc9488 2009/09/08 04:54:02 kernel:R10: ffff8800bdae6000 R11: 0000000000000000 R12: 0000000000000000 2009/09/08 04:54:02 kernel:R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 2009/09/08 04:54:02 kernel:FS: 0000000000000000(0000) GS:ffff88013f8d49c0(0063) knlGS:00000000f7e306c0 2009/09/08 04:54:02 kernel:CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b 2009/09/08 04:54:02 kernel:CR2: 00000000080905e0 CR3: 00000000bdba3000 CR4: 00000000000006a0 2009/09/08 04:54:02 kernel:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2009/09/08 04:54:02 kernel:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2009/09/08 04:54:02 kernel:Process check_cs (pid: 14719, threadinfo ffff8800bdae6000, task ffff88013d7b3390) 2009/09/08 04:54:02 kernel:Stack: ffff88013d1243c0 0000000000000004 004000001a010045 ffffffff802962cc 2009/09/08 04:54:02 kernel:0000000000000004 00000000fffc9488 0000000000000000 ffffffff80228282 2009/09/08 04:54:02 kernel:0000000000000000 0000000000000000 0000000000000000 0000000000000000 2009/09/08 04:54:02 kernel:Call Trace: 2009/09/08 04:54:02 kernel:[<ffffffff802962cc>] ? sys_close+0x8c/0xf0 2009/09/08 04:54:02 kernel:[<ffffffff80228282>] ? ia32_sysret+0x0/0xa 2009/09/08 04:54:02 kernel: 2009/09/08 04:54:02 kernel: 2009/09/08 04:54:02 kernel:Code: ff 66 90 89 f2 be 41 02 00 00 e9 f4 fd ff ff 66 66 66 90 48 83 ec 18 48 89 1c 24 48 89 6c 24 08 48 89 fb 4c 89 64 24 10 45 31 e4 <48> 83 7f 28 00 48 89 f5 74 50 48 8b 47 20 48 85 c0 75 35 48 89 2009/09/08 04:54:02 kernel:RSP <ffff8800bdae7f48> 2009/09/08 04:54:02 kernel:CPU 4 2009/09/08 04:54:02 kernel:Pid: 14719, comm: check_cs Tainted: G D 2.6.27.10-oe64-00000-g9b2116f #12 2009/09/08 04:54:02 kernel:RIP: 0010:[<ffffffff802961b8>] [<ffffffff802961b8>] filp_close+0x18/0xa0 2009/09/08 04:54:02 kernel:RSP: 0000:ffff8800bdae7db8 EFLAGS: 00010246 2009/09/08 04:54:02 kernel:RAX: ffff88013b352010 RBX: db5abe1715001600 RCX: 0000000000000032 2009/09/08 04:54:02 kernel:RDX: ffff88013b352000 RSI: ffff88013d1243c0 RDI: db5abe1715001600 2009/09/08 04:54:02 kernel:RBP: 0000000000000002 R08: ffff88013d8e8ff0 R09: 000013519eba9e10 2009/09/08 04:54:02 kernel:R10: ffff88002802baf0 R11: ffffffff803fde80 R12: 0000000000000000 2009/09/08 04:54:02 kernel:R13: ffff88013d1243c0 R14: 0000000000000001 R15: 0000000000000001 2009/09/08 04:54:02 kernel:FS: 0000000000000000(0000) GS:ffff88013f8d49c0(0000) knlGS:0000000000000000 2009/09/08 04:54:02 kernel:CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b 2009/09/08 04:54:02 kernel:CR2: 00000000080905e0 CR3: 000000012ae23000 CR4: 00000000000006a0 2009/09/08 04:54:02 kernel:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2009/09/08 04:54:02 kernel:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2009/09/08 04:54:02 kernel:Process check_cs (pid: 14719, threadinfo ffff8800bdae6000, task ffff88013d7b3390) 2009/09/08 04:54:02 kernel:Stack: 0000000000000001 0000000000000002 ffff88013e0f4740 ffffffff80237390 2009/09/08 04:54:02 kernel:000000000000000b ffff88013d7b3390 0000000000000000 0000000000000000 2009/09/08 04:54:02 kernel:0000000000000000 ffffffff80237a9b ffff88013df8e000 ffff880000000000 2009/09/08 04:54:02 kernel:Call Trace: 2009/09/08 04:54:02 kernel:[<ffffffff80237390>] ? put_files_struct+0x70/0xc0 2009/09/08 04:54:02 kernel:[<ffffffff80237a9b>] ? do_exit+0x17b/0x8b0 2009/09/08 04:54:02 kernel:[<ffffffff8069cf27>] ? oops_end+0x87/0x90 2009/09/08 04:54:02 kernel:[<ffffffff8069cab9>] ? error_exit+0x0/0x51 2009/09/08 04:54:02 kernel:[<ffffffff802961b8>] ? filp_close+0x18/0xa0 2009/09/08 04:54:02 kernel:[<ffffffff802962cc>] ? sys_close+0x8c/0xf0 2009/09/08 04:54:02 kernel:[<ffffffff80228282>] ? ia32_sysret+0x0/0xa 2009/09/08 04:54:02 kernel: 2009/09/08 04:54:02 kernel: 2009/09/08 04:54:02 kernel:Code: ff 66 90 89 f2 be 41 02 00 00 e9 f4 fd ff ff 66 66 66 90 48 83 ec 18 48 89 1c 24 48 89 6c 24 08 48 89 fb 4c 89 64 24 10 45 31 e4 <48> 83 7f 28 00 48 89 f5 74 50 48 8b 47 20 48 85 c0 75 35 48 89
This is the primary machine, the second takes over the shares and the primary system hangs. We have to do a reset to get this machine online again.
Where´s the problem?
Thanks
Reply With Quote