Error messages and system hangs

[chkohlruss]

Haven´t sent the logs to the support yet.
A memory test did not show any errors.
And yes, it´s only at the primary machine.

[jisaac]

That looks very similar to an error we got a few weeks ago. I recommend you open a ticket with support and have them look at the logs.

[chkohlruss]

Opened a ticket.
Could you tell me what the error was in your case?

[jisaac]

I got this on one of my DSS units (configured for iSCSI failover). I basically lost web access to the unit after this error occurred, attempted to reboot via console, and was unable to do that (it appeard to hang on the shutdown process). I had to go to the datacenter and power it off and back on. The iSCSI replication took over so everything was presented from our second DSS, although we had some issues with that as well. It was a long night to get everything back up.

2009/08/31 00:03:01|------------[ cut here ]------------
2009/08/31 00:03:01|CPU 4 
2009/08/31 00:03:01|Pid: 22488, comm: iscsi-scstd Not tainted 2.6.27.10-oe64-00000-g9b2116f #12
2009/08/31 00:03:01|RIP: 0010:[<ffffffffa01cad78>]  [<ffffffffa01cad78>] session_free+0x138/0x140 [iscsi_scst]
2009/08/31 00:03:01|RSP: 0000:ffff88007faed848  EFLAGS: 00210286
2009/08/31 00:03:01|RAX: ffff880106f69088 RBX: ffff880106f68000 RCX: 0000000000000000
2009/08/31 00:03:01|RDX: ffff880106f69000 RSI: 00000000ffffffff RDI: ffff880106f68000
2009/08/31 00:03:01|RBP: 000005d8210f0040 R08: ffff880106f68000 R09: ffff880115f8e920
2009/08/31 00:03:01|R10: ffff88012e6ee4d0 R11: 0000000000000000 R12: ffff88011efa2e20
2009/08/31 00:03:01|R13: ffff88007faedb78 R14: 0000000000000000 R15: ffff88011efa2e08
2009/08/31 00:03:01|FS:  0000000000000000(0000) GS:ffff88012f8fd9c0(0063) knlGS:00000000f7cbb6c0
2009/08/31 00:03:01|CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
2009/08/31 00:03:01|CR2: 00000000093c2000 CR3: 000000007fada000 CR4: 00000000000006a0
2009/08/31 00:03:01|DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
2009/08/31 00:03:01|DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
2009/08/31 00:03:01|Process iscsi-scstd (pid: 22488, threadinfo ffff88007faec000, task ffff880116d00290)
2009/08/31 00:03:01|Stack:  ffff880106f68000 ffffffffa01cb0d9 0000200000000001 0000000000000000
2009/08/31 00:03:01|0100000200010000 ffff88011c2d8000 000005d8210f0040 0000000100000000
2009/08/31 00:03:01|000005d8210f0040 0000080000000000 0000000000000800 ffff88011efa2e00
2009/08/31 00:03:01|Call Trace:
2009/08/31 00:03:01|[<ffffffffa01cb0d9>] ? session_add+0x359/0x480 [iscsi_scst]
2009/08/31 00:03:01|[<ffffffffa01c97b8>] ? ioctl+0x368/0x430 [iscsi_scst]
2009/08/31 00:03:01|[<ffffffff8069c592>] ? __down_read+0x12/0xa0
2009/08/31 00:03:01|[<ffffffff804256c8>] ? __down_write_trylock+0x48/0x60
2009/08/31 00:03:01|[<ffffffff804256c8>] ? __down_write_trylock+0x48/0x60
2009/08/31 00:03:01|[<ffffffffa001d663>] ? di_read_unlock+0x73/0x130 [aufs]
2009/08/31 00:03:01|[<ffffffffa001c497>] ? h_d_revalidate+0x4c7/0x6b0 [aufs]
2009/08/31 00:03:01|[<ffffffff8069c592>] ? __down_read+0x12/0xa0
2009/08/31 00:03:01|[<ffffffff804256c8>] ? __down_write_trylock+0x48/0x60
2009/08/31 00:03:01|[<ffffffff804256c8>] ? __down_write_trylock+0x48/0x60
2009/08/31 00:03:01|[<ffffffff80425701>] ? __up_read+0x21/0xb0
2009/08/31 00:03:01|[<ffffffffa001d663>] ? di_read_unlock+0x73/0x130 [aufs]
2009/08/31 00:03:01|[<ffffffffa001c497>] ? h_d_revalidate+0x4c7/0x6b0 [aufs]
2009/08/31 00:03:01|[<ffffffffa001ce09>] ? aufs_d_revalidate+0x249/0x4f0 [aufs]
2009/08/31 00:03:01|[<ffffffff804256c8>] ? __down_write_trylock+0x48/0x60
2009/08/31 00:03:01|[<ffffffff80425701>] ? __up_read+0x21/0xb0
2009/08/31 00:03:01|[<ffffffff80425701>] ? __up_read+0x21/0xb0
2009/08/31 00:03:01|[<ffffffffa0004a57>] ? aufs_read_unlock+0x37/0x60 [aufs]
2009/08/31 00:03:01|[<ffffffffa001ce09>] ? aufs_d_revalidate+0x249/0x4f0 [aufs]
2009/08/31 00:03:01|[<ffffffff80425701>] ? __up_read+0x21/0xb0
2009/08/31 00:03:01|[<ffffffff802af5c1>] ? mntput_no_expire+0x21/0x120
2009/08/31 00:03:01|[<ffffffff802aa487>] ? __d_lookup+0xb7/0x150
2009/08/31 00:03:01|[<ffffffff80274ea2>] ? __alloc_pages_internal+0x92/0x430
2009/08/31 00:03:01|[<ffffffff80281cc1>] ? handle_mm_fault+0x431/0x840
2009/08/31 00:03:01|[<ffffffff802d0bef>] ? compat_sys_ioctl+0x31f/0x3f0
2009/08/31 00:03:01|[<ffffffff80284efe>] ? do_munmap+0x26e/0x310
2009/08/31 00:03:01|[<ffffffff80425701>] ? __up_read+0x21/0xb0
2009/08/31 00:03:01|[<ffffffff80228282>] ? ia32_sysret+0x0/0xa
2009/08/31 00:03:01|
2009/08/31 00:03:01|
2009/08/31 00:03:01|Code: 48 c7 41 08 00 02 20 00 48 c7 83 a8 10 00 00 00 01 10 00 48 8b bb c8 10 00 00 e8 d4 74 0c e0 48 89 df e8 cc 74 0c e0 5b 31 c0 c3 <0f> 0b eb fe 66 66 66 90 41 57 45 31 ff 41 56 41 55 49 89 f5 be 
2009/08/31 00:03:01|RSP <ffff88007faed848>

[chkohlruss]

Looks very similar to our Problem
How do you resolved this problem?

[symm]

Hey jiassic and chkohlruss

the errors that you posted are different,
jiassic may be drive related
he is showing these:
2009/08/31 00:03:01|[<ffffffff8069c592>] ? __down_read+0x12/0xa0
2009/08/31 00:03:01|[<ffffffff804256c8>] ? __down_write_trylock+0x48/0x60
2009/08/31 00:03:01|[<ffffffff804256c8>] ? __down_write_trylock+0x48/0x60
2009/08/31 00:03:01|[<ffffffffa001d663>] ? di_read_unlock+0x73/0x130 [aufs]
2009/08/31 00:03:01|[<ffffffffa001c497>] ? h_d_revalidate+0x4c7/0x6b0 [aufs]
2009/08/31 00:03:01|[<ffffffff8069c592>] ? __down_read+0x12/0xa0
2009/08/31 00:03:01|[<ffffffff804256c8>] ? __down_write_trylock+0x48/0x60
2009/08/31 00:03:01|[<ffffffff804256c8>] ? __down_write_trylock+0x48/0x60
2009/08/31 00:03:01|[<ffffffff80425701>] ? __up_read+0x21/0xb0
2009/08/31 00:03:01|[<ffffffffa001d663>] ? di_read_unlock+0x73/0x130 [aufs]

while chkohlruss
you have different errors
2009/09/08 06:19:54 kernel:[<ffffffff80237390>] ? put_files_struct+0x70/0xc0

2009/09/08 06:19:54 kernel:[<ffffffff80237a9b>] ? do_exit+0x17b/0x8b0

2009/09/08 06:19:54 kernel:[<ffffffff80238244>] ? do_group_exit+0x34/0xa0

2009/09/08 06:19:54 kernel:[<ffffffff80228282>] ? ia32_sysret+0x0/0xa

Best is to send the entire logs to suport to be sure.
Did you guys hear back from them?

[jisaac]

Support sent me a patch file, which I believe is an update to SCST (version 1.0.1.1). I have not actually implemented the patch due to the necessity of shutting down all of my VM's to do the upgrade. That's a pretty hefty maintenance window to try to schedule.