Error messages and system hangs

[chkohlruss]

Here are some other error messages we get:

2009/09/08 11:52:08	kernel:CPU 2 

2009/09/08 11:52:08	kernel:Pid: 22750, comm: apache2ctl Tainted: G      D   2.6.27.10-oe64-00000-g9b2116f #12

2009/09/08 11:52:08	kernel:RIP: 0010:[<ffffffff802961b8>]  [<ffffffff802961b8>] filp_close+0x18/0xa0

2009/09/08 11:52:08	kernel:RSP: 0000:ffff88007fb99ea8  EFLAGS: 00010246

2009/09/08 11:52:08	kernel:RAX: ffff88013f24c010 RBX: db5abe1715003e40 RCX: 0000000000000100

2009/09/08 11:52:08	kernel:RDX: ffff88013f24c000 RSI: ffff880129471680 RDI: db5abe1715003e40

2009/09/08 11:52:08	kernel:RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000000000002

2009/09/08 11:52:08	kernel:R10: 0000000000000008 R11: ffffffff803fde80 R12: 0000000000000000

2009/09/08 11:52:08	kernel:R13: ffff880129471680 R14: 0000000000000001 R15: 0000000000000001

2009/09/08 11:52:08	kernel:FS:  0000000000000000(0000) GS:ffff88013f871740(0000) knlGS:0000000000000000

2009/09/08 11:52:08	kernel:CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b

2009/09/08 11:52:08	kernel:CR2: 00000000f7f6dacc CR3: 0000000000201000 CR4: 00000000000006a0

2009/09/08 11:52:08	kernel:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000

2009/09/08 11:52:08	kernel:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

2009/09/08 11:52:08	kernel:Process apache2ctl (pid: 22750, threadinfo ffff88007fb98000, task ffff88013d79b890)

2009/09/08 11:52:08	kernel:Stack:  0000000000000001 0000000000000002 ffff88013e76e500 ffffffff80237390

2009/09/08 11:52:08	kernel:0000000000000000 ffff88013d79b890 0000000000000000 0000000000000000

2009/09/08 11:52:08	kernel:0000000000000000 ffffffff80237a9b 00000000fffffff2 0000000000000000

2009/09/08 11:52:08	kernel:Call Trace:

2009/09/08 11:52:08	kernel:[<ffffffff80237390>] ? put_files_struct+0x70/0xc0

2009/09/08 11:52:08	kernel:[<ffffffff80237a9b>] ? do_exit+0x17b/0x8b0

2009/09/08 11:52:08	kernel:[<ffffffff80238244>] ? do_group_exit+0x34/0xa0

2009/09/08 11:52:08	kernel:[<ffffffff80228282>] ? ia32_sysret+0x0/0xa

2009/09/08 11:52:08	kernel:

2009/09/08 11:52:08	kernel:

2009/09/08 11:52:08	kernel:Code: ff 66 90 89 f2 be 41 02 00 00 e9 f4 fd ff ff 66 66 66 90 48 83 ec 18 48 89 1c 24 48 89 6c 24 08 48 89 fb 4c 89 64 24 10 45 31 e4 <48> 83 7f 28 00 48 89 f5 74 50 48 8b 47 20 48 85 c0 75 35 48 89 

2009/09/08 11:52:08	kernel:RSP <ffff88007fb99ea8>

2009/09/08 06:19:54	kernel:CPU 6 

2009/09/08 06:19:54	kernel:Pid: 23675, comm: apache2ctl Tainted: G      D   2.6.27.10-oe64-00000-g9b2116f #12

2009/09/08 06:19:54	kernel:RIP: 0010:[<ffffffff802961b8>]  [<ffffffff802961b8>] filp_close+0x18/0xa0

2009/09/08 06:19:54	kernel:RSP: 0000:ffff88002f4c7ea8  EFLAGS: 00010246

2009/09/08 06:19:54	kernel:RAX: ffff88013d356810 RBX: db5abe1715005cc0 RCX: 0000000000000003

2009/09/08 06:19:54	kernel:RDX: ffff88013d356800 RSI: ffff88013ebd9080 RDI: db5abe1715005cc0

2009/09/08 06:19:54	kernel:RBP: 0000000000000002 R08: ffff88002f4c6000 R09: 0000000000000000

2009/09/08 06:19:54	kernel:R10: ffff880028073af0 R11: ffff88013fa43858 R12: 0000000000000000

2009/09/08 06:19:54	kernel:R13: ffff88013ebd9080 R14: 0000000000000001 R15: 0000000000000001

2009/09/08 06:19:54	kernel:FS:  0000000000000000(0000) GS:ffff88013f9afbc0(0000) knlGS:0000000000000000

2009/09/08 06:19:54	kernel:CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b

2009/09/08 06:19:54	kernel:CR2: 00000000f7f6eacc CR3: 0000000000201000 CR4: 00000000000006a0

2009/09/08 06:19:54	kernel:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000

2009/09/08 06:19:54	kernel:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

2009/09/08 06:19:54	kernel:Process apache2ctl (pid: 23675, threadinfo ffff88002f4c6000, task ffff88007fab2790)

2009/09/08 06:19:54	kernel:Stack:  0000000000000001 0000000000000002 ffff88013d07dd00 ffffffff80237390

2009/09/08 06:19:54	kernel:0000000000000000 ffff88007fab2790 0000000000000000 0000000000000000

2009/09/08 06:19:54	kernel:0000000000000000 ffffffff80237a9b 00000000fffffff2 0000000000000000

2009/09/08 06:19:54	kernel:Call Trace:

2009/09/08 06:19:54	kernel:[<ffffffff80237390>] ? put_files_struct+0x70/0xc0

2009/09/08 06:19:54	kernel:[<ffffffff80237a9b>] ? do_exit+0x17b/0x8b0

2009/09/08 06:19:54	kernel:[<ffffffff80238244>] ? do_group_exit+0x34/0xa0

2009/09/08 06:19:54	kernel:[<ffffffff80228282>] ? ia32_sysret+0x0/0xa

2009/09/08 06:19:54	kernel:

2009/09/08 06:19:54	kernel:

2009/09/08 06:19:54	kernel:Code: ff 66 90 89 f2 be 41 02 00 00 e9 f4 fd ff ff 66 66 66 90 48 83 ec 18 48 89 1c 24 48 89 6c 24 08 48 89 fb 4c 89 64 24 10 45 31 e4 <48> 83 7f 28 00 48 89 f5 74 50 48 8b 47 20 48 85 c0 75 35 48 89 

2009/09/08 06:19:54	kernel:RSP <ffff88002f4c7ea8>

[symm]

Could be problems with memory
what version and how much RAM do you have in your system?

May also want to send logs to support to see what they say.
keep us posted

[chkohlruss]

Have 4 GB in each machine.
I change the memory and try it again.

[symm]

What did support say about the logs?
did you try to run a memory test?
Is it only happening on the primary machine ?

[chkohlruss]

Haven´t sent the logs to the support yet.
A memory test did not show any errors.
And yes, it´s only at the primary machine.

[jisaac]

That looks very similar to an error we got a few weeks ago. I recommend you open a ticket with support and have them look at the logs.

[chkohlruss]

Opened a ticket.
Could you tell me what the error was in your case?